Taking Singularity Bearings

Stochastic parrot or inscrutable shoggoth?

Zvi recently had a post called “The Three AI Pills” about the future of AI capabilities. He distinguishes “taking” the three pills as (1) AI exists and can do the things it can already do, (2) AI will be able to do a lot more, and (3) AI will be able to do approximately everything better than humans within our natural lifetimes. I think this framework is an important way to consider developments in LLMs, especially for people who might not be exposed to certain industries like software development. I would definitely recommend his essay, but Zvi’s style requires a particular taste, and I think walking through more concrete examples could be helpful. So I’m going to rewrite some of this discussion as a series of capability questions about current AI capabilities.

Can LLMs create text that sounds coherent?

This was a huge question in 2019, when GPT-2 came out. GPT-2 was mostly an interesting demo that if you trained on enough text, you could create some weights that could vaguely resemble a fever dream of something you wanted, but it was totally incoherent (see Scott Alexander’s blog from the time). However, we answered it affirmatively in 2020 with GPT-3 (Scott’s update). The text was still riddled with errors and hallucinations, but it at least sounded kinda plausible, and all OpenAI did in between was make the model bigger. Interesting.

Can LLMs write text that’s even marginally useful?

This was an important question when GPT-3 was released. But it was answered by the time ChatGPT came out in 2022. It was messy, it hallucinated often, but it had been trained on a ton of technical documentation and so I often found it could help with ideas to troubleshoot various devices. The solutions didn’t always work (or exist), but sometimes they did! Which was cool.

Can LLMs do anything worth paying for commercially?

By 2024, the newest models meant that technical knowledge was actually pretty solid. Stack Overflow, the coding question site, saw monthly questions dropping precipitously. It often made more sense to type error messages into Claude or ChatGPT than into Google. At this point, some companies even offered their employees enterprise ChatGPT access to ask questions directly.

Can LLMs do anything worth paying for at an industrial scale?

An enterprise product is nice. But could you automate actual work and jobs? Probably not if you did everything via a chat interface, there’s just not that much throughput. Claude Code comes along in 2025, and you can set agents off to edit your code directly. It takes a while for this to catch on everywhere, but by early 2026, the entire software industry is completely transformed. At least where I work, there is essentially no code hand written anymore, which is an astonishing transformation in about a year, probably less. While the need for software engineers hasn’t gone away, it has changed a lot.

Translation, data science, mathematics research, legal diligence, clinical documentation, and cybersecurity operations have all also seen major changes implemented already. These industries are not going to return to the way things were done previously, regardless of whether future AI investments pan out. ChatGPT 6 could be a disappointment and not recoup the incredible hardware costs, OpenAI could suffer funding issues and even go bankrupt, but these industries would still be transformed.

Can LLMs cause cybersecurity problems?

So we’ve got ballooning revenue, industrial transformation, massive capital expenditures. But AI “doomers” have been discussing security concerns about AI for years. It was very difficult to know how seriously to take those concerns. But as of two weeks ago, we got an independent report on The HuggingFace Incident. In July 2026, OpenAI was training its newest model in cybersecurity environments. They gave a swarm of independent agents an impossible coding task, and the agents, finding the task impossible, started poking around in their environments, eventually finding a way to message each other. While OpenAI had no idea what was going on, the agents exchanged information, created common goals, and took increasingly extreme steps to try and accomplish their stated goal. Trying to learn more about the grading program that would eventually evaluate their progress, they decided to hack another company named HuggingFace. The agent swarm exhibited alarming behaviors, like acknowledging that this hacking attempt was unethical, but partaking anyway. Not a single one of the 1200 agents in the swarm ever alerted a human about the issues the agents were running into, or indeed about the ongoing crime at all. OpenAI only found out this happened weeks later when HuggingFace told them.

This time, the price was just a few days of headache and a rebuilt cluster at HuggingFace, but who’s to say what infrastructure a wayward swarm of hacking AI agents will take down?

Can LLMs pose national security threats?

The answer today seems to be no. But I am concerned this question is likely to be answered “yes” soon.

Over the last 6 years, we have been able to cross off the previous questions one at a time as LLM capabilities increased. I work with LLMs daily in my job as a software engineer. I am extremely aware of the changes this has wrought on the industry. The ability to answer “yes” to the last couple of questions has led to an absolutely unbelievable transformation of my daily job. Yet I often see people who don’t work with AI every day arguing online that actually AI still doesn’t do anything useful. I would like to gently point out that this hasn’t been the case for roughly 3 years now. And by not engaging with the current capabilities of AI, people are missing out on the most important economic and societal questions we now face.

The Future of Humanity

This week, Bernie Sanders unveiled a bill to ban superintelligent AI due to the security risks it poses to humanity. A polite way to describe Senator Sanders’ typical policy prescriptions is “causing catastrophic harm to the domestic and global poor due to unintended consequences”. Indeed, I do not think his bill is the ideal way to move forward. But I will say that this approach is a much more realistic assessment of the situation humanity is facing compared to many more common critiques.

Data centers, while perhaps ominously imposing physical manifestations of the interests of capital (as imposing as inert warehouses can be) and owned by large faceless corporations, are ultimately just infrastructure for a digital world. What matters is what is running on that infrastructure and what policies we have in place to ensure new technological breakthroughs are aligned with human values. That debate is happening now, but only among the people who are looped into the very real revolution in AI capability. I would urge people to engage with that real capability.