Unreliable general intelligence is useless

When I was a child, I really thought being able to do math in my head — say, 2 digits multiplied by 2 digits — meant you were smart and intelligent. I learned that I could do it (barely, tbh) and many kids couldn’t, and I used to make fun of them (no wonder kids hated me). To this day, my cousins are angry at me for making them do that in front of their parents.

Obviously it only took a couple of years for me to realize how idiotic it was of me to think that had anything to do with being smart or intelligent.

We don’t even know what we don’t know

I think the most accurate and useful way of thinking about what intelligence (and being “smart”) is, is to consider it a “bundle” of different features (similar to consciousness itself).

Planning, arithmetic, simulation, learning, language, reasoning, memory, meta-recognition, visual recognition, ... the list is very, very long and unclear and full of surprises.
The most “intellectually” honest view we can take on this subject is that we don’t even know what we don’t know.

Meaning, “intelligence”, let alone “AGI”, is a very poorly defined term. Depending on where we draw the line for the minimum size of the bundle, anything from a crow (memory, tool use) to a whale (shared culture) to a virus (self-replication) can be considered “intelligence”.

I know some tech bros (unironically) think of some traits as a good measure of intelligence — playing chess or being a polyglot or memorizing — and obviously, that’s a stupid oversimplification. Arguably the greatest military strategist who ever lived was “not impressive” at chess.¹Napoleon only has 4 recorded games, but we can all agree he wasn’t at any pro or competitive level even in his time, for sure. Einstein famously forgot his own address and had to call his office to ask where he lived.²The truthfulness of this story has been disputed, since there is no formal record, but many people who knew him said it sounds very plausible knowing his usual behavior. The version told here is from Walter Isaacson’s Einstein: His Life and Universe (2007).

But his ability — I’ll call it intuition for lack of a better term — to start from “the speed of light is the same regardless of the observer” and get to “time and space must be relative depending on the observer” is clearly some sort of “intelligence” (by definition). But what is it even? How are you going to quantify it, or say we’ve achieved it?³Demis Hassabis, at the India AI Summit in February 2026: “The kind of test I would be looking for is training an AI system with a knowledge cutoff of, say, 1911, and then seeing if it could come up with general relativity, like Einstein did in 1915.” He added that it’s clear today’s systems couldn’t do that. Source. (The example above is special relativity, from 1905. General relativity came ten years later.)

OpenAI defines “general intelligence” as “highly autonomous systems that outperform humans at most economically valuable work”.
Okay, but the socioeconomic value of each feature in the bundle changes all the time. Now, being able to do 52514 × 25353 in your head is nothing more than a party trick that has no practical value. This skill would have been useful if you were working for the court of the Roman Empire, I suppose, where they had data but no way to calculate it fast. But then, if you go back in time just a little more, to where you don’t ever have to deal with a number like 52514, again, this would have been a completely useless skill.

So the recent discourse (more like corporate marketing) around “is this AGI or not”, I honestly find pointless and just wasteful. Like, we don’t even fully understand nor have a shared definition of the word we use. What are we even talking about?

Usefulness and reliability

The usefulness of a given automation, depending on its reliability, varies a lot from task to task. The things we’ve been using LLMs for until now are good examples of getting a fair level of usefulness even with very flaky reliability:

• Tasks that don’t have a verifiable or singular answer, so the error margin is huge anyway (AI girlfriend, life coaching, …)
• Tasks that do have a logically complete, verifiable result, where a human identifying the issue and fixing it is easy (coding, math)

But obviously, to call it “general” it should cover “most economically valuable tasks”.
Most tasks are not like this.

Think about autonomous driving. When we say it “works”, we obviously mean and imagine it being better than a human in every way. It’ll never get tired or get distracted. It’ll never miss anything from the sensors. It will be more “reliable” than any human driver.⁴I understand a lot of people will want to argue that it should be compared to the “average driver” and that anything better than the “average driver” should be allowed. That’s just stupid and wrong: “We want to cause mass job displacement and huge social unrest in order to adopt automation that is ‘slightly’ better than the ‘average’ human driver, which doesn’t even make economic sense right now either!”? We don’t imagine it randomly going onto roads that don’t exist or ignoring the law with a 5% chance. If so, we wouldn’t say “it’s 95% accurate”, we’d say “it’s not good enough”.⁵At the same time, I’m not disputing the fact that self-driving cars, at least Waymo, are safer than the average human driver. But they still make many mistakes that an “average” human driver would not make, such as freaking out over a puddle.

How about a rocket? Where, quite literally, if the LLM hallucinated a single word or ignored one word from the specification document, the whole rocket could explode.

LLMs are unreliable and not logically complete, by design.⁶There are many explanations for this, ranging from the lack of neuro-symbolism from Yann LeCun et al., but The Illusion of Thinking from Apple is my favorite. Even though it underplays their significance, an LLM IS “autocomplete on steroids.” It’s just that everybody underestimated how many problems we can solve with autocomplete.

To be clear, I’m not saying LLMs are entirely useless. They’re already useful for many tasks, and I’m certain they’ll create many new business opportunities and startups. As a matter of fact, I myself built an LLM-based agentic web browser at a previous startup, and now I’m building custom automation and agents for a rocket company.

But that’s not what those companies are claiming. They don’t claim “LLMs are a novel technology that is really good at solving certain problems”.
They’re calling it “artificial general intelligence that can kill us all with a 10% chance”.

Aside from the many layers of logical paradox in that statement (how could something be AGI if it’s unreliable?), the answer is simple here: unreliable general intelligence is useless.

1. 1.Napoleon only has 4 recorded games, but we can all agree he wasn’t at any pro or competitive level even in his time, for sure. ↩
2. 2.The truthfulness of this story has been disputed, since there is no formal record, but many people who knew him said it sounds very plausible knowing his usual behavior. The version told here is from Walter Isaacson’s Einstein: His Life and Universe (2007). ↩
3. 3.Demis Hassabis, at the India AI Summit in February 2026: “The kind of test I would be looking for is training an AI system with a knowledge cutoff of, say, 1911, and then seeing if it could come up with general relativity, like Einstein did in 1915.” He added that it’s clear today’s systems couldn’t do that. Source. (The example above is special relativity, from 1905. General relativity came ten years later.) ↩
4. 4.I understand a lot of people will want to argue that it should be compared to the “average driver” and that anything better than the “average driver” should be allowed. That’s just stupid and wrong: “We want to cause mass job displacement and huge social unrest in order to adopt automation that is ‘slightly’ better than the ‘average’ human driver, which doesn’t even make economic sense right now either!”? ↩
5. 5.At the same time, I’m not disputing the fact that self-driving cars, at least Waymo, are safer than the average human driver. But they still make many mistakes that an “average” human driver would not make, such as freaking out over a puddle. ↩
6. 6.There are many explanations for this, ranging from the lack of neuro-symbolism from Yann LeCun et al., but The Illusion of Thinking from Apple is my favorite. ↩