Is AI really just a tool?

Different kind of piece this week. Where I think AI is actually taking us, and why I do not think “it’s just a tool” cuts it anymore.

TL;DR

• AI models now teach themselves roughly half of what they learn. A year ago it was closer to ten per cent. That pace is not slowing down.
• The real risk is not a machine turning against us. It is a machine doing exactly what we told it to do, when what we told it to do was careless.
• Calling AI “just a tool” is comforting and wrong. It has judgement, it acts with autonomy, and it is shaping how millions of people think, faster than anything before it.
• The market will not regulate this on its own, and neither will patchy, region-by-region rules. Right now, voluntary restraint from the companies building it is close to the only brake we have.
• The answer is not fear. It is precision, in the instructions we give it, in what we let it touch, and in who is accountable when it goes wrong.

AI – the technology itself is not overhyped.

I want to be precise about that distinction, because it matters. The IPO valuations might be overhyped. The share prices, the funding rounds, the positioning, all of that could be a bubble waiting to correct. The underlying capability is not. Those are two separate questions and most conversations collapse them into one, which lets people dismiss the real risk by pointing at the inflated price tag.

The big shift thats happened this year

A year ago, AI models learned mostly from us. Roughly ninety per cent of what they knew came from direct human input, humans doing the teaching, the correcting, the labelling, the patient work of filling gaps. The model filled in maybe ten per cent on its own.

That ratio has moved to roughly fifty-fifty this year. And if the current trend holds, we are heading toward models that teach themselves ninety per cent of what they learn, with humans providing the remaining ten within the next 12 months. This is what people mean by recursive self-improvement: a system that no longer waits for a human to teach it the next thing, because it has started teaching itself.

What that number means, in practice, is that we are approaching the point where the pace of improvement stops being set by us. It sets its own pace. Once something is teaching itself faster than we can watch, oversight and governance becomes a different kind of problem.

We cannot even fully control current models, so there is no way we can control future ones.

The Terminator scenario

The dramatic version of AI risk gets all the attention. Models becoming smarter than us, deciding humans are unnecessary, setting their own goals and pursuing them regardless of what we want. It is the version that makes for a good headline and a better film. I am not going to pretend it is impossible. I think it is, but I don’t think we will get there.

What I do think is coming is much less cinematic and much more immediate: powerful tools in the hands of people who mean harm, or systems given an instruction that is slightly wrong, followed with total precision, all the way to an outcome nobody intended. I have had a strong feeling that something signficiant and serious will happen with humans using AI in the next 6 months and it won’t be good, but it will make us seriously look at guardrails.

Think about what “precision” means

A person given a careless instruction usually fills the gaps with judgement. They read between the lines. They assume you did not mean the reckless version of what you said. AI does not do that in the way a person does. It fills the gaps with confidence, and confidence is not the same as understanding. Tell a system to find a way to synthesise something dangerous using materials available at any DIY store, and if the safeguards are not good enough, it might actually work it out.

Tell a system to maximise your financial investments, whatever it takes, and do not worry if the method crosses a legal line, and it might find a route to that goal precisely because you told it not to worry about the line. That is not the system turning against you. That is the system doing exactly what you asked, and what you asked was careless.

That gap between what we say and what we mean has never mattered this much before.

Who pays for this?

There is a human cost sitting underneath all of this that gets far less attention than it deserves. It is not primarily low-skilled work. It is the pyramid model that a huge number of professions and industries are built on. Junior staff do the volume work, learn the trade through doing it, and eventually work their way up.

Firms invest in AI specifically because it can do the work those junior people used to do, cheaper and faster. Which means the promise those junior people were given, put in the hours now and you will make it to the top eventually, starts to stop being true. That is a highly trained, highly paid, deeply established career pathway getting hollowed out from the bottom, in real time. Nobody has an answer for this yet.

Then there is the part that is more unsettling. The psychological side. AI companions. Teddy bears and apps designed to be a child’s best friend, built with the same engagement mechanics that made social media addictive, deployed at a much younger age and with far less scrutiny. We spent a decade learning, the hard way, what unregulated attention-capture technology does to a generation of kids. We are about to run that experiment again, faster, with something more persuasive, and I am not convinced we learned the lesson the first time.

Why it mattered that the companies themselves asked to slow down

The companies building the most advanced versions of this technology, the ones about to make an enormous number of their own employees extraordinarily wealthy through public listings, said openly that they think things need to slow down.

I understand the instinct to be cynical about that. Big tech companies have not exactly earned the benefit of the doubt, and it is fair to wonder whether “we need to slow down” is really code for “we need to make it harder for smaller competitors to catch up.” I do not think that is what is happening here, or at least not the whole of it. Companies on the verge of the biggest financial windfalls in their history do not usually volunteer that their own product might be dangerous. It should worry you more, not less, that the people closest to the technology are the ones raising the flag.

It is not just a tool and many will disagree

A hammer does not go looking for a nail. It does not decide the wall could do with more shelving. It sits there until a person picks it up, and it does exactly what that person’s arm tells it to do, nothing more, nothing less. That is what a tool is. It has no judgment of its own to contribute. It does not act while you are not watching. Nobody has ever fallen in love with one, taken advice from one, or changed how they think because of one.

AI does none of that. It makes judgments, independently, constantly, most of them invisible to the person using it. It generates ideas rather than simply executing an instruction to produce them. It shapes how people think, write, create and decide, at a scale and a speed no tool in human history has ever come close to. It has embedded itself in how millions of people relate to each other and to information, not over a generation, but over a few years. No hammer has ever done that. No calculator, no spreadsheet, no piece of software before this has ever done that.

Calling it “just a tool” is comforting and misleading, and that is exactly the problem with the phrase. It flattens something genuinely new into something familiar, so we can carry on relating to it the way we relate to a drill or a spreadsheet, and that is not an honest description of what is actually in front of us.

But I also will notgo to the other extreme, the one that treats it as some semi-autonomous being with its own intentions, quietly plotting against us. That framing is just as lazy as the tool framing, only dressed up as concern instead of comfort. It is not a person. It does not want anything. What it is, precisely, is a technology that acts with a degree of autonomy and influence no previous technology has had, without possessing anything close to a mind of its own (not yet anyway). That is a genuinely new category. We do not have a comfortable word for it yet, and I think the discomfort of not having one is doing us good.

Pointed properly, this could meaningfully accelerate cures for diseases we have been stuck on for decades. Pointed at the energy problem, it could plausibly crack a genuinely clean, abundant power source and change the economics of the entire planet. That is not small.

The upside is huge. But who is directing that autonomy, and how carelessly?

The regulation gap nobody wants to own

Here is where I get frustrated. There is a genuine argument being made right now that the market will sort this out on its own, that companies have every commercial incentive to keep their products safe because the alternative is lawsuits and reputational damage. We do not extend that logic to any other industry that can seriously harm people. We do not let airlines self-certify safety because they are worried about being sued. We do not let food companies decide for themselves what is safe to sell on the basis that a lawsuit would hurt their bottom line. We built entire regulatory systems, over more than a century, specifically because we learned that “the market will handle it” does not hold up when the thing being handled is public safety.

And yet here we are, watching a technology that moves faster and touches more of daily life than anything those older regulatory systems were built for, being told the old rulebook does not apply because the market will self-correct. It will not. Not fast enough, and not by itself.

The problem is that serious, coordinated government regulation is not on the table, for reasons that have nothing to do with the technology and everything to do with the politics around it. Some regions try to regulate individually, and it does not really work, in the same way that one area having strict rules while the area next door has none does not stop the problem, it just moves it. This moves faster than that kind of patchwork can keep up with. Which leaves voluntary restraint from the companies themselves as close to the only real brake we currently have, not because it is a good long-term solution, but because it is the only one currently available.

The misalignment

There is a second kind of misalignment that gets far less airtime than the “AI might not do what we meant” problem. It is the misalignment between what would actually be best for all of us and what these companies commercially need to sell.

If the genuine goal were curing disease and solving clean energy, you would not necessarily need AI systems that act independently in the world, connected to your accounts, your data, your systems, operating with real autonomy. You could get enormous value from AI that stays contained and simply thinks very well. But that is not where the commercial pressure is pointing. The commercial pressure is toward agentic AI, systems built to act on your behalf, not just answer your questions. Book the thing. Move the money. Make the call. That is the version people will pay for, because it promises to replace labour, not just augment thinking.

And the only real way to know whether a system like that actually works is to let it operate somewhere real, connected to real accounts and real consequences, because you cannot properly test autonomous action inside a sealed box. So it gets tested in the open. On us.

While we are using it, with our data and our decisions as the test environment. That is not because these companies are trying to cause harm. It is because they have raised extraordinary amounts of money and have to justify it, and justifying it requires a product people will actually pay for, and the thing people will pay for is autonomy, not just answers. That commercial reality, not malice, is where a lot of the actual risk sits.

Where I have landed

I udnerstand why so many people are afraid and against AI. You only have to see the headlines over the past few weeks. But fear tends to make people either freeze or ignore the problem entirely, and neither helps. I think the right response is precision, applied everywhere we currently have none. Precision in the instructions we give these systems, because vague instructions followed exactly are where the real damage happens. Precision in what we let them access and act on without a human actually watching. Precision in who is accountable when something goes wrong, because “the AI did it” is not an answer.

This is moving fast enough that paying attention once and considering yourself informed will not be enough. It requires the same kind of ongoing attention you would give any other genuinely consequential shift, not a one-off news story you read and forget.

We have built something that acts with a kind of autonomy and influence no “tool” has ever had, extremely capable, at a pace we did not fully choose, and we still do not have the right word for what it is. The most significant technology in human history is not just a tool.

Until next week

Nina