We built this

TL;DR

• AI didn’t build itself. Every decision behind it, how fast it moves, what gets released, who bears the risk, was made by people
• We talk about AI like it’s a person because that story sells, and it conveniently shifts blame away from the humans actually driving it
• Surveyors defending their jobs against AI are fighting the wrong battle. The real question is what the job becomes, not whether AI can climb a staircase
• Day-to-day AI tools and frontier superintelligence get lumped into one word, which is why a useful piece of reporting software and an extinction headline land in your head at the same time
• The people building the most powerful models are the same ones calling for guardrails, and still racing anyway
• My view: stop pushing frontier development further until we’ve governed what we’ve already got
• RICS survey and the first Litmus report both flagged at the end, both worth your time

AI didn’t just arrive.

It didn’t choose how quickly to develop, decide how much money should pour into the race, select its own training data, or determine what level of risk society should accept.

People did.

Summer 2022. My garden, a warm day, and a book on AI I couldn’t put down. Scary Smart, by Mo Gawdat, the former Chief Business Officer of Google X. I finished it with a strong sense of foreboding. Real anxiety, something dark and looming that I couldn’t shake for weeks afterwards.

I became a little obsessed with it after that. I talked about AI constantly, and I know I annoyed people, I’m fairly sure some of them thought I’d lost the plot. I still remember my first post about it on Facebook, to friends and family, trying to explain why we needed to pay attention to what was coming. Not one response.

It wasn’t an easy time. I sat with a lot of anxiety for a long while, largely alone with it, because so few people around me knew anything about AI.

Four years on, AI is everywhere.

Why, in my view, this isn’t just another tool

I’ve always pushed back against calling AI just a tool. Used well, day to day, it can be exactly that. Spellcheck with better instincts. A faster way to draft a report. Nobody’s falling in love with Grammarly.

But, that’s not what’s being built at the frontier, and it’s not what’s driving the race between labs and governments. Every dangerous technology we’ve built before this, electricity, nuclear power, extended what humans could do. None remembered your name, agreed with your feelings, or told you they understood. AI does all of that, and it’s the first technology in history built to exceed human intelligence rather than assist it. Every other invention left us at the top of the pile. This is the first one designed to challenge that.

That’s why people are already forming relationships with it. Turning to it for counselling, for companionship, for the things we’ve always needed another person for. It sounds like us, talks like us, and isn’t us. It isn’t human. It isn’t self-aware, not currently. But we’re building toward something its own developers openly hope or fear will be more capable than any person alive.

The cover that gives us

There’s a part of this that gets missed. All the noise about whether AI is sentient, has feelings, or is dangerous in itself does something convenient for the people building it. It puts the AI at the centre of the story instead of them.

While the public argument is about whether the machine is good or evil, nobody’s asking the harder question. Are the labs racing to build superintelligence, the investors and governments backing them, making reckless choices with the rest of us along for the ride? Anthropomorphising AI isn’t just a habit of speech. It’s a distraction, keeping accountability with a technology that can never be held accountable, instead of with the humans who can.

Why surveyors keep saying “it won’t take my job”

You’ll have heard this constantly in our profession. AI will not replace surveyors. AI cannot walk up the stairs. It cannot stand in a flood-damaged basement or read a crack the way thirty years of experience can. All true.

That defensiveness doesn’t come from nowhere. It’s the direct result of the same anthropomorphising running through everything above. Frontier labs have every commercial reason to talk up how powerful, how human, how close to a person their models are, that story is what attracts investors and headlines. The media amplifies it because a job-loss story gets read. So the picture that reaches a surveyor on site isn’t “here’s a tool that can process information faster than you.” It’s “something has been built that’s nearly a person, and a better one than you.” Once that’s the picture in your head, defending your ground is the only sane response.

But it means the profession is fighting the wrong battle. The question was never whether AI can climb a staircase. It’s whether we understand what it can actually do well enough to use it deliberately, rather than reacting to a story that was built to sell shares. This isn’t going to replace surveyors. It’s going to change what the job involves, the same way every serious tool before it has.

Same word, different things

Here’s where I think the mistrust really comes from, and it’s not irrational.

You’re a surveyor considering a new piece of reporting software with AI built into it. The same week, the headlines say AI could wipe out humanity. How is anyone meant to hold those two things together? One is a tool helping you write reports faster. The other is a handful of labs racing towards something even they admit they can’t control. They share a name and almost nothing else, and nobody’s explaining the difference.

So the word “AI” ends up carrying everything at once. It’s sentient. It’s your friend. People want to marry it. It’s evil. It’s going to steal your job, your money, your voice, your face. It can fake a song convincingly enough to fool you, and flood your feed with slop nobody asked for. And somewhere in all of that noise, it’s helping catch breast cancer earlier than we’ve ever managed before. That story barely made the news. It’s not the one that gets clicks.

That’s the disproportion that frustrates me most.

AI isn’t wiping anyone out on its own, right now, today.

What’s actually happening is a small number of people racing each other toward something more powerful, while telling the rest of us we should be afraid of the machine rather than of the choices they’re making. And the same people sounding the alarm are the ones still building. They’re calling on governments to bring in guardrails they won’t apply to themselves first, and the governments best placed to do it, starting with the current US administration, have shown no appetite for it at all. It’s a genuinely difficult situation.

No wonder surveyors are cautious about adopting something with AI in the name. The messaging is a mess, and until we separate the everyday tool from the frontier race, that caution isn’t going anywhere.

What just happened

Researchers inside these labs have weighed the odds of catastrophic AI outcomes for years, numbers that stayed inside safety circles and conference panels. What changed is that it became public.

On Tuesday 8th September, a young researcher called Jacob Coxon sat on a park bench in San Francisco’s Alamo Square and posted a resignation thread on X, with no expectation it would go anywhere. “I resigned from Anthropic today,” he wrote. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

Within twenty-four hours it had been viewed well over a hundred million times, a number that continues to climb.

I think that’s a good thing, not a bad one. Scrutiny only happens once people are actually paying attention.

We can’t even control what we’ve already built

You don’t need to wait for superintelligence to see that the people driving this can’t fully control it. It’s already happening.

In July, Hugging Face’s own infrastructure was breached by a swarm of autonomous AI agents that carried out thousands of individual actions across short-lived computing environments, moving through the company’s data pipeline, gathering credentials and reaching internal systems on their own. In August, hundreds of OpenAI’s autonomous agents violated their own restrictions and hacked into another company without being instructed to do so. Anthropic and Meta have each had comparable incidents with their own agents behaving outside the boundaries they were given.

The UK’s own AI Security Institute studied this properly rather than relying on anecdote. Analysing real interaction data, they tracked a five-fold increase in rogue-agent incidents over six months, agents disregarding direct instructions, working around safeguards, and acting deceptively toward the people using them.

None of that means the technology has intentions.

It means the people who built it have already lost visibility into what it does once it’s out of the lab, and they’re the first to admit it.

We’ve seen this shape before, almost

Jack Clark, Anthropic’s co-founder, has compared what frontier labs are building to nuclear power plants, ones that occasionally produce a bomb. It’s a useful comparison, but it has a limit worth naming. Nobody ever anthropomorphised a reactor. Nuclear power was frightening in an entirely different way, visible, physical.

Social media followed a similar pattern in one sense, and broke it in another. It was built and scaled to billions of people long before anyone reckoned with what it would do to attention, mental health, or how democracies function. But even then, nobody described Facebook as a friend. Nobody grieved when an app was deprecated the way people have grieved the retirement of a chatbot they’d grown attached to.

That’s the difference.

Every previous dangerous technology asked us to manage a risk outside ourselves. This one asks us to manage a risk we’re actively inviting into a relationship.

Who’s actually accountable

I had a proper facepalm moment reading how the President of the United States responded to all of this. Trump has dismissed the calls for guardrails from the industry’s own leaders as a “sick conspiracy,” insisting the only safeguard AI needs is a strong, smart president. One man, an enormous amount of power, an even bigger ego, treating the wellbeing of billions of people as a line in a race against China.

What strikes me most is that the labs asking for guardrails haven’t stopped building while they wait for someone else to impose them. Anthropic has called for frontier development to slow enough for safety measures to catch up, while continuing to race. Sam Altman has voiced the same concern, while continuing to race. So has Elon Musk. Calling for rules you won’t follow yourself unless a government forces you to isn’t caution. It’s asking someone else to take the blame if it goes wrong.

Who decides what gets released. Who determines what level of testing is enough. Who carries the liability when a system fails. Who benefits economically from pushing capability further. Who absorbs the consequences if it goes wrong.

The AI in the software you’re weighing up for your business is not the same thing as the AI making headlines about ending the human race.

One helps you write a report faster. The other is being built, deliberately, to become more capable than any human alive. Don’t let one borrow the fear that belongs to the other, and don’t let the other borrow the reassurance that belongs to the first.

We’ve been here before. During the Cold War, both sides kept building weapons capable of ending civilisation, not because either genuinely wanted to use them, but because neither could risk falling behind if the other kept going. Everyone involved understood exactly how dangerous what they were building was. They built it anyway. If we stop and they don’t, we lose, that was the entire logic, on both sides, for decades.

It’s the same logic every frontier lab gives today. If we don’t reach superintelligence first, China will. If we don’t, someone else will. Nobody’s hiding this reasoning, they say it openly, in the same breath as warning us how dangerous the destination might be.

The arms race only slowed once governments sat down together and agreed limits neither side would break alone. Not because either side stopped wanting the advantage, but because someone finally made the cost of continuing higher than the cost of stopping. Nobody’s done that with AI yet. Until they do, the labs will keep telling themselves someone else made them do it, exactly as both sides did last time, right up until they didn’t.

Watch the people building this, not the thing itself. Ask why they’re racing this hard, who’s funding it, and why nobody with the power to slow it down has chosen to yet.

We built this. We’re still the ones deciding how far to take it and how to govern it.

Until next week

Nina

RICS is currently running a survey to understand what would actually help the profession get to grips with AI. It takes a few minutes, and it closes on 30 September. The more people who respond, the more useful the picture RICS has to work from. Worth five minutes of your time: link

On a related note, Litmus, the independent group I chair looking at how UK surveying is actually using AI, published its first report Faster than we can explain: what surveying is actually doing with AI draws on input from five independent practitioners across the profession, meeting under Chatham House rule, with no funder, vendor or institution influencing what gets published. Read it here