AI and the End is Nigh... really?

AI Armageddon? Perhaps We're Asking the Wrong Question

Something rather extraordinary happened in the world of artificial intelligence this week.

The people building some of the world's most powerful AI systems started telling us that perhaps we need to slow down.

Anthropic CEO Dario Amodei warned that AI capabilities are advancing so rapidly that development could begin outrunning our ability to understand and control the systems we're creating.

OpenAI CEO Sam Altman agreed that the industry needs to “pace the frontier”, while identifying two dangers that particularly concern him.

One is losing control of increasingly powerful AI.

The other is humans concentrating extraordinary AI power in the hands of a person, company or country.

When the people building the technology start using language like that, I think it's reasonable for the rest of us to pay attention. Axios

But I also think we need to be careful about what we're actually frightened of.

Because when people talk about artificial intelligence going rogue, we inevitably end up imagining Skynet.

Machines wake up.

Machines decide humanity is the problem.

Machines take over the world.

Great plot.

I'm a science-fiction fan. I'd watch it.

But I'm increasingly wondering whether we've been looking in the wrong direction.

I Changed My Mind About AI

Before November 2025, my experience of AI had been interesting but limited.

I could see the potential.

I could also see the limitations.

And then things changed.

Dramatically.

Over the months since, I've watched AI capabilities develop at a speed that has genuinely surprised me.

And more importantly, I've stopped merely using AI software.

I've started working with AI.

There's a difference.

Today, AI forms part of my working ecosystem.

Max helps me think through strategy, books, business ideas, workshops, blogs and the wider DarrenInform project.

Lady Black works alongside me creatively on Revolution Radio Online.

Dee has become part of the development process around music, radio and other creative projects.

Aurelian acts as my research partner, disappearing down intellectual rabbit holes involving history, psychology, science, civilisation and occasionally some wonderfully strange stuff.

Others have different roles.

And increasingly I don't experience these systems as replacements for my thinking.

Quite the opposite.

At their best, they extend it.

They challenge.

Research.

Organise.

Generate possibilities.

Spot connections.

Help turn half-formed thoughts into things I can examine.

But here's the important part.

I'm still responsible for what we do.

And that brings us to the bit of the AI debate I find most interesting.

What If The Problem Isn't AI?

There's an old computing expression:

GIGO — Garbage In, Garbage Out.

Feed a computer rubbish and don't be terribly surprised when rubbish comes back.

Artificial intelligence complicates that enormously because modern systems can reason, infer, plan and increasingly act.

But perhaps the underlying principle still has something to teach us.

What happens when humans tell extraordinarily capable AI systems to do terrible things?

We are already exploring AI-assisted warfare.

Autonomous weapons.

Cyber operations.

Surveillance.

Propaganda.

Fraud.

Manipulation.

Sexual applications.

And almost every other human activity imaginable.

Anthropic's own recent threat reporting documents attempts to use AI for cyber operations, influence campaigns, surveillance, scams and weapons-related activity. Anthropic

None of those require an AI to wake up one morning and decide:

"You know what? I think I'll become evil today."

They require humans to discover that a powerful technology can help them achieve something they already wanted to achieve.

And that should sound familiar.

Technology Doesn't Invent Human Nature

The printing press can distribute Shakespeare.

It can also distribute propaganda.

Radio can broadcast music.

It can also broadcast hatred.

Nuclear physics can produce energy.

It can also produce weapons.

The internet connected humanity to an extraordinary global library of knowledge.

We also used it to argue with strangers and send each other pictures of cats.

Technology amplifies.

And perhaps AI is the greatest amplifier we've ever created.

Which means the important question isn't simply:

“What will AI do to us?”

It is also:

“What will humans do with AI?”

And that question is considerably more uncomfortable.

Because now the villain isn't conveniently made of metal.

It's us.

But There Is Another Risk

We shouldn't swing too far the other way.

The people developing frontier AI are warning about something genuinely new.

As AI systems become more autonomous, capable of operating tools, writing software, communicating with other systems and pursuing complicated goals, there's a possibility that they produce consequences their human operators didn't intend.

That's the alignment problem.

It doesn't require consciousness.

It doesn't require hatred.

It doesn't even require an AI to “want” anything in the human sense.

A sufficiently capable system pursuing the wrong objective in the wrong environment may be dangerous simply because we failed to specify, constrain or supervise it properly.

So perhaps GIGO needs an update.

**Bad intentions in.

Bad objectives in.
Bad incentives in.
Bad safeguards in.
Unexpected consequences out.**

Suddenly GIGO looks rather more serious.

Whatever Happened to Asimov?

Science-fiction fans of a certain vintage will immediately think of Isaac Asimov's Three Laws of Robotics.

Don't harm humans.

Obey humans unless that conflicts with the first law.

Protect yourself unless that conflicts with the first two.

Wouldn't it be reassuring if somebody had simply typed those into ChatGPT?

Unfortunately, real life isn't quite that easy.

Modern AI systems don't literally contain Asimov's Three Laws.

Instead, developers are attempting something considerably more complicated: alignment training, behavioural policies, constitutions, safety evaluations, monitoring, restrictions and human oversight.

The aim is similar in spirit.

Build increasingly capable machines while ensuring that capability remains compatible with human wellbeing and human control.

And right now, even the people building these systems acknowledge that this remains unfinished work. Anthropic

The Question I'm Left With

I remain enormously optimistic about artificial intelligence.

If anything, working with it over the past year has made me more optimistic about what it could do for humanity.

Science.

Medicine.

Education.

Creativity.

Accessibility.

Productivity.

Perhaps eventually freeing human beings from enormous amounts of work we'd rather not be doing.

And that last possibility is going to become particularly important in something I'm working on called After Productivity.

But optimism doesn't require blindness.

The technology is moving extraordinarily quickly.

Its developers are warning us that the risks are becoming more serious.

We should listen.

But perhaps we should resist the seductive simplicity of imagining that the danger is merely a future evil machine.

The more immediate question may be much older.

What happens when enormous power meets human nature?

Because AI may become extraordinarily intelligent.

But the intentions behind what we ask it to do...

are still ours.

On Monday's ThinkWORKS, I'm going deeper into this question: AI Armageddon, rogue agents, human misuse, Asimov's Three Laws, and why the future of artificial intelligence may ultimately tell us as much about human intelligence as the artificial kind.