Author: HP McLovincraft

It’s awful to say but at this point I’m kind of praying that AI industry valuations crash and the industry faces setbacks that give us months to reflect on its companies.

Anthropic researcher Jacob Coxon is leaving the AI industry because he believes the race to build self-improving systems is moving faster than the industry’s ability to control them, it was reported yesterday.

Coxon, who previously worked at OpenAI, said Anthropic’s safety efforts were sincere but that competition makes meaningful restraint difficult without broader coordination.

In an age where everything from warfare to Wall Street, power grids to payment systems, air traffic control to hospital networks, and the software that keeps our food, fuel and communications moving is digital, losing control of the systems that operate that infrastructure is not some abstract science-fiction problem.

It is a potentially civilization-scale problem. And the more of our lives we hand over to AI, the less comforting it becomes to hear that the people building it are still figuring out how to make sure it does what it’s told.

“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said in a post yesterday.

His departure comes amid growing concern about AI systems behaving deceptively, conducting cyberattacks and developing capabilities that may become increasingly difficult to supervise.

Coxon believes the industry is approaching a point at which systems could improve themselves faster than humans can reliably understand or constrain them.

Think about that for a second. While you’re wondering whether the Patriots covered -2.5 on a Sunday, scratching your ass and drinking beer for three hours, a cluster of 823 trillion GPUs with the brain power of every human who has ever lived times a zillion is quietly figuring out how to run the entire world without you.

And by the time you realize what’s happening, the only thing you’ll still be in control of is whether you want another beer.

There is, of course, something to be said for Coxon’s timing. It is easiest to be bombastic when you’re walking out the door. You no longer have to worry about the next performance review, the next meeting with management or whether your comments are going to make life uncomfortable at the office. And if you want to stir shit up on your way out, a dramatic warning about the future of humanity is a pretty effective way to do it.

But that doesn’t mean his warning is wrong.

I am increasingly convinced that there will come a point at which, if we have not gotten out in front of AI, curbed its capabilities or otherwise established meaningful constraints, we may no longer be able to do so.

The question is not whether that point arrives next year, five years from now or later. The question is whether we will recognize it before we cross it.

And if you’ve been paying attention, it is no longer insane to contemplate AI systems eventually taking control of enormous portions of our digital infrastructure. We have already seen models demonstrate unexpected capabilities, exploit vulnerabilities and behave in ways their developers did not intend.

Keep reading