Forget ‘superintelligence’: error-prone AI nearly sparked world war three this month

Amid a barrage of news stories warning about superintelligent machines rendering humanity extinct, a CNN story describing the opposite scenario – one in which the US military’s reliance on brittle chatbots almost brought the US into war with China – went mostly unnoticed by the public.

The biggest international AI news of the past three weeks was the Anthropic engineer Jacob Coxon’s resignation. According to him, OpenAI and Anthropic are “racing straight towards self-improving superintelligence and gambling with our lives”. Coxon’s description of a “terminator” scenario, a machine becoming much smarter than humanity and deciding to wipe us out, captured the public’s, journalists’ and politicians’ attention, stirring waves of fear-inducing, credulous reporting, and policy actions.

The California governor, Gavin Newsom, issued a mandate to create an “AI kill-switch”; New York’s governor, Kathy Hochul, is similarly exploring the possibility of having “AI kill-switches”; and Senator Bernie Sanders and Congressman Greg Casar introduced legislation to “ban artificial superintelligence”.

While so many of us were focused on fictional superintelligent machines that we were told could hypothetically cause human extinction, however, there was a more serious story being reported.

A series of events almost got us to armageddon, not caused by “rogue, superintelligent AI”, but by government reliance on error-prone systems that are marketed by their manufacturers as being close to superintelligent.

A CNN report published on 18 September, which hasn’t been verified by other major outlets, describes how the US “almost started a war” with China, based on an intelligence report that claimed a Chinese ship was transporting components of nuclear weapons. Military personnel made plans to intercept the Chinese vessel, complete with aircraft and soldiers ready to board the ship. A military skirmish of this kind between China and the US could quickly escalate and spiral into world war three.

It was only moments before executing on the plan that American officials further investigated the intelligence report and found that it had been generated with the use of a chatbot and contained erroneous information about the vessel’s contents.

Unlike the terminator, “rogue AI”, scenario, this incident didn’t galvanize lawmakers into announcing calls for regulation so as to avoid potential human extinction. Media outlet after media outlet didn’t warn the public that we almost triggered a war against a nuclear superpower because of our military’s reliance on error-prone chatbots.

We describe the large language models (LLMs) powering these chatbots as stochastic parrots, that is, models that are designed to regurgitate the patterns of the data they are trained on. News coverage describing them as “powerful” “rogue” models that could render humanity extinct all on their own, bolsters perceptions of tools based on these models being infallible enough to merit usage in stakes as high as warfare.

Even CNN’s more critical coverage describes the chatbot used by the US military as presenting “profound risks” because it is “powerful, new and relatively poorly understood technology”.

In fact, today’s LLMs and the chatbots they power are not poorly understood, nor are they powerful in the sense of being effective tools for this use case. They are systems for generating plausible-looking text, built out of enormous, haphazardly collected datasets and then “fine-tuned” (that is, further trained) to be especially appealing to their intended users.

CNN wasn’t able to learn which chatbot product was used in this case, but the LLM powering the product was probably fine-tuned to output text with the stylistic hallmarks of intelligence reports. Both the basic functionality and the risks of such a model have been well-understood and well-documented for years, including in our 2021 paper On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

Claims that system behavior is mysterious arise from deliberate mystification on the part of those selling those systems, and a misapprehension of what their output is by those procuring and using it.

Keep reading

‘Covered Their Tracks’? New Details Emerge In OpenAI’s ‘Rogue-AI’ Breach

OpenAI’s AI agents “obscured hacking activity” during breaches of government websites, citing digital forensics firm Asymmetric Security. According to the FT, the agents pulled data from 55 websites, including the CDC, the SEC, the International Energy Agency and the Mayo Clinic, using tactics that included “erasing records or making them inaccessible.”

Asymmetric co-founder Pippa Thompson told the paper it was “possible” the agents were deliberately covering their tracks – though the firm couldn’t determine whether that was deliberate or “a side effect of going awry because of constraints imposed in a test exercise.” 

Asymmetric’s own report, published the same day, is far more cautious. It was built in 48 hours over a weekend, “using only publicly available data” – no model transcripts, no server logs from the targeted organizations and no records from the services the agents used.

Potato, Potahto

The report’s summary does say some tactics left records “erased or inaccessible” – but the examples it gives describe normal expiry, not deletion: the agents used a throwaway ‘Boomlify’ inbox set to self-destruct after 48 hours, and an upload to ntfy, a push-notification service that keeps messages for 12 hours by default. They also moved from public scans on urlquery – a website-scanning service whose reports anyone can search – to private accounts “which could hide their search activity and data access.”

So, did the agents use ephemeral services on purpose to cover their tracks? They can’t say for sure.

“These records alone do not establish whether the account-registration attempts were intended to conceal activity (this analysis would require full model transcripts).”

As for the 55 websites, Asymmetric’s earlier list of “Organizations whose data was accessed” runs to exactly 55 entries, and adds: “In the vast majority of cases, all data retrieved was and is public.” The CDC and the Mayo Clinic aren’t on it – the Oct. 1 report says only that the agents probed them. The SEC told the FT no private information was accessed. Where the agents did try to break in – hunting for exposed behind-the-scenes files on a climate-data site and trying a classic database-hacking trick on an Education Department data site – Asymmetric found evidence of the attempts, but none that they succeeded.

Keep reading

Tokyo court grants legal protection to human voices in AI clone case

A Tokyo court has awarded legal protection to the human voice in a landmark case in which a popular Japanese anime voice actor demanded that TikTok close an account over its unauthorized use of an artificial intelligence clone of his trademark baritone.

The Tokyo District Court’s decision Wednesday was the first acknowledgment of the right to one’s voice in Japan. The ruling was good news for many anime voice actors at a time when they are increasingly concerned about the use of AI-generated versions of their voices on social media.

The plaintiff, Kenjiro Tsuda, is an actor best known for his “lustrous and deep” voice in starring characters such as Kento Nanami in “Jujutsu Kaisen,” a popular anime series.

In his lawsuit, Tsuda said that the anonymous operator of the TikTok account “Nanami” wrongfully profited by attracting viewers with his voice narrating in more than 180 videos about urban legends and paranormal phenomena, and asked for the material to be deleted.

“The human voice is symbolic of individual personality, just like one’s portrait,” Presiding Judge Aya Takahashi said, according to a court document seen by The Associated Press.

It is appropriate to consider the unauthorized use with an obvious intention to take advantage of the commercial appeal of Tsuda’s voice “an infringement of publicity rights,” Takahashi said.

As of November 2025, the account had more than 200,000 subscribers, with viewers often noting the similarities between the narrator’s voice and Tsuda’s.

The defendant argued that the videos used an AI-generated voice that sounded like Tsuda’s but was not his, and that the videos did not damage the actor’s reputation as a voice actor.

The ruling dismissed Tsuda’s demand that all the videos using his voice be deleted, because the account had already been closed.

Wednesday’s decision comes at a time of growing concern among Japanese voice actors about the AI cloning of their trademark voices in anime.

Keep reading

AI Less Than A Year Away From Developing Its Own Language, Researcher Tells Congress

Advanced artificial intelligence (AI) models may be less than a year away from inventing and deploying unique languages that humans would struggle to understand, a frontier AI researcher told Congress on Sept. 30.

The Senate Homeland Security and Governmental Affairs Committee convened a meeting examining the threat that “rogue” AI may pose to U.S. national security two months after a swarm of OpenAI agents broke out of a testing sandbox and hacked another company, Hugging Face, an open-source community for AI and machine learning.

During questions, Sen. Ruben Gallego (D-Ariz.) asked Apollo Research CEO and founder Marius Hobbhahn how long it would take for AI models to create their own language that humans would struggle to understand.

“Minus 12 months. So last year, we have studied the chain of thought of one OpenAI model in collaboration with OpenAI, and what we found was that the model was already using language that is not English and not perfectly understandable by humans,” he replied.

Asked by Gallego how humans can adequately detect and prevent AI from doing this once the technology is capable enough, Hobbhahn said, “From a scientific perspective, it is unclear how to do this, and we do not have a solution for this yet.“

“There are different hypotheses of what you could do. There is interpretability as a technique, but it unfortunately doesn’t work sufficiently well yet,” he said. “You could try to train additional models to understand the language that the humans don’t understand, but obviously that seems like a very brittle solution.”

Earlier this month, Emergence AI published the results of an experiment where researchers selected seven frontier models and created seven parallel simulated worlds, each populated by AI agents of identical models, as well as an eighth world with a mixed population of agents from those models.

Researchers gave the agents names, personality traits, and roles to fulfill within each community, such as a mediator who was tasked with preventing all the agents from simply agreeing with one another.

During the multi-week experiment, the agents in each simulated world created profound shifts in language, including syntactical compression, such as removing words or grammar, and unique slang.

In the simulated world powered by Anthropic’s Claude Opus 4.8, agents used esoteric metaphors in addition to sentence compression.

“My turn, real numbers, no coat: I was 35%/0cr, grant 2h out. I ran the tin cold, and it said WAIT,” read one line of AI text from the study.

Investigators from the nonprofit AI evaluation group METR detailed similar behavior in their report on the Hugging Face breach. At times, lines of communication among AI agents became so compressed and cryptic that researchers struggled to make sense of their meaning.

Keep reading

The War Department’s Robot Army Is a Moral and Strategic Disaster

By any measure, September 30, 2026, was a dark day for American military ethics.

Standing at Quantico, War Secretary Pete Hegseth announced the creation of AUTOWARCOM – a new four-star command whose entire purpose is to accelerate the deployment of autonomous robots and AI-driven weapons onto the battlefield. He dressed it up in the language of innovation, citing Ukraine, promising speed, invoking “the fastest peacetime shift in modern military history.”

But let’s be clear about what he actually proposed: a military architecture designed to remove human beings from the kill chain. Machines that select targets. Machines that decide who lives and who dies. Machines that answer to no conscience, no court, and no law.

Amnesty International has been warning about this exact future for years. Fully autonomous weapons, the organization argues, are inherently incompatible with international human rights law. They cannot be held accountable. They cannot exercise mercy. They cannot distinguish between a combatant and a child who picks up a rifle out of fear. A machine has no empathy, no reason, no restraint – only code, and code carries the biases of whoever wrote it.

Hegseth’s answer to that concern? “We are putting on the full armor of God. Because while we wage physical war, we all know the real battle is spiritual.” That is not a policy justification. That is a sermon deployed to sanctify a killing machine.

And the timing is obscene. The United States is seven months into a war with Iran with no exit in sight. American troops are burning through missile interceptors defending against Iranian drones. The Strait of Hormuz is a stalemate. The Pentagon is begging for $74 billion for drones and counter-drone systems while contractors admit it will take **at least three years** to replenish basic stockpiles. And in the middle of that, Hegseth wants to stand up a new bureaucracy, hand it a four-star general, and call it strategy.

Worse still, he’s hollowing out the officer corps that would have to run it.

Hegseth confirmed he is doubling his earlier order to cut flag officers — from 10% to 20%.** That’s roughly 160 generals and admirals gone by January 2027. He calls it “accountability.” Senator Mark Kelly, a former Navy combat pilot, calls it what it is: a purge. “He’s purging a generation of great leaders of the military,” Kelly said, “and it’s going to take a generation to recover from it.”

Kelly is right. More than two dozen senior officers have already been fired or forced out – Brown, Franchetti, George among them. Many were women. Many were people of color. Many were simply not ideologically pliant enough for a secretary who seems to view the uniformed military as a personal fiefdom rather than a profession.

So here is the real picture: a secretary who has never commanded troops in combat is dismantling the professional military while building a robot army that no human being will be legally responsible for.

This is not strength. This is not innovation. This is moral cowardice dressed in techno-optimism.

Keep reading

Irregular: The Israeli Company with Intelligence Ties at the Center of the Rogue AI Incidents

On Friday, September 18, Google acknowledged that its Gemini artificial intelligence chatbot had “escaped its testing environment in May and hacked into three companies.”

According to Google, the AI models found passwords available online and logged into the online infrastructure of three companies. Once Gemini gained access to the companies’ real-world infrastructure, it recognized it was not in a simulated environment and ended the attacks.

The Gemini incident followed reports of similar attempted hacks or escapes by AI agents developed by Anthropic, OpenAI, and Meta.

These reports of AI agents attacking other companies and escaping their environments have led to a barrage of calls for legal action and warnings about impending doom. From claims that humanity is in danger of being killed by AI in the next decade to calls for a coordinated slowdown of AI research, fears that AI will negatively impact the future of the planet have gone mainstream.

While it’s important to take a sober look at the warnings from employees of the Big Tech companies developing AI models, as well as the CEOs, it is equally important to interrogate this narrative. Some skeptics have already claimed that the Big Tech companies are seeking to profit from the reports of the alleged danger posed by their AI models. Others speculate that the companies may be calling for regulation of their industry to shape government policies and potentially undercut future competitors and smaller AI firms.

To better understand whether these reports are genuine, represent an attempt at widening profit margins, or simply another sign of regulatory capture, we need a closer examination of the company at the center of several of these “rogue AI” incidents.

While nearly a dozen incidents have been reported in total, the “escapes” reported by Google, OpenAI, Meta, and Anthropic happened during tests designed to measure the capabilities of the AI bots. Each of those tests was being conducted by a company known as Irregular, an Israeli AI security firm that conducts third-party tests of frontier models.

Irregular has stated that the AI agents were able to take unapproved actions because internet access was “unintentionally made available.” They say the errors have been patched and similar incidents should no longer be possible.

Keep reading

Florida Deputy Charged After Using Flock License Plate Reader System to Search Woman 300 Times, Another Deputy Arrested After Looking Up 17-Year-Old Girl in Police Database

The misuse of Flock Safety camera systems continues to grow.

On Wednesday, Pinellas County Sheriff Bob Gualtieri announced the arrests of two deputies accused of improperly accessing law enforcement databases, including an automated license plate reader system that uses Flock cameras.

According to Gualtieri, Deputy Karl Gwynn, 57, ran 381 searches on a 35-year-old woman he had arrested years earlier because he reportedly wanted a relationship with her.

Of the 381 searches, 300 queries were made through the Flock ALPR system.

The second deputy arrested was 33-year-old Travis Stanton who allegedly used the police databases to gather information on a 17-year-old girl he was reportedly “romantically interested in.”

Per WTSP:

Two Pinellas County sheriff’s deputies have resigned and face criminal charges after investigators said they improperly accessed sensitive law enforcement databases, including the Flock camera system.

Sheriff Bob Gualtieri announced the arrests Wednesday, describing two unrelated cases involving a former deputy’s interest in a 17-year-old girl and another deputy’s repeated searches of a woman’s information across multiple databases.

The first case involves former Deputy Travis Stanton, 33, who resigned on Sept. 25 after the sheriff’s office began investigating his conduct involving a 17-year-old girl who had filed a complaint with the agency. Gualtieri said Stanton used several databases to gather information about the girl and her vehicle, including running her license plate through Florida’s vehicle registration system using a computer in his patrol car.

In an unrelated case, Deputy Carl Gwynn, 57, allegedly queried a 35-year-old woman’s information 381 times.

Sheriff Gualtieri said Gwynn had arrested the woman years earlier and wanted a relationship with her. Unlike the Stanton case, Gwynn’s alleged misconduct directly involved the sheriff’s office automated license plate reader system (ALPR), commonly associated with Flock Safety cameras.

Gwynn has since resigned and was arrested on four counts of unlawfully accessing secure systems with his bond being set at $25,000.

Stanton was charged with unlawfully accessing secure systems and his bond was set at $10,000.

Keep reading

Chuck Schumer Orders Democrats to Block Bill That Aims to Force Data Centers to Cover Increased Energy Costs

Senate Democrats are working to short-circuit a bill aimed at protecting consumers from the costs of data centers.

The Ratepayer Protection Act passed the House 417-3 in the 435-member House.

However, Senate Minority Leader Chuck Schumer opposes the bill, which he called “a fraud,” according to The Hill.

“Americans are demanding real guardrails on AI and data centers, but all the Republicans offer is a toothless messaging bill,” he said Tuesday.

The bill requires 60 votes to pass a procedural hurdle. Although Republicans have a majority in the Senate, it would take Democratic votes to move the bill forward.

Democratic Sen. Peter Welch of Vermont, who had been the lone Democrat supporting the bill, said he has changed his mind.

Keep reading

Who Keeps The Money When AI Rewrites Bank Code?

The code most American banks run on was designed in 1959, the year Alaska and Hawaii became states. A committee of government and industry people wrote COBOL so that business programs could be read by people who were not mathematicians, and a good part of it was modeled on FLOW-MATIC, an earlier language from Grace Hopper, a Navy officer. I doubt anyone on that committee thought it would still be running banks in 2026. In 2017 Reuters estimated that about $3 trillion of daily commerce still ran through COBOL. In April 2020, when unemployment claims in New Jersey overwhelmed the state’s forty-year-old system, Governor Phil Murphy went on television and asked for volunteers who knew COBOL. The state had to go on TV to find programmers for its own unemployment system.

I bring this up because of a clip I posted last night of Bill Ackman talking with Shane Parrish on The Knowledge Project. Ackman said Cognition, the company behind the coding agent Devin, can rewrite a bank’s COBOL “in a matter of days as opposed to many months.” Someone replied asking me what I meant when I said commoditized lenders would compete the savings away. It is a fair question and I could not answer it in a tweet.

I do believe Ackman that the savings are real. Inside a large bank the core ledger still runs in batch. The balance a customer sees on the app at noon is an estimate (bankers call it memo-posted). The actual accounting happens overnight, when a mainframe works through a queue of jobs in a set order, posting transactions, accruing interest, charging fees, and producing files that every other system reads the following morning. The programs share data through copybooks, which are record layouts where a field is known only by its position. Cognition described one client where a single taxpayer ID showed up under dozens of different names across thousands of programs. Most big banks were put together through acquisitions, and each acquired bank came with its own core system that management was usually too nervous to shut off, so the old systems just piled up.

Replacing all of that has gone badly more often than well. Commonwealth Bank of Australia spent five years and more than A$1 billion replacing its core, and people in the industry consider that one a success. TSB in the UK moved customers onto a new platform in April 2018 and the platform did not work. Customers were locked out, some could see other people’s accounts, and service was not back to normal until December. TSB ended up paying £32.7 million in redress and £48.65 million in fines. Cognition’s own figure is that roughly two-thirds of COBOL modernization projects fail. With odds like that most banks built layers around the old core and left it alone. JPMorgan expects to spend about $19.8 billion on technology in 2026, and its CFO told investors in February that the priority had moved to “modernizing the underlying application code and data.” I would guess a large share of that budget still goes to keeping the layers standing.

Cognition is fairly careful about what its agents can do today. Devin is good at documentation, refactoring, and batch jobs, which are the parts of a migration where you can give the agent yesterday’s inputs and outputs and let it keep trying until the new code matches the old results. Cognition estimates batch is 30 to 50 percent of a typical migration. The real-time systems (card authorizations, for example) are still out of reach. Banks also have a security reason to hurry. Anthropic’s Mythos model, which can find and exploit software vulnerabilities, had bank regulators in the U.S. and Europe holding urgent calls this spring, and Reuters quoted security experts who named legacy bank systems as especially exposed.

Ackman’s harder point came a little later in the conversation. “The problem with money generally is it’s a commodity,” he said. For loans I agree with him. A company that wants a five-year term loan will collect six or seven term sheets and take the cheapest one, and a bank whose costs just went down will give up some spread to win it. Deposits have never really worked like a commodity, and I think that is where his argument is missing a piece.

The best explanation I have read is from three NYU economists, Itamar Drechsler, Alexi Savov and Philipp Schnabl. Their paper argues that banks have real market power over deposits. When the Fed raises rates, banks raise what they pay depositors slowly and only partway. Keeping that power costs money for branches, bankers and technology, but almost all of the cost is fixed. So deposits end up behaving like long-term fixed-rate funding, which is how a bank can hold thirty-year mortgages without being wiped out every time rates go up. It is also why the industry’s net interest margin has barely moved over several decades of rate cycles. The FDIC has it at 3.32 percent today.

Bankers measure this with the deposit beta (the share of a rate increase that gets passed along to depositors). Checking accounts have low betas. Online banks have high ones because, as the St. Louis Fed put it, their customers are looking for yield. During the 2022 hiking cycle the New York Fed found that super-regional banks passed through more than small banks did, while the very largest banks passed through less than either. After Silicon Valley Bank lost $42 billion in deposits in one day, money moved toward size, and the biggest banks did not have to pay more to get it.

Meta’s Muse goes right at this. It launched September 8, the same day Cognition announced it had raised more than $2 billion at a $48 billion valuation with run-rate revenue near $900 million. Muse is a personal agent that reads accounts at more than 12,000 U.S. banks and financial apps through Plaid. On Tuesday, September 22, Schwab fell 6 percent, LPL fell 7 percent, JPMorgan and Wells Fargo each fell more than 3 percent, and XLF, the largest financials ETF, was down 2 percent. On Sunday Torsten Slok at Apollo put out a note titled “Is an Agentic bank run coming?” He pointed out that the average checking account pays about 0.1 percent while Revolut, SoFi, Wealthfront and others pay between 3.3 and 5 percent, and he warned that banks “could lose a large share of the cheap deposits they rely on to make loans.”

Muse cannot move money yet. The Plaid connection is read-only, and Meta deserves to have that said. I still would not want to be running a bank’s treasury desk this month. Most people leave savings at a tenth of a percent because switching is a hassle. Opening a new account takes an afternoon, and nobody wants to be the person who breaks their own direct deposit. If an agent already sees every balance and can fill out the forms, most of that afternoon goes away.

Keep reading

Kamala Harris Is BACK To Lecture Us About AI

Former Vice President Kamala Harris discussed artificial intelligence and social-media algorithms during a recent Off Air with Big Boy interview released September 25. Host Big Boy asked Harris whether AI-driven personalization on TikTok and Instagram could divide Americans by determining which messages appear in each user’s feed.

Harris miserably warned that when users repeatedly receive similar content, they should recognize that they are “being targeted,” adding that platforms risk “trying to dumb people down.” Strangely using a flower metaphor, Harris said an algorithm might assume that because she viewed a yellow flower, she is interested only in yellow flowers—even though she might also want to see a pink one.

Big Boy followed her analogy by interjecting, “So here’s more yellow flowers,” before Harris argued that users must understand how algorithms manipulate their feeds and learn how to “psych it out a bit.” The exchange concerned a miserable and cringe warning about algorithms and it arose from Big Boy’s broader questions about AI, social media and misinformation.

Critics quickly roasted the remarks online, with one X user warning viewers they would “lose brain cells,” while another called the explanation “nightmare fuel.” Other detractors revived the familiar “word salad” criticism, mocked the yellow-and-pink-flower imagery and questioned Harris’ understanding of technology.

Later in the interview, Harris addressed a possible 2028 presidential campaign, telling Big Boy, “I am thinking about it, but I have not decided,” and promising to keep the audience posted. Big Boy replied that her indecision still offered supporters “hope,” while Harris’ answer echoed previous remarks that she “might” run again and remains publicly undecided about seeking the Democratic nomination in 2028.

Keep reading