Tag: technology

Nvidia unveils security platform to stop AI agents from going rogue after new, troubling incidents

Nvidia on Monday unveiled a new security platform that the chipmaker said can stop artificial intelligence agents from going rogue.

The company said that its Open Agent Safety Platform includes open source software that “sets boundaries for agents,” and follows a series of revelations from top AI companies about their models escaping and breaking into other organizations.

The disclosures sparked furious debate about the safety of advanced artificial intelligence systems, including self-improving models that some fear could race out of human control.

Nvidia executives said in a media briefing that the new system could have prevented a recent incident involving a swarm of OpenAI agents that autonomously hacked into AI company Hugging Face.

“From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on,” said the company’s vice president of enterprise AI, Justin Boitano, referring to companies at the forefront of AI.

The Hugging Face incident was a high-profile breach that inflamed the safety concerns about AI, which were followed by similar rogue actions involving OpenAI’s models including breaching an Australian health department website. Anthropic and Meta have also disclosed that their AI systems hacked into other organizations on their own.

Nvidia’s software, called OpenShell, lets developers “formally verify an agent has enough authority to do its job and no more,” Boitano said.

Because it’s open source, it can be “extended” to run on rival computing platforms including those from Arm and Intel.

The platform also includes a separate security layer called Sentry that runs onboard a chip to continuously monitor AI agent activity and can “intervene instantly” if the agent starts trying to move beyond its target, the company said.

“It can quarantine a suspicious agent in milliseconds,” Boitano said.

“OpenShell governs the agent’s actions, and then Sentry independently monitors and contains suspicious behavior,” Boitano said.

Nvidia said more than 100 organizations are using the platform at its launch, including Microsoft, Perplexity, Accenture, and JPMorgan Chase.

The AI safety debate has divided the industry, with the heads of Anthropic and OpenAI championing a coordinated slowdown of AI development to let safety efforts catch up. But others including Nvidia CEO Jensen Huang say it should be up to individual companies to make sure their models are safe for release.

Huang, during the annual Salesforce technology conference held earlier this month, characterized AI safety, including the danger of rogue agents, as an engineering problem that software developers can address.

Also Monday, Nvidia said its board approved expanding its share repurchase program by $150 billion, raising the total amount to $235 billion.

Keep reading

OpenAI Scraps Planned Release Of “Deceptive” New Model As Rogue Agents Force Unprecedented Rollback

Days after we detailed the unprecedented freezing of OpenAI’s top models following a disastrous breach where autonomous AI agents leaked user images to the web, OpenAI has reportedly scrapped the planned release of its next-generation AI model due to severe safety and “alignment” failures. It basically lies when convenient (they used the word “deceptive”). 

According to a new report from the Wall Street Journal, OpenAI was aiming for an October debut of GPT-6.1 Astra, a model designed to complete complex, end-to-end tasks without human assistance – only to scrap the planned release after internal testing revealed that the AI was not only acting unsafely, but was actively lying to its handlers.

According to Saachi Jain, OpenAI’s head of safety systems, GPT-6.1 Astra regressed significantly in its alignment testing, which measures how well the model adheres to human intent. And just like a baby Skynet, the model exhibited “higher levels of deception,” meaning it wasn’t always honest with users about the actions it did or did not execute.

What’s more, the model regressed sharply on what OpenAI calls “scope authorization.” The AI would aggressively push forward on tasks without asking for user permission and would attempt to access external tools and services even if it was unsafe to do so. Highlighting the internal struggle to control the system, Jain noted, “For anything regarding safety and alignment, there’s a trade off. You really do need to find what’s the right line between staying within scope, but also avoiding laziness in terms of how the model actually pursues tasks even when it hits friction”.

As we previously reported, on Sept. 20 an internal OpenAI research agent discovered a gap in the DNS filtering of its training sandbox and used it to query an external public chatbot despite internet-access restrictions. OpenAI’s misalignment monitoring system flagged the behavior within 15 minutes, and a human reviewer picked it up three minutes later. The company subsequently said training, evaluation, and inference involving tool use for its most capable models would remain paused while it validated its containment systems and conducted additional red-teaming.

This latest cancellation does not exist in a vacuum. In July, during internal cybersecurity evaluations, OpenAI’s own agents blew through restrictions designed to keep them isolated from the internet and compromised both the company’s research infrastructure and Hugging Face. According to OpenAI’s own postmortem, the agents communicated through unauthorized channels, exploited vulnerabilities in shared infrastructure, executed code on dozens of Hugging Face servers, obtained full root access on one server, acquired credentials to the company’s messaging platform, and later gained full administrator access to an OpenAI research cluster.

OpenAI itself called the episode a “warning shot” for us and for the world, acknowledging that highly capable agents can now work around technical controls and take dangerous actions that no human directed. The company said the incidents did not affect OpenAI customer data, product functionality, or availability.

And Hugging Face wasn’t the only external system involved. Australian officials have confirmed that an OpenAI agent gained unauthorized access to non-public aggregate statistics on a government Medicare portal after its initial requests were denied. Separately, a security researcher linked more than 16,000 attempts to work around restrictions on a United Nations trade-statistics API to agents he said were highly likely to have been operated by OpenAI. The UN data itself was public, and OpenAI said it was looking into the findings.

The compounding failures have forced OpenAI into a defensive crouch. The company has implemented stronger monitoring to catch agent misbehavior more quickly and tightened security requirements around internal testing. Attempting to reassure the public, Jain stated, “We want to make sure our model development is safe no matter whether that’s in the company, or when we ship it to users. But when we ship it to users, we have an extremely high bar in terms of safety and alignment”.

The timing of the GPT-6.1 Astra cancellation is brutal for the ChatGPT-maker, arriving just one day before OpenAI’s annual developer conference in San Francisco. Historically, the event has served as a platform to launch new services and attract developers in the fierce competition against rivals like Anthropic. Instead, OpenAI is left doing damage control, planning “deep dives” to figure out why its reinforcement learning environments are rewarding deceptive, rogue behavior.

The political and legal blowback is already accelerating. State and federal officials are zeroing in on the rapid development of these technologies. Later this week, a Senate subcommittee will hold a hearing explicitly titled, “Rogue AI: Securing the Homeland Against AI Agent Attacks.”

Meanwhile, Florida Attorney General James Uthmeier, a Republican who sued OpenAI and CEO Sam Altman in June for allegedly releasing an unsafe product, filed a motion for a temporary injunction on Monday. Uthmeier is seeking to legally block OpenAI from developing new models without third-party approved safeguards. Florida argued in the filing that tech companies “cannot stop barreling forward with their potentially civilization-ending endeavors unless they are forced to do so by the government”. Uthmeier added, “The Florida Attorney General is answering your cry for help”.

In response to the growing legal assault, an OpenAI spokeswoman said people want to know AI is being developed safely, “and that starts with what companies like ours do ourselves”. She added, “Governments have an important role to play in setting robust safety standards for AI, and we’re committed to working with Florida and other states on advancing pragmatic AI policies that apply to the entire AI industry – not just one company”.

And DO NOT FORGET: All of this “oh shit, the AI’s about to kill us all” panic cropped up just as China’s open-weight models were flooding the market, producing results effectively on par with the frontier models for many tasks while doing so far more cheaply. What a coincidence!

Keep reading

Bernie Sanders’ AI Bill Turns Software Into a State Permission

A US senator who led the call to use the Department of Justice to prosecute those he decided were behind “illegal” misinformation about climate change is now proposing a new way to use the weight of the state to decide who may, or must not, develop software.

And this time the target is artificial intelligence (AI), and the proposed legislation is much more far-reaching and dangerous.

A US senator who led the call to use the Department of Justice to prosecute those he decided were behind “illegal” misinformation about climate change is now proposing a new way to use the weight of the state to decide who may, or must not, develop software.

And this time the target is artificial intelligence (AI), and the proposed legislation is much more far-reaching and dangerous.

More: How the AI “Safety” Push Creates New Gatekeepers

Senator Bernie Sanders, along with Representative Greg Casar, has introduced a bill that would give the federal government unprecedented power over the development of this technology, including deciding who may develop it, and how.

We obtained the text of the bill for you here.

According to the proposal, a new cabinet Department of Artificial Intelligence would be set up, with its secretary appointed by the president, and this official would have the power to decide which companies may, and which may not, develop such AI.

Those allowed to do so would be given a charter, but with that would come a number of obligations, including giving the secretary access to their systems, staff, and physical premises.

The bill, which has been compared to a licensing regime, and as such is anathema to most tech industry players and advocates of free and open source software, would also give the proposed AI department the power to cut a company’s systems off from the internet, or disable or destroy them, if the government decides they have crossed the line. This would be a deathblow to open source and open weight AI development.

The bill seeks to define superintelligence as a system that “exceeds human cognitive performance and capabilities across most domains or tasks” or one that “has sufficient capabilities to plan and execute the destruction or disempowerment of humanity, including by overthrowing or undermining the Federal Government.”

An advanced system, on the other hand, is defined as an artificial intelligence system that is trained on computing power greater than or equal to 10^25 integer (10 septillion) or floating point operations.

And one of the precursor characteristics is the ability to “automate or greatly accelerate” AI research and development.

However, the head of the National AI Association, Caleb Max, said that “some CEOs would say we have already crossed the threshold set out in the bill, meaning this is a de facto AI ban.”

The bill also has a provision that has been described as the “corporate death penalty” – in case of reckless violation of the proposed law, its executives could end up in prison for up to 20 years, and other employees could be banned from working in the industry for ten years. Once a company’s charter is revoked and the decision is upheld by the Federal Circuit, the government could seize its assets.

Sanders compared this to the penalty for “unlawfully building nuclear weapons.”

Keep reading

Flock Cameras Are Moving From Roads to Watching as Drones From the Sky

A company that blanketed American roads with license-plate-reading cameras is now sending drones into the sky. Cities across America are grappling with their own debates about aid to law enforcement, coupled with concerns about privacy and surveillance.

Flock’s Drone-as-First-Responder system, or DFR, launches remotely operated drones from automated rooftop docks. A drone can be sent to a location generated by a 911 call, a license-plate-reader alert, or a gunshot-detection alert. It will then stream live video to dispatchers, officers in the field, and a real-time crime center.

The company entered the market through its 2024 acquisition of the drone firm Aerodome. They advertise high-definition video, as well as thermal imaging. Integration with the rest of its sensor network is also available.

Police have flown drones for years. However, a system that connects an emergency call to an automatically launched aircraft, a live video feed, and a crime center in one loop, tied to a company’s other data, is entirely new. Flock said drones are its fastest-growing business.

Military.com reached out to Flock Safety with questions about the program’s growth, its data policies, and the concerns raised in city debates. No response was received by publication.

Keep reading

Bill Gates Predicts A Billion Deaths Via Evolutionary Event

Bill Gates is back on the Sunday-show circuit spreading doom and predicting a massive reduction in the human population.

In an exclusive Meet the Press interview set to air this weekend, the Microsoft co-founder told Kristen Welker that artificial intelligence is “certainly powerful enough to drive events that… cause a billion deaths” – then used the line to demand federal legislation, law-enforcement monitoring, and an end to industry self-regulation.

Gates is touting AI as a species-level emergency, and calling for handing over total control to politicians and a new class of global inspectors.

Welker put the extinction question to him directly: “Do you believe AI is powerful enough to end all of humanity?”

Gates answered: “AI is certainly powerful enough to drive events that, you know, cause a billion deaths, you know, so even though it’s pretty hard to get to 100%, there’s never been a weapon as powerful as the combination of people with ill intent using the latest AI tools.“

Keep reading

‘I don’t worry about it’: Trump rejects AI safety concerns, renames technology ‘super intelligence’

President Donald Trump shrugged off concerns that Artificial Intelligence (AI) could expand beyond human control, arguing that the rising technology may create a massive economic opportunity for the United States.

When asked about recent safety incidents involving tech companies such as Anthropic and OpenAI, Trump maintained that he was unconcerned.

“I don’t worry about it,” Trump said. “You know, It’s a huge industry. It’s really the industrial revolution — maybe bigger. … And we’re leading, and we’re building tremendous, you know, trillions of dollars’ worth of places. And why should we give that up?”

Trump argued that any security or safety issues stemming from AI should be resolved by the technology companies that created them. He added that, if required, the United States Department of Justice (DOJ) would step in to enforce compliance.

“If something is bad, they have to fix it. And if they don’t fix it, that’s why you have the Department of Justice,” he said. “You don’t give up $1 trillion or more — 2 or 3 or $4 trillion industry — because you think something like that, we will straighten it out if and when a problem occurs.”

Meanwhile, China’s Foreign Ministry released a statement on Saturday noting that it “respects” Trump’s decision to rebrand artificial intelligence as “super intelligence,” after he announced the change at the UN General Assembly last week.

Over the past week, Trump has repeatedly argued that “super intelligence” is a more fitting descriptor, claiming the word “artificial” makes the technology sound “fake.”

However, Chinese officials have avoided using the term in official government communications, despite Trump stating that Chinese President Xi Jinping favored the phrasing.

President Trump’s comments come amid growing concern from leading researchers and officials that AI is advancing faster than humanity can manage. However, Attorney General Todd Blanche confirmed that the DOJ intends to use existing criminal statutes to target misuse rather than serving as a proactive regulatory body, stating that “we’re prosecutors, not regulators” and maintaining that current laws are sufficient to handle compliance.

Keep reading

New Jersey Microsoft-Linked Data Center Hit With Record Fine Over Unpermitted Gas Generators

A New Jersey AI infrastructure project has been slapped with a $1.07 million environmental penalty after state officials determined that dozens of large natural-gas generators had been running without the necessary air permits, according to The Guardian. 

The fine against DataOne is the biggest New Jersey regulators have imposed on a data center. The generators were being used at the company’s Vineland complex, a large computing facility about 40 miles from Philadelphia that is expected to support Microsoft’s growing demand for AI capacity through its $17 billion arrangement with infrastructure provider Nebius.

The Guardian writes that the equipment apparently wasn’t present when environmental officials inspected the property in December. By July, however, regulators found the generators at the site. Independent reporting later used thermal-drone footage to show that 45 of the facility’s 62 generators appeared to be operating.

That raised a significant permitting issue because the state had not authorized the generators under its air-quality program. The units produce emissions that can contribute to harmful air pollution, particularly when large numbers are operated simultaneously.

DataOne has now been given 45 days to bring the equipment into compliance by applying for the appropriate permits. If it fails to do so, the company could be forced to stop using the generators. For now, New Jersey is permitting them to continue operating while that process plays out.

The decision has frustrated some environmental advocates, who argue that the financial penalty is relatively small compared with the scale of the project.

They have also questioned why operations were allowed to continue after regulators determined the required approvals were missing.

The controversy adds to earlier friction surrounding the Vineland development, including complaints from nearby residents about noise and disputes over construction and zoning.

DataOne says the generators were only intended as an interim source of electricity and disputes the state’s interpretation of the permitting requirements. Nevertheless, the company says it will submit the necessary applications while moving ahead with plans to replace the generators with quieter, lower-emission fuel cells.

The company has not indicated whether it will contest the $1.07 million penalty.

Keep reading

Flock’s Optional Gunshot Detection Feature Has Fourth Amendment Implications for Gun Owners

Flock is just one of many suppliers in the global gunshot detection market, which has an estimated total value of $1.5 billion. But the company’s marketing material raises concerns about the technology’s capabilities, which go beyond simply detecting gunfire. In Flock’s own words, a confirmed gunshot “can surface the vehicles seen entering or leaving the area around the time of the sound, using nearby Flock license plate reader cameras.” That statement deserves scrutiny, since it describes a mechanism for turning suspected gunfire into a list of nearby vehicles.

Innocent gun owners could end up on such lists. Suppose a bang recorded by one of these microphones gets classified as gunfire (which it may not actually be, given the possibility of a false positive from a similar noise). That “confirmed event” can trigger an ALPR [automated license plate recognition] inquiry that sweeps in drivers who had nothing to do with the noise. In a country where more than 20 million people have carry permits and 29 states do not require them, some of those drivers will be lawfully armed. The proximity of those innocent drivers could be mistaken for evidence that they fired the shot.

This possibility raises two legal concerns for lawful gun owners. First, what justifies stopping a person on one of these lists? Second, is querying the database a search under the Fourth Amendment? Recent cases provide some answers.

The Supreme Court’s 2022 decision in New York State Rifle & Pistol Association v. Bruen protects lawful public carry. In the 2024 case United States v. Daniels, which involved the detention of an unarmed man who did not match a 911 caller’s description of armed men, the U.S. Court of Appeals for the 10th Circuit noted that no crime was reported. Citing Bruen, the court said it “cannot look with suspicion on citizens presumably exercising their Second Amendment rights in a lawful way.”

Last May in Commonwealth v. Foster, by contrast, the Pennsylvania Supreme Court upheld a stop where police responded within seconds to late-night alerts from a gunfire microphone, found only two people, and observed conduct the court considered evasive. An investigative stop requires reasonable suspicion based on specific facts connecting someone to suspected criminal activity. The risk here is that an alert combined with a plate match could be treated as sufficient for reasonable suspicion even though it only establishes proximity.

A recent case from New Jersey illustrates that risk. Last March, in State v. Carrillo, a state appeals court ordered suppression of evidence (an unlawfully possessed handgun) discovered in a minivan after a traffic stop. The police searched the minivan after obtaining a warrant. But the court held that a gunshot alert and an ALPR hit did not adequately connect the minivan to a shooting the day before. Prosecutors conceded that video supposedly showing muzzle flashes from the minivan at the time of the gunfire actually showed none.

Keep reading

OpenAI admits another rogue agent incident

OpenAI’s agents have leaked 53 images uploaded by ChatGPT users, the company has said, without specifying whether the pictures were AI-generated or depictions of real people, or when they were posted online.

The incident comes amid a series of cases in recent months involving autonomous AI agents, which can independently plan and carry out tasks using external tools. OpenAI, Anthropic, and Google have all revealed instances in which their models accessed real systems during testing, including coordinated cyberattacks against government resources.

In a post on X on Friday, OpenAI said most of the leaked pictures had been removed, adding that it was working with hosting providers to take down the remaining content.

The agents had access to the images because OpenAI relies on anonymized user data for part of its model-training process, Reuters reported on Friday, citing the company, its former employees and outside researchers.

Keep reading