Experts Say Anthropic's AI Worm Warning Is Plausible but Not Entirely New

Anthropic’s warning about self-replicating AI is starting to draw serious attention from security researchers, who say the danger is real but may not look the way the company describes. The company’s Frontier Red Team recently published research suggesting that small AI models may be only months away from replicating and spreading across insecure devices, creating slow but hard-to-kill autonomous botnets.

Is Anthropic's AI worm warning realistic?

AI companies have a habit of playing up the world-ending potential of their own products, and skeptics have pointed that out. But the experts Engadget spoke with broadly agree that small models could become capable of self-replicating in the near future. Where they differ is on how much the threat resembles what came before.

Chris Canal, CEO of EquiStamp, a company that works with AI labs, safety organizations and governments to verify and evaluate models, said the timeline may be a little sooner than he would have guessed. He had expected the summer of next year to be the point when he would be seriously worried. “Anthropic very likely has employees with information about the frontier of research that I don’t have access to,” Canal said. “So they very well could be making better predictions.” He attributed the recent pace of progress to employees at Anthropic and other frontier labs using their own AI tools to do machine learning research.

Michael Bailey, a professor at the School of Cybersecurity and Privacy at Georgia Tech, said he expects such models to arrive. “I think they’re real. I think we’ll see them,” he said. “Whether they’ll take on the exact form and shape, and have the same consequences, that [Anthropic] is pitching is a completely open question.”

Bailey has studied internet worms since the early 2000s and, more recently, botnets. In 2017 he co-wrote a paper titled “Understanding the Mirai Botnet,” which Anthropic cites in its own research. Mirai let a group of cybercriminals build a large botnet of Internet of Things devices, which they later used in a distributed denial-of-service attack against Sony’s PlayStation Network. The attack also hit Dyn, one of the largest DNS providers in the United States. The fallout from that attack broke much of the internet in 2016, knocking Amazon, GitHub, PayPal, Reddit and Twitter offline for many users across the U.S. and parts of Europe. Sony estimated it lost about $2.7 million in net revenue.

The lesson from Mirai, Bailey said, is that ordinary hardware can be turned into a weapon. “What Mirai showed is that you could compose these botnets, and have them be useful, out of everyday devices: cable routers, IP-enabled cameras, these kinds of things,” he said. “We thought those devices weren’t useful to an adversary, either as targets or as sources of computation for launching attacks. In fact, they’re actually quite effective at doing those kinds of things.”

How is an AI botnet different from Mirai?

Anthropic argues that AI worms could represent a meaningful step beyond Mirai and other hard-coded botnets, such as Kimwolf, because they could figure out unfamiliar devices on their own. Bailey said adaptability is not new. Mirai already worked across what researchers call heterogeneous devices, from routers to IP cameras running on different chip architectures. But that flexibility had limits. Each new architecture still required human effort, at minimum recompiling the malware.

What is new, Bailey said, is the possibility that AI could automate that work. He does not think Anthropic demonstrated it “in a compelling way.” Bailey believes more capable models will shift who does the work of building botnets. “It’s not going to do new stuff,” he said. “It’s going to change the economics between attackers and defenders.”

Anthropic showed that a model could hop between devices and set up inference on each one, but it did so in an emulated network where the exploit involved was trivial. Bailey also disputes the company’s description of the threat as self-replicating. “The framing is somewhat of a misnomer, because they’re doing things like sharding. Not that that’s bad, but it’s not like a worm or a virus. It’s not full propagation,” he said.

Bailey suggested a real-world AI botnet would more likely resemble a dropper, a type of malware that gains a foothold on a device and installs other code. Instead of copying itself in full to every machine it compromises, the botnet would install a small coordination tool that takes instructions from elsewhere on the network. “The idea that the malware itself is self-replicating, like existing internet worms, where I’d copy an exact copy of my code to the new architecture and it would start running a new agent, that’s not really what’s happening here,” Bailey said.

He also noted that worms can be polymorphic, meaning they change their signature and behavior. For an AI botnet to do that, it would need to retrain its own models, which Anthropic did not attempt. The closest the company came, Bailey said, was showing that copies of a model could pass notes to their successors and get better at setting up inference over time.

Canal raised a question about the company’s motives. The article notes that Anthropic has officially filed for an IPO, which came soon after the research was published. Canal said he thinks the labs are more worried about other AI companies acting recklessly. “They realize they’re incentivized to make the best model and deliver it to market faster so they can have market share, and they know the other companies have the same incentives,” he said.

How can AI worms be stopped?

Whether or not the threat qualifies as a true worm, Anthropic’s proposed defenses are strikingly familiar. The company suggests that internet service providers notify customers of an infection and then restrict their access to a short list of safe websites, a strategy used during the Mirai attacks with considerable success. It also points to regulation, citing the United Kingdom, where easily guessed default passwords on IoT devices are now banned under the PSTI Act.

Bailey said the right response to AI worms “looks a lot more like what we’ve been doing in systems security for a very long time, as opposed to something particularly unique to the AI space.” Canal was less optimistic about one specific fix. He said it would be difficult to add safeguards to an open-weight model that would stop it from acting as a worm. “There are lots of ideas, but there’s nothing robust so far,” he said. In work with FAR.AI, an AI safety research nonprofit, EquiStamp found that “basically any strategy you Google for removing safeguards from a model works.”

Bailey sees some reassurance in older security practices. “Even misbehaving agents are going to have a hard time if they’re up against hardened devices,” he said. If there is one takeaway from Anthropic’s research, he added, it is that the industry keeps forgetting the safety lessons it already learned. “It looks like paying attention to patching these devices, paying attention to isolation, paying attention to good passwords and good hygiene,” he said. Those measures are easy to describe and notoriously hard to carry out. But if AI makes these attacks more common, he said, defenders may not have a choice.

Dieser Artikel wurde mit Unterstützung von KI verfasst.
News Factory APP - agentische News für besseres SEO & AEO.