A security researcher planted nearly 2 million trackable fake secrets into AI prompts sent through the network of routers and resellers that many AI apps rely on. Then he watched where they ended up. Many of them surfaced in places they should never have reached.

The experiment was designed to test whether sensitive information can leak as it passes through the chain of widely used AI services, and what happens to that information if it does. Many AI apps do not connect directly to a major model provider such as Anthropic or OpenAI. Instead, they send requests through brokers like OpenRouter or Vercel. These so-called AI routers, sometimes called AI gateways, receive prompts from app users and send them to an available inference engine, often picking the one with the lowest price or the best fit for the task. The developer hires the router to choose the model.

How did researchers test whether AI prompts leak data?

HD Moore, chief research officer at Dragos, built the test around what he called canaries, a reference to the birds once carried into coal mines to warn of gas. He assembled Amazon Web Services access keys tied to empty accounts, passwords for services such as SSH, and unique URLs, DNS names and private git repository references. None had ever been used. He controlled the infrastructure that received any request using them. He placed each secret in exactly one prompt sent to a single provider, so that any leak could be traced back to the link responsible.

Three hosting providers breached

The most serious finding was that attackers had compromised three separate hosting providers. The intruders reused what Moore described as “a couple hundred” of the AWS keys.

Moore wrote that each key was sent in one request to one provider, all within a few short windows in late October, and that every request sent during those windows was stolen. Some of the keys came from requests the provider rejected before any model ran. That, he wrote, showed the attacker was copying requests at the provider’s front door, not somewhere deeper in the system. In the other two breaches, some of the stolen requests were marked as Zero Data Retention, a setting meant to prevent storage. “A privacy flag doesn’t help when the attacker is already inside,” he wrote.

The people behind the breaches tested the stolen keys, in some cases within hours of the secrets being sent. They searched S3 buckets, secrets managers and parameter stores for more credentials. They tried to enable hosted models in AWS Bedrock and added their own IAM users to keep access. In total, the bait keys were used “tens of thousands” of times in AWS API calls.

Secrets sent to two of the breached providers were used together, almost immediately, from the same rented server. Moore concluded that the same adversary had almost certainly compromised both.

Resellers and a Chinese AI lab

Which hosting providers and resellers exposed secrets?

Some of the secrets were taken by the resellers themselves. Moore found that certain sellers offered low prices to attract prompts, then turned on request logging and waited. He identified them by sending requests for the same model through different sellers. When a credential later showed up only after being routed through one seller and not another, he knew the reseller, not the model provider, was responsible.

The test also showed that some providers were pulling secret URLs out of prompts, in apparent violation of their privacy terms. The clearest case involved a Chinese AI lab. Its systems fetched the secret URLs from more than half of the prompts sent to its models between July and October, Moore wrote. That included roughly two-thirds of the requests marked “no data collection” or Zero Data Retention. The scraper usually appeared two or three days after the request.

The problem reached beyond direct traffic. Several US hosting providers resell the lab’s models, and prompts sent to those providers set off the same scraper. Moore wrote that a customer who chose a US provider specifically to keep data in the country “didn’t get what they paid for.” Later, a git client on a different cloud began trying to clone the private repositories referenced in the prompts, but only for prompts that used this lab’s models.

Moore also reported that one US hosting provider was crawling a secret URL more than two months after it appeared in a single inference call to its search endpoint. Another ran a scraper that returned for URLs days after each request, no matter which AI router had sent it.

What users can do

What can users do to keep AI prompts private?

Moore said the most effective way to prevent leakage is to host models on on-premises servers. That option is costly, however. For companies that cannot afford it, he recommended using a first-tier provider that offers “one contract and a privacy policy to read.” Even then, he said, nothing guarantees that data will not leak, though the approach lowers the odds. He also advised users to avoid default provider or reseller pools and to choose upstream providers themselves.

For anyone trying to keep secrets out of AI prompts, the findings point to a hard problem with no quick fix. The chain of routers, resellers and hosts means a sensitive prompt can be logged or copied at several points, and privacy settings at one link may not protect the others.

Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.