Five LLM Gateways, Five Failures: Researchers Read Across Customers Through the Prompt Cache

A preprint announced on arXiv at 00:00 UTC on 19 August measures where LLM API relay paths break prompt-cache isolation. The finding is uniform: all five open-source gateways tested, connected to both OpenAI and Anthropic, failed to bind customers to upstream credentials by default, and all five exposed cross-customer cache reads on both providers.

What the attack is

Prompt caching makes repeated prefixes cheap by storing them upstream. When a gateway pools many customers behind one credential, the cache is shared, and timing differences reveal whether another customer's prefix is already present. In production measurements covering 80.5% of eligible token volume on OpenRouter, the authors identified cross-account reads for 12 of 28 labels, representing 33.7% of volume. On one production route they recovered eight consecutive target positions without any access to the target.

What the common telling gets wrong

This is a measurement of an isolation gap, not a breach. No customer data was stolen, no incident occurred, and nothing here is evidence of exploitation in the wild. The researchers constructed the conditions and measured what leaked. Reporting it as "LLM gateways hacked" or "prompts stolen" describes an event that has not happened.

It is also a preprint: announced on arXiv, not peer-reviewed, and not independently replicated. Second, the failure is a default, not a law of nature — the gateways can be configured otherwise, and the paper's point is that they are not, out of the box.

The number that makes it awkward

The proposed defence — placing cache identity splits after reusable public prefixes — costs 1.7% to 2.5%. A defect whose remedy is under two and a half percent of spend is hard to defend as a considered performance trade-off, which is what makes the default indefensible rather than merely unfortunate.

Why relays matter

Gateways sit between applications and model providers to handle routing, failover, spend control and multi-provider abstraction. Their entire value is pooling — and pooling credentials is exactly what collapses the boundary between tenants. Any team running a shared relay in front of a paid model API should be able to state which credential a given customer's request travels on.