What is an LLM gateway?

An LLM gateway, also sold as an AI gateway, is a proxy that sits between your applications and the model providers. LLM stands for large language model, the engine behind modern AI, which you pay for by the token: the unit models read and write, roughly three-quarters of a word. The gateway is the central place for API keys, rate limits, spend tracking, and failover across models, so a hundred internal apps do not each hold their own provider key and their own untracked bill.

If you are approving an AI budget, this is the layer that determines whether you can answer three questions later: what did we spend, who spent it, and could the same work have run cheaper. Without a gateway, those answers live scattered across provider dashboards and engineers' memories.

Why it exists, and what breaks without one

AI costs scale with how much your agents read and write, not how often you call them, and agents read and write a lot. The biggest single lever on the bill is model routing: automatically sending simple tasks to a cheap, fast model and reserving frontier models for the requests where judgment matters. Routing needs a place to live, and that place is the gateway.

The second lever is catching pricing changes and cache behavior. When Anthropic cut cache-hit pricing 75% in 2026, from $1 to $0.25 per million tokens on its newest models (Anthropic pricing docs), the saving only materialized for teams whose calls actually landed as cache hits. A gateway is where you would see that a reordered prompt was silently billing at full input rates.

The third reason is damage control. AI model evaluator METR disclosed that a single stolen API key led to $600,000 in consumed model credits (Dark Reading). Hard spend caps per key, enforced at a gateway, put a ceiling on exactly that loss.

One caution belongs in every evaluation: the gateway concentrates your provider keys and your traffic, which makes it a target. In September 2026, the Cybersecurity and Infrastructure Security Agency (CISA) added an improper-authentication flaw in the widely used LiteLLM gateway (CVE-2026-59822) to its Known Exploited Vulnerabilities catalog, meaning attackers were exploiting it in the wild (CISA). A control point you do not patch becomes the breach.

What to ask vendors

• Can spend be attributed per team, per application, and per agent, not just per provider account?
• Are budget limits hard caps that stop traffic, or alerts that arrive after the money is gone?
• How are routing policies defined, and can we test a policy change against last month's real traffic before enabling it?
• Where do provider keys live, how are they rotated, and who can read them?
• What is the security track record and patch turnaround? Ask specifically how fast known exploited vulnerabilities get fixed in your deployment model.
• How much latency does the proxy add, and what happens to traffic when the gateway itself is down?
• If we leave, does our code keep working? Gateways that mimic a standard API shape are cheaper to exit.