Models that stay inside

We ship AI gateways on the same contract as the rest of the platform: rotated keys, per-team quotas, caching when the prompt repeats, traces you can join with the rest of the telemetry. LiteLLM or a proxy you own — the logo does not matter. What matters is that ten API keys stop living in repositories.

This note is the long version: what the gateway actually does, how you control cost without slowing the product team down, what changes when inference has to run inside, and how you operate all of it without inventing a new discipline.

Why a single entry point

The pattern we meet is almost always the same. Three teams discovered models separately, each with its own key, its own SDK and its own way of counting tokens. The bill arrives aggregated and nobody can attribute it. When the vendor changes a model or introduces a rate limit, the incident shows up in three different places and gets diagnosed three times.

A gateway solves that because it turns a diffuse external dependency into an internal interface with an owner. The properties we are after:

• Identity: every call carries team, service and environment; the vendor key never leaves the gateway.
• Policy: which models each team may use, at which rate limit, under which spend cap.
• Routing: a logical model name resolves to a concrete provider, and that mapping changes without redeploying the services that consume it.
• Resilience: retries with jitter, circuit breaking, and fallback to an alternative model when the primary returns 429 or 5xx.
• Observability: input and output tokens, cost, time to first token and total latency, all with the same labels as the rest of your telemetry.
• Data safety: redaction or blocking before the text leaves the network, not after.

The contract with the product team

Adoption is won with compatibility. If the gateway speaks the same dialect as the SDK they already use, migrating costs one environment variable; if you invent your own API, the gateway becomes a toll booth and people route around it.

-  OPENAI_API_KEY=sk-live-...            # provider key sitting in the service
-  OPENAI_BASE_URL=https://api.openai.com/v1
+  OPENAI_API_KEY=${GATEWAY_TOKEN}       # internal token, one per team
+  OPENAI_BASE_URL=https://ai.internal/v1

# the service asks for a logical name, not a provider model
model: "summarise-fast"

That logical name is the piece that pays off most over time. When a cheaper model appears, or a workload has to move to internal inference, you change the mapping in one place. Without logical names, every provider change is a pull-request campaign across the whole monorepo.

model_list:
  - model_name: summarise-fast
    litellm_params: { model: openai/gpt-4o-mini, api_key: os.environ/OPENAI_KEY }
  - model_name: summarise-fast          # fallback
    litellm_params: { model: vllm/qwen2.5-7b, api_base: http://vllm.ai.svc:8000/v1 }

router_settings:
  routing_strategy: latency-based-routing
  num_retries: 2
  fallbacks: [{ summarise-fast: ["summarise-internal"] }]
  cooldown_time: 60

Cost: budgets, not surprises

Model cost behaves unlike the rest of your infrastructure: it scales with user behaviour and prompt size, not with replica count. A badly placed retry loop, or a context that grows without truncation, can multiply the bill within a day, and the usual way to find out is at the end of the month.

What we put in the gateway from day one:

• Budget per team and per environment, on daily and monthly windows, with defined behaviour on exhaustion: degrade to a cheaper model or reject, never fail silently.
• Per-request token ceilings, so a runaway context is cut before it is billed.
• Caching of identical responses and, where it applies, provider-side prompt prefix caching, usually the best effort-to-saving ratio available.
• Attribution inside the metric itself: cost labelled by team, service, model and feature, so the conversation is about a specific feature and not about “AI”.

# the useful alert is not "cost went up", it is "cost per request went up"
sum by (team, feature) (rate(llm_cost_usd_total[1h]))
  / sum by (team, feature) (rate(llm_requests_total[1h]))
  > 1.5 * avg_over_time(... [7d])

The distinction matters: if total cost rises because the product has more users, that is good news. If cost per request rises, something changed in the prompt, the model, or the retry logic, and you want a page.

When the data cannot leave

Then the gateway does not point at a vendor. It points at inference in your cluster: vLLM, Ollama, GPU Operator, the same GitOps you already use. Product still talks to one endpoint. Infra decides whether that call leaves or stays.

Serving a model inside is not just deploying one more container. The differences worth planning for:

• Weights are artifacts of tens of gigabytes. Version them and cache them on a volume or in an OCI registry; downloading on every start turns a restart into a twenty-minute outage.
• CPU-based autoscaling is useless. The useful signal is request queue depth or KV cache utilisation, and cold start for a GPU pod is measured in minutes.
• Concurrency is managed inside the inference server, not with replicas. Continuous batching and a cap on concurrent sequences buy far more throughput than adding pods.
• GPUs are expensive and scarce: reserve them with node selectors, taints and priority classes, and decide who may preempt whom before the first conflict.
• Time to first token is the number the user perceives. Total latency is the number your capacity perceives. Measure them separately.

Without that switch between outside and inside, “AI in the product” becomes a permanent exception: another network, another bill, another incident nobody knows who owns. With it, it is one more service on the map.

The data boundary is declared, not remembered

“Don’t send personal data to the API” is not a control, it is a hope. The control lives on the request path: network policy stops product pods from reaching the internet, so the only possible exit is the gateway, and the gateway enforces classification.

In practice, each logical route carries a sensitivity label. Routes marked internal may only resolve to models running inside, and that constraint is checked in CI against the configuration, not at runtime when it is already too late.

Logging counts too. Storing full prompts is tempting for debugging and is exactly the compliance problem you were trying to escape. We store metadata and hashes by default, and content only under sampling, with short retention and audited access.

Failure modes that do not look like the usual ones

• Silent degradation: the model returns 200 and the answer is worse. It only surfaces with periodic evaluations over a fixed case set, running as one more job.
• Vendor version drift: pin concrete model versions where possible and treat an upgrade like a deploy, with a canary and an abort criterion.
• Retry storms: long timeouts plus the client SDK’s automatic retries, stacked on top of the gateway’s, multiply cost during a vendor incident. Retries happen in exactly one layer.
• Streaming and cancellation: if the user closes the tab and nobody cancels the request, you keep paying for the full generation.
• Cache as a leak: a cached response with a badly built key can serve one tenant’s content to another. The key includes tenant and policy, always.

The same team operates it

We do not hand over a notebook and leave. The gateway lands with a pipeline, a runbook, and alerts that only fire when cost or latency leaves the agreed band. Its configuration lives in Git and gets promoted like any other service: by digest, with a canary, with a rehearsed rollback.

The runbook answers concrete questions: what to do when the primary provider is down, how to move a workload to internal inference while live, how to raise a team’s budget during a launch and how to revert it afterwards, and who approves a new route being allowed to leave the network.

If a product piece is missing around it — an internal portal for tokens and quotas, an on-call bot, a cost-per-feature dashboard — we build that too.

If you already have three teams calling models and no way to know what it costs or what text leaves the network, send us two lines. You get an initial read and a ballpark quote within 24h.