Managed LLM APIs vs. Your Own vLLM Server: A Question of Workload Shape, Not Ideology

Teams tend to argue about Bedrock and Vertex AI versus self-hosted vLLM as though one of them has to be correct. It's the wrong frame. Managed APIs and a GPU-backed inference server built on vLLM and llm-d serve different workload shapes, and the same open-weights model often shows up in both. What changes is the economics, the latency floor, and how much control you have over the cache.

Where the managed API is the right call

For intermittent or modest traffic, the API wins on almost every axis. A basic FAQ chatbot, an internal document summarization endpoint, a nightly classification job, these have low duty cycles. You pay per token, you pay nothing when idle, and you inherit a frontier model without touching a GPU.

A vLLM deployment inverts that cost structure. GPU instances bill by the hour whether or not requests arrive. That only pencils out at high sustained utilization: heavy agentic traffic, large batch pipelines, real-time features hit continuously during business hours. Below that threshold, self-hosting means paying for idle accelerators to save on tokens you weren't sending anyway.

So the first question isn't "which is better." It's: is your GPU going to be busy?

Where it gets interesting: workloads with large, repeated context

The calculus shifts when a large share of every request is identical to the last one. Think of a system that ships a full database schema, a set of detailed tool descriptions, and a long system prompt on every call, then appends a short user query. The repeated prefix may not be the majority of tokens, but it's substantial and it's stable.

This is exactly what caching exists for, and it's where the two approaches genuinely diverge.

Prompt caching on managed APIs comes with a floor

On a managed API, you don't cache; you request a cache checkpoint and hope you cleared the vendor's minimum. Those minimums are model-specific and not small.

Vertex AI (Gemini context caching), minimum cache token count, applying to both implicit and explicit caching:

Note what that table does not say: the lightweight model is not the cheap option for caching. Current Gemini 3-series Flash models sit at the same 4,096-token floor as the Pro models. Reaching for a smaller model to cut latency doesn't buy you a smaller cache minimum.

Amazon Bedrock (Claude models), minimum tokens per cache checkpoint:

The inversion is the point. Sonnet 4.5, the higher-reasoning model, caches from 1,024 tokens. Haiku 4.5, the model you'd pick precisely because it's fast and cheap, demands 4,096. If your stable prefix is a 2,500-token schema plus tool definitions, it caches fine on Sonnet and never caches on Haiku.

1. Failure is silent. Below the threshold, the request succeeds normally. No error, no warning. You just pay full input price and eat the full prefill every single time. The only signal is cacheReadInputTokens sitting at zero, which nobody notices unless they're already watching for it.
2. The published numbers move and don't always match reality. Sources disagree on whether Sonnet 4.5 on Bedrock is 1,024 or 4,096, and developers have reported Gemini 2.5 Flash enforcing 1,024 where the docs say 2,048. You are building latency guarantees on a constant the vendor can change.

Miss the threshold and every request re-prefills the whole prefix. That's directly visible to users as slower time-to-first-token.

KV caching on your own vLLM server has no floor

vLLM's automatic prefix caching works at a different level of the stack. It caches KV blocks, the actual attention state, keyed by content hash, at a block granularity you configure. There's no minimum-tokens policy, because there's no vendor policy at all. A 600-token prefix caches as readily as a 60,000-token one.

That gives you two things a managed API structurally can't:

• Model choice. You pick the open-weights model that fits the job, at the size that fits your latency budget. Frontier reasoning models are slow by construction; a well-chosen lightweight model can cut TTFT and end-to-end latency dramatically. For a workload where a user is waiting, that difference is the product.
• Cache policy as a tunable, not a constraint. Block size, cache capacity, eviction, offloading to CPU or disk, all yours to set against your actual prefix distribution.

Two more reasons that have nothing to do with tokens

Compliance. In healthcare, banking, insurance, and similar regulated settings, "where does the data go" is the question that stalls launches for months. A vLLM server inside your own VPC means inference data never leaves your network boundary. That collapses a large amount of vendor-assessment, DPA, and data-residency work into an architecture decision you already control.

Red teaming. You cannot meaningfully study a model you can only reach through a filtered endpoint. Running open weights yourself lets you probe jailbreak behavior directly, vary sampling parameters, inspect logprobs, and study how a particular model's weights and post-training shape its refusal behavior. That's safety research a managed API is designed to prevent you from doing.

The catch: someone has to engineer this

None of the above is free. The failure mode of self-hosting isn't that caching doesn't work, it's that the cache is on the wrong GPU.

Standard round-robin load balancing is actively harmful here. Each vLLM replica holds a different KV cache state. Route a request to a replica that doesn't hold its prefix and you've thrown away the computed work and re-run prefill from scratch, the exact cost you stood up the cluster to avoid.

This is what llm-d is for. It sits above the model servers and does prefix-cache-aware routing: scoring pods by how much of the incoming request's prefix they already hold, combined with load-awareness so you don't hammer one hot replica. Its precise mode consumes KV-cache events emitted by vLLM to maintain an accurate global view of where cache blocks actually live. Reported results are substantial, roughly 3× lower mean TTFT versus a cache-blind baseline in the project's own benchmarks, and 2× faster TTFT with 3× higher throughput in a Red Hat/Tesla deployment on MI300X.

But this is real systems engineering: Kubernetes, a gateway with an endpoint picker, cache-event plumbing, autoscaling that respects cache warmth, and observability good enough to prove hit rates in production. It is not a config flag.

The heuristic

• Low or bursty volume, short prompts, no data-residency pressure → managed API. Don't build a cluster to save on tokens you aren't sending.
• Sustained high volume, large repeated prefixes, latency-sensitive users, or a regulated data boundary → vLLM behind llm-d, and budget for the engineering.
• Stable prefix under ~4k tokens plus a need for a fast lightweight model → this is the case where managed prompt caching quietly fails you, and self-hosting is most clearly justified.

The uncomfortable version: managed prompt caching is a product, with thresholds set for the vendor's infrastructure, not your prompt. KV caching on your own server is a mechanism. If your workload lives near the threshold, you don't really control your own latency.

References

1. Amazon Web Services. Prompt caching for faster model inference, Amazon Bedrock User Guide.
2. Anthropic. Prompt caching, Claude Platform Docs (per-model minimum cacheable prefix lengths).
3. Caylent. Amazon Bedrock Prompt Caching: Saving Time and Money in LLM Applications (Claude 4.5-family thresholds, 5-minute vs 1-hour TTL).
4. tidyverse/ellmer, Issue #954, Add prompt caching support to chat_aws_bedrock() (summary of Bedrock per-model minimums and silent below-threshold behavior).
5. Google Cloud. Context caching overview, Vertex AI / Gemini Enterprise Agent Platform (implicit vs. explicit caching, minimum cache token count).
6. Google. Context caching, Gemini API docs (per-model minimum input token table).
7. Google AI Developers Forum. Context caching docs list wrong minimum token count for Gemini 2.5 Flash (documented vs. enforced minimums).
8. llm-d. Project README, intelligent routing, advanced KV-cache management, published benchmark results.
9. llm-d. Feature: Precise Prefix Cache Aware Routing (routing from vLLM KV-event data).
10. llm-d. Announcing the llm-d community (prefix-aware routing TTFT benchmarks, disaggregated serving).
11. Red Hat Developer. Master KV cache aware routing with llm-d for efficient AI inference (EPP, Gateway API Inference Extension, prefix-aware scoring).
12. vLLM. Automatic Prefix Caching, vLLM documentation.