Your inference bill is decided by caching, not by the model

Sarvam Token Factory publishes three token rates and reports a 95.7% cache hit rate. That ratio, not the headline price, decides what your AI feature costs.

Your inference bill is decided by caching, not by the model

Sarvam AI opened its Token Factory to developers on 16 November, serving its own Sarvam 105B alongside open-weight models — GLM 5.3, DeepSeek V4.1 Flash 1017, DeepSeek V4 Pro, Kimi K3.1 Pro, Gemma 4 31B — through one OpenAI-compatible API, from GPU clusters in India. Inc42 frames it as India's inference-as-a-service moment, with L&T, E2E Networks' Jarvislabs and others moving the same way.

The interesting number in Sarvam's post is not the headline price. On DeepSeek V4.1 Flash the company says it serves over 450 billion tokens a day, and 95.7% of prompt tokens come from cache. That ratio moves your bill more than switching models does.

Three prices, not one

Sarvam publishes three rates per model: input, cached input, output. On DeepSeek V4.1 Flash Standard the introductory rates are ₹14, ₹0.25 and ₹55 per million tokens.

A cached input token costs roughly one-fifty-sixth of a fresh one, so "₹14 per million" is a ceiling, not a price. What you actually pay depends on how much of your prompt the cache recognises:

  • 0% cached — ₹14.00 per million input tokens
  • 90% cached — ₹1.63
  • 95.7% cached — ₹0.84

Two teams on identical traffic can pay sixteen times different amounts at the same published rate. The difference is prompt structure, not procurement.

The hit rate is decided by your prompt

Prefix caching is exact and positional: a cached prefix is reusable only if the leading tokens are byte-identical to an earlier request. Everything after the first divergence is recomputed at full price.

So two choices are yours, not the vendor's. Put stable content first — system instructions, tool schemas, retrieved documents, few-shot examples at the top; the user's question, timestamps, request ids and shuffled JSON at the bottom. Drop a datetime.now() into your system prompt and you invalidate the cache for the whole fleet, silently, at 56x the price. Keep serialisation byte-stable: tool definitions regenerated from a dictionary with unstable key order render as a new document on every call. Sort the keys.

Sarvam's own post describes a three-tier cache with cache-aware routing and admission control. Routing matters because a cache only helps if the request lands on the worker holding it; a naive load balancer scatters identical requests and destroys the hit rate it was meant to deliver.

Once caching works, output is the bill

Take 400 million input tokens and 60 million output a month. At DeepSeek V4.1 Flash Standard rates with no cache that is about ₹8,900. At a 95.7% hit rate, input falls to roughly ₹336 and the total lands near ₹3,640.

Output is now about 90% of the bill, and the optimisation problem has moved. Trimming the system prompt was the right work at 0% cache; at high hit rates it is pennies. What remains is output volume:

  • Cap max_tokens per call site from measurement, not from the context window.
  • Separate answer length from reasoning effort. Reasoning tokens bill at the output rate, so a chain-of-thought that walks three dead ends costs like an answer three times as long.
  • Prefer fields to prose. A JSON response with three keys beats a paragraph you then parse.

Buy latency only where it is felt

DeepSeek V4.1 Flash runs on two endpoints: Standard at ₹14/₹55 per million, Interactive at ₹25/₹105 — about 1.8x on input, 1.9x on output. The difference is queueing behaviour under load, not quality.

Run one endpoint for everything and an overnight re-index competes with the chat box, paying a latency premium it does not need. Routing by call site is a config change with a measurable bill impact: coding agents and user chat on Interactive; summarisation, backfills and eval runs on Standard.

Check the calendar too. Those rates are introductory through the end of 2026; a cost model assuming they hold in January 2027 has an expiry date. Price the post-introductory case now and know what volume makes you renegotiate.

Residency is a claim you have to document

Inference on Indian GPU clusters is useful when an enterprise buyer asks where data is processed under DPDP — it does not have to leave the country. It is not, by itself, compliance. Localisation answers where; it supplies no purpose limitation, retention period or breach path. Nor does it settle the licence terms of whichever open-weight model you pick.

The practical minimum: a processor agreement naming the sub-processor, prompt and completion logging disabled or bounded at your gateway, and a written answer to "what do you retain, and for how long". On-premise deployment, which Sarvam also offers for critical institutions, moves that obligation to you rather than removing it.

The escape hatch is the real feature

Open weights plus an OpenAI-compatible API means switching providers is a base_url and a model string. That is leverage: no vendor's deprecation schedule can end your product. Preserve it by making model choice configuration — an env var and a routing table — and pinning the exact version identifier, such as deepseekv4.1-flash-1017, so a silent model update does not become a silent behaviour change in your evals.

A checklist you can run this week

  1. Log input, cached-input and output tokens separately per call site. Two columns cannot show the number that dominates your bill.
  2. Compute your own cache hit rate, not the provider's aggregate.
  3. Move volatile strings out of the prefix; sort serialised tool schemas.
  4. Split interactive and batch traffic across endpoints.
  5. Set max_tokens and reasoning effort per call site from measured data.
  6. Track cost per completed user task, weekly, and price the post-introductory scenario before the rates change.

None of this is exotic — it is the instinct that had people read query counts before buying a bigger database. The inference layer was abstracted into a per-token price so nobody would have to think about it, which is why the teams that do think about it pay a fraction of what everyone else does. NetPy builds and reviews these systems; NetPyCloud runs the infrastructure inside India.

Header image: "AMD Fiji GPU package with GPU, HBM memory and interposer" by C. Spille/pcgameshardware.de, licensed CC BY-SA 4.0, via Wikimedia Commons. Illustrative image.