Most teams ship RAG the same way: embed documents, retrieve the top-k chunks, paste them into the prompt, ask the model to summarize. In a demo with fifty PDFs, that works. In production with CRM exports, ticket threads, wiki pages, and live database tables — asked hundreds of times per day by agents and copilots — the bill arrives fast.
Input tokens dominate cost. A single agent turn that stuffs 30–50 retrieved chunks plus a schema dump plus conversation history can easily hit 20,000–40,000 input tokens. Multiply by users, retries, and multi-step tool loops, and “we added AI search” becomes a real FinOps problem.
90% savings is not magic compression. It is architecture: stop sending data the model already has tools to fetch.
This article walks through where tokens leak in production RAG, five changes that compound into massive savings, and when vector search still belongs in the stack — alongside structured retrieval, not instead of thinking clearly about what actually needs to land in context.
1. Where production RAG tokens actually go
Before optimizing chunk overlap, map the spend. In typical enterprise copilots, input tokens cluster in four buckets:
| Bucket | What gets sent | Why it explodes at scale |
|---|---|---|
| Retrieved chunks | Top-k passages from vector DB, often 500–1,500 tokens each | High k, wide chunks, and “just in case” retrieval on every turn |
| Schema & docs in system prompt | Table lists, column names, API specs, internal wikis | Copied into every request instead of fetched once via tools |
| Retry & replan loops | Same chunks re-sent after failed or vague answers | Agent re-retrieves because grounding was fuzzy, not scoped |
| Conversation history | Prior turns including full prior retrievals | Context window fills with redundant chunk text |
Teams often optimize embedding model cost while ignoring the bigger line item: paying for the same enterprise knowledge on every single question. That is the lever that gets you to 90%.
2. A worked example: “Which accounts does Alex own?”
Consider a common internal question against CRM data spread across Postgres and MongoDB. Two architectures, same user intent.
Naive RAG path (~28k input tokens / turn)
Embed CRM exports and support tickets. Retrieve 40 chunks (~500 tokens each) = ~20,000 tokens.
Add system prompt with table/column cheat sheet = ~6,000 tokens. User message + history =
~2,000 tokens. Model guesses which chunk mentions “Alex Anderson” and which column is
owner_user_id.
Structured retrieval path (~2.5k input tokens / turn)
Playbook already defines crm_user, crm_account, and
owns_account. Agent calls query_graph with entity + relationship. Runtime returns
three structured rows = ~200 tokens. System prompt stays thin; schema lives in MCP tools,
not the prompt. History stays small because answers are compact JSON, not chunk dumps.
3. Five changes that compound in production
-
1
Retrieve rows, not paragraphs If the question has a schema — customer, account, owner, payroll row — compile it to a query and return JSON. Vector search is for prose; SQL, graph plans, and MCP tools are for facts. A 200-token result beats a 10,000-token chunk bundle every time.
-
2
Move schema out of the system prompt Do not paste your entire data dictionary into every request. Expose
get_playbook_context,introspect_source, andlist_entityvia MCP. The agent pulls schema when needed, once per task — not on every chat message forever. -
3
Scope retrieval with access rules up front Unscoped RAG retrieves broadly, then filters in the prompt (“only show Alex’s data”). That wastes tokens on rows the user should never see. ReBAC and playbook
accessblocks filter at query time — return five allowed rows, not five hundred chunks you hope the model ignores. -
4
Query in place; stop re-embedding operational data Pipelines that copy Postgres → lake → vector index pay twice: storage/sync cost and prompt cost when stale snapshots get retrieved. Federated query-in-place returns live rows without duplicating them into embedding space. See why we stopped moving data for AI.
-
5
Tool-first agents; summarization last Order matters: plan → execute → summarize. When agents dump retrieval into context before planning, they pay for text they may not need. A thin plan step (
plan_query) followed by execution keeps the LLM’s job to natural language ↔ structured intent, not shuffling megabytes of text.
4. RAG vs structured retrieval — use both, not one
Cutting tokens does not mean deleting your vector database. It means not using vectors for everything.
| Question type | Best retrieval | Typical token profile |
|---|---|---|
| “What is our refund policy for EU customers?” | Vector RAG over policy PDFs / wiki | Moderate — a few relevant passages |
| “List accounts Alex owns in the CRM” | Graph / SQL via playbook + bindings | Low — small JSON result set |
| “Summarize this 80-page contract” | Vector RAG + map-reduce or section routing | High — unavoidable for long docs |
| “Can employee 1042 see this payroll row?” | Governed query_graph with ReBAC |
Low — boolean + proof, not chunk soup |
| “Why did ticket #8842 escalate?” | Hybrid — structured ticket metadata + RAG on thread body | Medium — scope structure first, prose second |
5. Tactical RAG optimizations (when you still need vectors)
For the document slice of your stack, these moves still matter — they just will not get you to 90% alone if operational data dominates traffic:
- Lower k dynamically — start at k=3; increase only on low confidence, not by default
- Smaller chunks with metadata headers — 256–400 tokens with title/source beats 1,500-token slabs
- Re-rank before inject — one cheap reranker step beats sending 20 mediocre chunks to GPT-4
- Strip boilerplate on ingest — headers, footers, and nav chrome are pure token tax
- Cache embeddings and answers — identical FAQ queries should not re-embed and re-retrieve
- Summarize history — drop prior chunk text from conversation; keep user intent and final answers
Treat these as hygiene. The step-change savings come from not treating your CRM as if it were a PDF collection.
6. The reasoning layer: ontology once, query many times
Reasoning as Code is the pattern that makes structured retrieval cheap at scale. A playbook defines business entities and relationships once — versioned in Git, reviewed like code — instead of re-explaining your schema in every prompt.
Anything CLI sits between agents and live sources. It does not store operational rows; it stores playbooks, bindings, and proof.
The LLM sees business vocabulary (crm_user, owns_account) and
compact results. Postgres and MongoDB stay where they are; the runtime queries them in place. Token spend
tracks answer size, not corpus size.
Compare that to embedding every account row and user profile into a vector store and hoping semantic search finds the right owner field. Same answer, fraction of the tokens, and a proof trail of which queries ran.
7. Production checklist
Before your next model upgrade or k-value tweak, ask:
- What percentage of queries are structured lookups vs unstructured doc search?
- Is schema duplicated in the system prompt on every request?
- Does retrieval respect row-level access before text hits the model?
- Are agents re-sending the same chunks on retry loops?
- Are operational tables being vectorized when SQL/graph would suffice?
- Do you measure tokens per successful answer, not just tokens per request?
The bottom line
RAG is the right tool for unstructured knowledge. It is the wrong default for enterprise facts that already live in tables, APIs, and graph relationships. Production token bills explode when every question pays the price of a literature review.
Cut costs by 90% — realistically, for operational workloads — by retrieving answers instead of evidence piles, keeping ontology in playbooks instead of prompts, and enforcing scope before context assembly. Keep vector search for prose. Ship a reasoning layer for everything else.
Smaller prompts are not a hack. They are what governed AI was supposed to look like.