Anatomy of AI Cost & Token Economics

Models, context, batching, caching, and cost guardrails.

How to ship without burning the budget — pick the right model, feed the right context, batch & cache aggressively, and measure relentlessly.

---

TL;DR

• Total cost = (input tokens × input price) + (output tokens × output price) — every layer scales with this formula.
• The biggest savings come from less context and shorter outputs, not just cheaper models.
• Tier your models: Haiku for routing & summaries, Sonnet for everyday code, Opus only when accuracy really matters.
• Batch · cache · stream · parallelize — four levers that compound.
• Instrument first. What you don't measure, you can't reduce.

---

Why this matters

LLM costs scale with use, not seats. A team that ships 50 PRs/week with AI assistance can easily 10× its bill in a month if no one is watching the meter. The fix isn't to use less AI — it's to use AI deliberately.

Bottom line: Token discipline is the difference between a $200/mo magic teammate and a $20,000/mo budget surprise.

---

The Big Picture — The Cost Formula

Total Cost = (Input Tokens × Input Price)
           + (Output Tokens × Output Price)

Cost varies by:

• 🧠 Model (Haiku < Sonnet < Opus)
• 📦 Context size (every token you send)
• 📝 Output length (often the bigger half!)
• 🛠️ Tool calls (each round-trip = more tokens)
• 💾 Caching (huge multiplier when reused)
• 🌍 Region / provider

🪙 Golden rule: Optimize the product of tokens × price at every step.

---

Model Price Guide (USD per 1M tokens)

💡 Prices are indicative and change frequently. Always confirm with the provider's pricing page before locking in a budget.

Model (as of mid-2025)	Input	Output	Best for
Claude Opus 4	$15.00	$75.00	Complex reasoning, critical tasks
Claude Sonnet 4	$3.00	$15.00	Balanced quality & cost ⭐
Claude Haiku 4	$0.80	$4.00	Simple tasks, summaries, routing
GPT-4.1	$10.00	$30.00	General-purpose, code, analysis
GPT-4o mini	$0.15	$0.60	High-volume, simple tasks
Gemini 1.5 Pro	$3.50	$10.50	Large context, multimodal

💡 Rule of thumb: Start with Haiku/mini → escalate to Sonnet → only then Opus.

---

How Context Drives Cost

Typical cost breakdown for a medium coding session:

Source	% of cost	Why
Context (input)	~55%	Files, history, system prompts
Output (generation)	~30%	Responses, code, explanations
Tools & misc	~15%	Tool calls, retries, overhead

Costs spike when you:

• ❌ Attach large files blindly
• ❌ Paste long logs / stack traces
• ❌ Keep old conversation history forever
• ❌ Ask for verbose outputs ("explain everything")
• ❌ Run many tool calls per task

💡 Tip: Trim input. Constrain output. Compact early.

---

Token Budgeting Framework

Estimate before you run. Measure after you run. Refine.

Task type	Recommended model	Est. input tokens	Est. output tokens	Est. cost (USD)
Quick answer / lookup	Haiku 4	500	300	~$0.001
Summarization	Haiku 4	2,000	600	~$0.003
Code generation (small)	Sonnet 4	2,000	1,500	~$0.028
Code review (medium)	Sonnet 4	8,000	2,000	~$0.054
Architecture / design	Opus 4	12,000	3,000	~$0.405
Multi-file refactor	Opus 4	20,000	4,000	~$0.600

💡 Tip: Multiply expected daily runs to forecast monthly burn. A 200× factor between Haiku and Opus is normal.

---

Choose the Right Model — Decision Tree

💡 Tip: Start cheap. Escalate only when the cost of being wrong is high.

---

Context Management = Cost Management

✅ Do this

• Load only needed files (@path/to/file).
• Use summaries, not full logs.
• Keep earlier conversation short.
• Enable auto-compact (50% threshold).
• Use structured outputs (JSON) — fewer wasted tokens.
• Drop irrelevant context between turns.

❌ Avoid

• ❌ Dumping entire codebases
• ❌ Pasting huge logs / traces
• ❌ Asking for essays when bullets do
• ❌ Repeating the same info
• ❌ Keeping solved threads alive
• ❌ Verbose explanations by default

💡 Tip: Every token in saves 1–10 tokens out (and dollars).

---

Batching & Parallelism

Batch similar tasks

Summarize 10 PRs at once → one call → return JSON list
Extract metadata from files → one prompt, batched files
Classify tickets in bulk → one prompt, many inputs
Generate changelogs → one prompt, multiple commits

Why it wins: shared system prompt + amortized overhead across N tasks.

Parallelize independent tasks

Wins on wall-clock time (faster sessions, smaller context per agent).

Cache aggressively

• Prompt caching: Reuse identical system prompts at fraction price.
• Embedding caching: Don't re-embed unchanged docs.
• Tool-output caching: Cache deterministic tool results.
• Invalidate when underlying data actually changes.

💡 Tip: A 90%+ cache hit rate on system prompts is achievable and pays for itself in days.

---

Reduce Output Cost (often the bigger half)

Instead of this…	…say this
"Write a long explanation with examples."	"Explain in 3 bullets with one example."
"Give me full docs."	"Give me an outline then fill key parts."
"List all edge cases."	"List top 5 critical edge cases."
"Verbose code review."	"Summarize findings by severity."

Output-shaping tactics

• 📋 Ask for JSON or bullet points.
• 🪶 Set max_tokens / max_length.
• ✂️ Use a concise tone: "Be brief."
• 🪜 Break into steps with small outputs each.
• 🧪 Summarize before deep dives.

💡 Tip: Output discipline = huge savings. Often 30–60% off a bill.

---

Instrument, Measure & Alert

Track these metrics

Metric	Why it matters
Input tokens	Detect bloated prompts
Output tokens	Detect verbose responses
Total cost (USD)	The bottom line
Cost per task	Right-size your unit economics
Cache hit rate	Caching effectiveness
Model usage mix	Are you over-using premium tiers?

Sample logging payload

{
  "ts": "2025-05-20T10:30:00Z",
  "model": "claude-sonnet-4",
  "input_tokens": 5421,
  "output_tokens": 812,
  "cost_usd": 0.0189,
  "task": "code_review",
  "cache_hit": false,
  "user": "alice",
  "session_id": "abc123"
}

Alert when

• 🚨 Cost per day > budget
• 🚨 Single request > threshold (e.g., $1)
• 🚨 Token usage > 90% of context window
• 🚨 Cache hit rate drops suddenly (regression)

💡 Tip: What gets measured gets optimized. Send these to your existing dashboard (Datadog, Grafana, etc.).

---

Budgets & Guardrails

Daily / monthly budget config

{
  "budgets": {
    "daily_usd": 50.00,
    "monthly_usd": 1000.00
  },
  "enforcement": {
    "mode": "soft",            // soft | hard
    "warn_at_pct": 80,
    "block_at_pct": 100
  }
}

Guardrail playbook

Threshold	Action
80% of budget	Warn user + team channel
50% context used	Auto-compact
80% context used	Downshift to cheaper model
Request > $X	Require human approval
100% budget	Hard block + escalation

💡 Tip: Budget pain today = budget automation tomorrow. Codify thresholds early.

---

Step-by-Step Setup

1. Classify your tasks and map to models

Create docs/model-routing.md:

| Task | Model | Why |
|---|---|---|
| Triage / routing | Haiku | Cheap, fast, simple |
| Code review | Sonnet | Balanced quality/cost |
| Critical refactor | Opus | Accuracy worth the cost |

2. Enable auto-compact in .claude/settings.json

{
  "env": { "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "50" },
  "features": {
    "autoCompact": true,
    "autoCompactMode": "balanced"
  }
}

3. Add a cost-logging hook

scripts/log-cost.sh:

#!/usr/bin/env bash
# Append per-call metrics to a daily file
DATE=$(date +%Y-%m-%d)
mkdir -p .claude/logs
echo "{\"ts\":\"$(date -Iseconds)\",\"model\":\"$CLAUDE_MODEL\",\"in\":$CLAUDE_INPUT_TOKENS,\"out\":$CLAUDE_OUTPUT_TOKENS,\"cost\":$CLAUDE_COST_USD}" \
  >> ".claude/logs/cost-$DATE.jsonl"

.claude/settings.json:

"hooks": {
  "PostToolUse": [
    { "matcher": "*",
      "hooks": [{ "type": "command", "command": "./scripts/log-cost.sh" }] }
  ]
}

4. Add a /budget slash command

.claude/commands/budget.md:

---
description: Show today's spend, top tasks, cache hit rate
usage: /budget [--month]
---
Read `.claude/logs/cost-*.jsonl` and report:
- Spend today vs daily budget (with % bar)
- Top 5 costliest tasks
- Model usage mix
- Cache hit rate
- Suggestion: any task to downshift?

5. Start cheap, escalate deliberately

Default model = Sonnet. Switch to Opus only via explicit /escalate command.

6. Review weekly

Weekly retro: which tasks blew the budget? Re-route them.

---

Variants for Different Team Settings

👤 Solo developer

• Default to Haiku for everything except complex refactors.
• One JSON log, one weekly glance.
• Budget: $20–50/mo typical.

👥 Small team (2–10)

• Shared model-routing.md.
• Per-user daily soft-cap, team monthly hard-cap.
• Slack alert at 80%.
• Cache shared system prompts repo-wide.

🏢 Enterprise / regulated

• Hard budgets per project.
• Cost attribution by team/cost-center via metadata in every call.
• Approval workflow for Opus / large-context requests.
• Centralized FinOps dashboard.
• Quarterly model re-tiering as prices change.

🌍 Open-source / community

• Cap per-contributor session to prevent abuse.
• Document recommended models in CONTRIBUTING.md.
• Provide a free-tier-friendly default config.

Bottom line: Same formula, escalating discipline.

---

Common Patterns & Recipes

Pattern 1 — Two-pass cheap-then-deep

Pass 1 (Haiku): classify and triage the diff.
Pass 2 (Sonnet): only deep-review files flagged risky.

Pattern 2 — Summarize-then-act

Step 1: Summarize the 20 files into a 200-line plan.
Step 2: Hand only the plan + 3 files to Opus.

Pattern 3 — Hierarchical routing

router:
  default: haiku
  if_keyword: ["refactor", "architecture", "design"] → opus
  if_size > 5000_tokens → sonnet

Pattern 4 — Cached system prompt

Reuse identical "You are a senior reviewer..." block.
Cache hit = ~10× cheaper input tokens.

Pattern 5 — Budget-aware retry

On model error or timeout:
  - First retry: same model
  - Second retry: downshift one tier
  - Then: surface to human

Pattern 6 — JSON-mode for parsability

Return ONLY this JSON:
{ "summary": "...", "issues": [...], "fixes": [...] }

Outputs shrink ~30% and your code can parse without LLM glue.

---

Best Practices ✅

• Use the smallest model that works.
• Feed only what's essential.
• Batch, cache, reuse, summarize.
• Measure, alert, iterate.
• Codify budgets & approvals.
• Right-tier weekly as prices and quality evolve.
• Document model routing so the team makes the same call.

Anti-Patterns ❌

• 🚫 Defaulting to Opus "to be safe".
• 🚫 Pasting entire codebases into context.
• 🚫 Asking for verbose explanations by default.
• 🚫 No caching of system prompts.
• 🚫 No metrics, no budgets, no alerts.
• 🚫 Treating context length as free.
• 🚫 Hardcoding model names in every script (impossible to retier).

---

Troubleshooting

Symptom	Likely cause	Fix
Sudden cost spike	New chatty prompt / large file attached	Audit recent diffs to prompts
Slow + expensive	Wrong tier (Opus on simple tasks)	Add router rule, downshift
Cache hit rate drops	Prompt changed slightly each call	Stabilize boilerplate; parameterize variants
Output too long	No format / max_tokens	Enforce JSON or bullets + cap
Token usage > 90%	Context bloat	Enable auto-compact (50%)
One user dominates spend	No per-user budget	Per-seat soft caps + Slack alert
Repeated tool errors	Retries on premium model	Budget-aware retry + downshift

---

Quick-Start Checklist

• ✅ Classify tasks and map them to models.
• ✅ Write prompts that are concise & structured.
• ✅ Enable auto-compact (50%).
• ✅ Cache what you can (system prompts, embeddings, tool outputs).
• ✅ Instrument & set budgets (daily + monthly).
• ✅ Review metrics weekly and iterate.

🧠 Mantra: Right model. Right context. Right pattern. Right time. Right results.

---

Starter Configs

.claude/settings.json — cost-savvy defaults

{
  "model": {
    "default": "claude-sonnet-4",
    "fallback": ["claude-haiku-4"]
  },
  "env": {
    "MAX_THINKING_TOKENS": "60000",
    "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "50"
  },
  "features": {
    "autoCompact": true,
    "autoCompactMode": "balanced"
  },
  "hooks": {
    "PostToolUse": [
      { "matcher": "*",
        "hooks": [{ "type": "command", "command": "./scripts/log-cost.sh" }] }
    ]
  }
}

model-routing.md — your team's rule book

# Model Routing

| Task                | Model    | Rationale            |
|---------------------|----------|----------------------|
| Routing / classify  | Haiku    | Cheap, fast          |
| Summarization       | Haiku    | Output is short      |
| Daily code work     | Sonnet   | Balanced             |
| Architecture        | Opus     | Accuracy matters     |
| Bulk transforms     | Haiku    | Batch + parallelize  |

## Escalation
Use `/escalate` to bump the next tier with a 1-line reason
(logged for the weekly review).

---

Glossary

• Token — Roughly ~4 characters of text; the LLM's unit of work and cost.
• Context window — Max tokens the model can "see" per call.
• Input tokens — What you send (prompt + system + history + tools).
• Output tokens — What the model returns. Often 5× the price of input.
• Prompt caching — Reusing identical prompt prefixes at a discount.
• Auto-compact — Automatic summarization to keep context lean.
• Routing / tiering — Picking the cheapest model that meets the bar.
• Batching — Combining many tasks into one call.
• FinOps — Discipline of managing cloud / AI costs.

---

Further Reading

• Search: "Anthropic prompt caching docs"
• Search: "OpenAI pricing & batch API"
• Search: "LLM cost optimization patterns 2026"
• Search: "FinOps for AI workloads"

---

🪙 Golden Rules

• 🛑 Never attach what you don't need.
• 🌱 Start small. Scale only when needed.
• 📊 Measure everything. Optimize relentlessly.
• 🤖 Automate the boring. Focus on the multiplier.

Remember: The cheapest token is the one you don't use.