AI Coding Token Costs Explained: How to Reduce Context, Output and Agent Costs

Throughout this series, we have focused on making AI coding agents more reliable.

But there is another problem that becomes increasingly important once developers begin using agents every day:

Cost.

A short conversation with an AI model may be inexpensive. An autonomous coding agent working through a large repository is different. It may:

• 01 read dozens of files
• 02 search the repository repeatedly
• 03 load project instructions
• 04 inspect documentation
• 05 receive thousands of lines of terminal output
• 06 run tests several times
• 07 maintain a long conversation history
• 08 generate code, explain changes, and delegate work to subagents

At that point, token usage is no longer an abstract API metric. It becomes part of the economics of software development.

And one of the easiest mistakes is assuming that reducing the amount of text the AI writes is the same as reducing the amount of information the AI processes.

It is not.

To optimize AI development properly, we need to understand where the tokens actually go.

First: what is a token?

Language models do not process text exactly as words. They process smaller units called tokens.

A token might represent:

• an entire short word
• part of a longer word
• punctuation
• whitespace
• pieces of source code

The exact tokenization depends on the model. For our purposes, AI usage can broadly be divided into:

AI Request
    │
    ├── Input Tokens
    │
    └── Output Tokens

Agentic coding makes the input side much larger than it first appears.

Input tokens: everything the agent has to read

Input tokens include the information supplied to the model before it produces its next response. That may include:

1. System instructions
2. Project instructions
3. Conversation history
4. Source files
5. Documentation
6. Tool results
7. Terminal output and test failures
8. Previous agent responses + current request

For a simple chat question, the input may be relatively small. For a coding agent halfway through a complex development task, it can become enormous.

Claude Code’s current documentation provides a good illustration: its context can include project instructions, files that have been read, conversation history, tool results, memory, loaded skills, and other system content. As the session grows, that context eventually needs to be cleared or compacted. Claude Code docs, “Context window”

The agent may repeatedly pay the cost of understanding its accumulated context before it writes anything new.

Output tokens: everything the agent generates

Output tokens are easier to see. They include:

Output tokens can also be relatively expensive depending on the model and provider. For example, current OpenAI token based pricing distinguishes between input, cached input, and output tokens, with output tokens priced separately—and often at a significantly higher rate than ordinary input tokens. OpenAI, ChatGPT rate card

That makes reducing unnecessary output useful. But with coding agents, it is only one part of the problem.

Fig. 01 — Read far more than you write

A coding agent can read far more than it writes

Consider a hypothetical coding task. The agent receives:

Then it generates:

Visible output: about 4,000 tokens. If we reduce the response from 4,000 tokens to 2,000 tokens, that is useful. But the agent still had to process roughly 70,000 input tokens.

This is why token optimization needs two separate strategies:

Both can save money. They solve different problems.

The hidden cost of long agent sessions

Imagine starting an AI coding session with:

Implement customer invoice exports.

The agent investigates the project. It reads billing files, then invoice templates, then database models, then tests. You fix something together. Later you say:

Now add filtering by date.

The agent already has a large history. Then the export needs to appear on the admin dashboard. Then the tests fail. Logs and test output appear. Eventually the session contains a substantial amount of information that may no longer be relevant to the current task.

Even if your latest request is only “fix the button label,” the model may still be operating inside a session containing a large history of previous work.

This is context accumulation. Long running sessions can become both expensive and less reliable.

Context windows are capacity, not free storage

A model advertising a very large context window can create the impression that filling it is harmless. It is better to think of the context window as a maximum working area. You still need to decide what belongs on the desk.

The second engineer technically has more information. They do not necessarily have a better working environment. The same principle applies to AI.

Fig. 02 — Capacity is not a working environment

Strategy 1: retrieve files instead of loading the repository

This is the most important input token optimization. Do not begin with “here is my entire codebase.” Begin with “find the code responsible for subscription renewal.”

1. Search
2. Find candidates
3. Read relevant files
4. Follow references

This is exactly why the retrieval strategies discussed earlier in this series matter. Repository search is not simply a convenience. It is a token management system.

If the task requires 6 files out of 4,000, successfully identifying those 6 files can save far more context than any clever wording inside the prompt.

Strategy 2: keep persistent instructions small

We discussed this in the AGENTS.md article, but it has direct cost implications.

Suppose your permanent instructions contain 15,000 tokens, and those instructions are loaded into every agent interaction. Now imagine an agent performs 50 model turns. Even where caching reduces the effective cost, you have still designed a system around repeatedly presenting a large amount of static information.

Compare that with an AGENTS.md of about 1,500 useful tokens, which directs the agent toward deeper documentation only when required.

Agent instructions should act as a routing layer. Not a knowledge dump.

Strategy 3: keep tool output under control

Tool results are an overlooked source of context bloat. Suppose an agent runs pnpm test and receives 20,000 lines of output. Does the model need all of it? Probably not.

The useful information may be:

3 tests failed

checkout.test.ts
Expected status: 200
Received: 500

subscription.test.ts
Expected: active
Received: cancelled

invoice.test.ts
Snapshot mismatch

If your agent infrastructure can provide focused failures rather than complete logs, you preserve context for actual reasoning. The same applies to Git diffs, database queries, logs, search results, compiler diagnostics, and API responses.

Do not ask how much information the tool can return. Ask what information the agent needs to make its next decision.

Strategy 4: use subagents for expensive exploration

Suppose the main agent needs to understand authentication. It reads 25 files and 18,000 tokens. Then it only needs the conclusion:

Authentication uses JWT access tokens, refresh tokens are stored in HttpOnly cookies, and authorization rules live in packages/auth/policies.

A subagent can perform the exploration in a separate context and return the useful findings.

1. Main context asks to investigate auth
2. Research subagent reads many files
3. Concise findings return
4. Main context stays small

The large investigation does not necessarily need to remain inside the main agent’s working context. Claude Code explicitly documents this pattern: subagents can investigate in separate context windows, returning summaries while keeping their large intermediate file reads out of the main session. Claude Code docs, “Context window”

This is progressive context management at the agent level.

Fig. 03 — Isolate the expensive search

Strategy 5: compact long conversations

Eventually, useful sessions become long. At that point, one strategy is compaction. Instead of retaining the entire history of prompts, responses, tool calls, and tool results, the system creates a structured summary.

45,000 tokens of history
            ↓
         compact
            ↓
5,000 tokens describing:

- current objective
- decisions made
- files changed
- architectural constraints
- unresolved failures

The agent continues from the compressed state. Claude Code, for example, automatically manages context as it approaches the model’s limit and supports explicit compaction of the conversation. Claude Code docs, “How Claude Code works”

Compaction does have a trade off. Summaries are lossy. An important detail can disappear. So the correct solution is not merely to compact everything aggressively.

Keep permanent knowledge in the repository and use conversation history mainly for temporary working state.

That way, important rules do not disappear when a session is compressed.

Strategy 6: start a new session when the task changes

Suppose you have just finished an authentication investigation. The next request is to redesign the homepage hero section.

Does the homepage task need the previous authentication investigation? Almost certainly not. A new session gives the agent project instructions, the homepage task, and relevant frontend files—instead of the entire authentication investigation, logs, tests, backend files, and then the homepage task.

Context continuity has value. But irrelevant continuity has a cost.

Continue the session when the new task depends on previous reasoning. Start fresh when it does not.

Now we get to output tokens

Everything so far has focused primarily on what the model reads. But AI agents can also be extremely verbose. Ask them to fix a TypeScript error and you may receive analysis, root cause narration, a plan, a recap of every edit, and a complete summary.

Sometimes this is useful. Often, during an autonomous coding task, you mainly need:

Fixed null handling in BillingService.

Tests:
✓ billing tests
✓ typecheck

Changed:
- BillingService.ts
- BillingService.test.ts

The code is the deliverable. The narration is overhead. This is the problem addressed by tools such as Caveman.

Caveman: what happens if we make the agent talk less?

Caveman is an open source skill designed to make AI agents communicate in an extremely compressed style. Its philosophy is essentially: keep the important information; remove linguistic filler.

Same core information. Far fewer words. The original Caveman project reported output token reductions around 65% in prose oriented benchmark scenarios. Caveman on GitHub

That sounds dramatic. But there is an important caveat.

Independent testing found a much smaller saving for coding agents

JetBrains independently tested Caveman on real agentic coding tasks using paired runs. Their result was very different. They measured approximately 8.5% fewer output tokens rather than 65%. JetBrains, “Speaking to AI agents like cavemen”

Why such a large difference? Because agentic coding output contains things Caveman deliberately does not compress: code, tool calls, diffs, error messages, commands, and structured data. These may need to remain exact. The only thing available for compression is the narration around them.

Even if you eliminate two thirds of the narration, you have not eliminated two thirds of the total output. That explains why a technique can save huge amounts of tokens in conversational prose but much less during autonomous coding.

This is a useful lesson about benchmarks

The Caveman example illustrates something broader than one skill. If someone claims “our technique reduces AI tokens by 65%,” the next question should be: 65% of what?

For real engineering decisions, the relevant metric is your own workload.

Did Caveman reduce quality?

This is perhaps the more interesting part of the JetBrains test. Their paired evaluation did not find a statistically meaningful degradation in task quality from forcing Caveman style output across the tested tasks.

That makes sense conceptually. The agent can still reason and use its tools. It simply communicates less narration back to the developer. This points toward an interesting optimization:

Reduce communication verbosity without reducing engineering verification.

We do not want fewer tests, fewer inspections, less reasoning, or less verification just to reduce tokens. We want the same useful work with less unnecessary text. Those are very different goals.

Caveman 2 moves toward input compression

The original Caveman style skill mainly addresses output. But the project has since expanded its ambitions. Its newer Caveman 2 tooling also targets the input side by compressing information before it reaches the model.

The project’s published WRAP benchmark reports 33.2% fewer provider reported input tokens across a controlled set of agent shaped workloads, while preserving the benchmark’s exact answer checks. The authors correctly label this as controlled benchmark evidence rather than production billing data. Caveman WRAP benchmark

This distinction is important.

Now we are attacking the more expensive problem for many agent workflows: context growth.

But compression has a risk: information loss

Imagine a log contains:

Payment failed.

Primary error:
Connection timeout.

Retry attempt 2:
Connection timeout.

Retry attempt 3:
Stripe response 402
card_declined

Customer:
CUS-7821

A poor compression system might summarize:

Payment failed due to repeated connection timeouts.

Unfortunately, it removed the most important detail: Stripe 402 — card_declined.

Compression saves tokens only if it preserves the information needed for the next decision. This means compression itself needs evaluation. The objective is not to make text as short as possible. It is to create the smallest representation that preserves decision relevant information.

That principle applies to conversation compaction, subagent summaries, tool output filtering, log compression, repository maps, documentation retrieval, and Caveman style systems.

Fig. 04 — Keep the signal

Caching changes the economics again

There is another complication. Not all input tokens necessarily cost the same. Some AI platforms support cached input, where repeated prompt content can be processed at a lower cost than completely new content.

For example, current OpenAI token based pricing explicitly distinguishes input, cached input, and output as separate usage categories.

This means removing 10,000 tokens of highly cacheable static content may not produce the same cost saving as removing 10,000 tokens of new tool output.

Token count and actual financial cost are related, but they are not always identical.

When optimizing a production AI system, measure the provider bill—not just the length of the prompt.

The wrong way to optimize tokens

Token reduction can become counterproductive. Suppose an organization tells its engineering agent to minimize tokens at all costs. The agent responds by reading fewer relevant files, skipping tests, avoiding documentation, producing almost no explanation, and making decisions with incomplete information.

The token bill falls. The bug rate rises. That is not optimization. It is moving cost from your AI provider to your engineering team. A single production incident can cost more than millions of tokens.

Minimize unnecessary token usage while preserving or improving engineering quality.

Think in terms of information density

This gives us a better concept than simply “short prompts.” Call it information density.

The second version is shorter. But more importantly, it has a better signal to noise ratio. Likewise, 20,000 lines of test logs may contain less useful information than a single failing assertion.

Token optimization is fundamentally an information architecture problem.

A practical token efficient AI coding workflow

Putting everything together, a cost aware agent workflow might look like this:

1. Start with concise project instructions
2. Search before reading
3. Load only relevant files
4. Delegate large investigations
5. Filter noisy tool output
6. Start fresh when the task is independent
7. Compact long running context when needed
8. Keep final narration concise

Notice what is not being removed:

We are removing noise. Not evidence.

Measure before optimizing

If AI usage matters financially to your organization, instrument it. Track metrics such as:

• 01 input and output tokens per task
• 02 cached input
• 03 number of model turns and tool calls
• 04 average context size
• 05 cost per successful task
• 06 task success rate and human correction rate

The most interesting metric may not be tokens per request, but cost per successfully completed task.

Agent B is more expensive per attempt. But potentially much cheaper per useful outcome. This is why token efficiency cannot be evaluated separately from quality.

The real optimization target

When developers first encounter token costs, they often focus on prompts. They try to shave words until “please inspect this file and determine whether…” becomes “inspect file bug.” That may save a few tokens. Meanwhile the agent loads 80,000 tokens of repository context. The priorities are backwards.

The largest savings often come from architecture:

• better retrieval
• smaller persistent instructions
• focused tool output
• fresh sessions
• context compaction
• subagent isolation
• caching
• high information summaries

Then, after those are handled, output compression can reduce the remaining waste.

The key principle

There are really two questions in AI token optimization:

What does the agent need to read? What does the developer need the agent to say?

Everything else should be challenged.

For the input side: give the model the smallest context that still allows it to make the correct decision. For the output side: return the smallest response that still communicates the result clearly.

Tools such as Caveman are interesting because they demonstrate that AI responses often contain significant linguistic overhead. But they also demonstrate the limits of output only optimization. In real coding workflows, a large portion of the cost can come from the agent repeatedly reading the world around it.

That means the biggest opportunity is broader than making AI speak like a caveman. It is designing AI systems with high information density from end to end.