I Tested Six Ways to Save GPT-5.6 Tokens. The Typical Task and the Recorded Workload Disagreed

TL;DR

• I compared six approaches meant to save tokens (Caveman, Ponytail, RTK, Context Mode, Mandarin prompts, and 5.6 Terra xhigh) with a baseline across ten real tasks. I repeated all seven arms twice for a total of 140 agent runs.
• On the recorded workload of ten tasks, none of the six reduced total tokens in both runs.
• Averaged per task, Caveman and the cheaper Terra model looked mildly cheaper both times, but the differences were small enough that run-to-run randomness could explain them.
• Context Mode and Mandarin used far more tokens no matter how I counted.
• The trajectories suggest changes in agent behavior: search, delegation, validation, and patch scope. Several local savings coincided with code review losses, patches that were not equivalent, and lower robustness.
• The only repeated drop in recorded cost came from switching to the cheaper 5.6 Terra xhigh model: −49% in dollars while the recorded workload used slightly more tokens.

An agent is not a file you can compress

Tools for saving tokens make the same pitch: lower usage without significant tradeoffs. This treats the agent like a file. Shrink the dialogue, filter the shell output, index the context, and the total gets smaller.

But an agent is not a file. It is a system that decides what work to do next, and every one of these interventions changes those decisions. An agent that receives compressed command output can respond by running more commands. An agent told to write less code gets to decide which requirements count as optional. Nothing about a smaller local surface guarantees a smaller trajectory, and the trajectory is what you pay for.

So that is the question this study actually asks: what does the agent do next, after its context is compressed?

What "saving tokens" means

"Did it save?" hides two separate choices: what you count (tokens or dollars) and how you summarize the ten tasks. I report every result both ways:

• The recorded workload: add up everything the ten tasks actually consumed and compare totals. This is closest to the real bill for this exact batch, but one or two enormous tasks can dominate the number.
• The typical task: pair the treatment with the baseline on each task, take the percentage change on each pair, and average those changes (geometrically). This stops the biggest tasks from deciding the answer, but it no longer adds up to the batch total.

Both views exist for tokens and for cost, so the tables below carry all four columns. Cost adds one wrinkle of its own: dollars follow the price structure, including cache discounts, so cost can move differently from tokens when the mix of cached input, fresh input, output, model, or reasoning effort changes.

Caveman shows why the distinction matters. Its workload totals moved +7% and −11%, while its paired point estimates were −5% and −9%. Terra's workload totals rose 6% and 5%, while its paired point estimates were −23% and −12%. Those four are descriptive point estimates, and all four sit in the noise band: each paired bootstrap interval includes zero. Caveman's −9% estimate carries a 95% interval from −24% to +6%, and Terra's −23% carries one from −62% to +38%. The scale of the noise is visible in the baseline itself, which cost $6.93 per task in run 1 and $9.00 in run 2, a 30% swing with the same tasks and the same configuration.

Methodology

The tasks are ten real merged changes from the production repo of Stet, the local eval harness I'm building. Stet replayed these tasks and graded each attempt on more than whether its tests passed or failed. I ran a baseline (5.6 Sol at medium effort) and six treatments, all with the Codex CLI:

• Caveman: a terse interaction prompt pack that strips conversational filler and compresses the agent's dialogue (your agent is now a Neanderthal).
• Ponytail: an output minimization instruction set that guides the agent to act like a lazy senior engineer. It pairs "The best code is the code never written" with a YAGNI (you aren't gonna need it) ladder to climb before writing anything and a cap on validation work.
• RTK: a wrapper that filters and compresses shell command output before it reaches the model.
• Context Mode: batches commands, indexes large results, and returns only the sections judged relevant.
• Mandarin mode: the claim that Chinese's denser tokenization saves tokens (中文最好的吗？).
• 5.6 Terra at xhigh: the simplest way to save money is to use a cheaper model.

Seven arms ran twice: 14 arm runs × 10 tasks = 140 agent runs. The repetitions used the same tasks, so they measure consistency rather than giving us 20 independent tasks.

Quality included tests, semantic equivalence, code review, deterministic footprint risk, and eight LLM graders scoring on a 0–4 scale across several rubrics meant to capture software engineering best practices: clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality.

Results

The first resource table reports exact totals for the recorded workload. The second reports inferential paired geometric estimates for a typical task. Caveman and Terra's negative paired token estimates remain in the noise band (due to low n).

First, the main question: did the mode consume fewer resources than the baseline? Each cell reports treatment versus baseline for repetition 1 and repetition 2. Negative means the treatment saved.

On this view, no treatment reduced total tokens twice. Terra alone had lower recorded cost twice, through a different model and price schedule.

Now here is the paired geometric estimate for a typical task:

This view changes the directional read. Caveman's typical task estimate was lower twice even though one recorded workload total rose. Terra's was also lower twice even though both workload totals rose. The point estimates for Context Mode and Mandarin remain higher under either estimator.

recorded workload totals · exact for this batch

no mode used fewer tokens in both recorded workloads

Workload tokensTypical taskRecorded cost

Candidate change against the same GPT-5.6 Sol baseline. Each square is one repetition across the same ten Stet tasks. Left of zero is lower; right of zero is higher.

-20%0%+20%+40%+60%+80%+100%

Caveman

R1 +7%

R2 −11%

Ponytail

R1 +16%

R2 −1%

RTK

R1 +18%

R2 −8%

Context Mode

R1 +94%

R2 +46%

Mandarin

R1 +86%

R2 +15%

Terra xhigh

R1 +6%

R2 +5%

Mode	Run 1	Run 2	Direction
Caveman	+7%	−11%	mixed
Ponytail	+16%	−1%	mixed
RTK	+18%	−8%	mixed
Context Mode	+94%	+46%	higher twice
Mandarin	+86%	+15%	higher twice
Terra xhigh	+6%	+5%	higher twice

Second, quality: each mode was paired against the baseline on the same task, giving 20 task pairs across the two runs. The table counts wins, losses, and ties out of those 20. For tests, equivalence, and code review, a win means the mode's patch did better. For footprint, a win means lower risk.

Third, the eight quality graders. Passing tests is not the same as producing a patch you would merge. Each cell is the mode's mean score minus the baseline's, on the 0–4 scale, averaged across both runs, where negative means the graders liked the mode's patches less.

The descriptive pattern worth investigating is that the two modes that push the agent to write less (Ponytail) or say less (Caveman) take their biggest hits on robustness and scope discipline. The diff minimality gains for RTK and Mandarin coincided with higher average token use, not lower.

Local compression did not guarantee a smaller workload

“Token saving” is not one intervention. These modes optimize different local surfaces:

• Ponytail tries to emit less code and less explanation.
• Caveman compresses the interaction.
• RTK compresses individual shell command results.
• Context Mode compresses and indexes retrieved context.
• Mandarin changes the prompt representation.
• Terra changes the model, reasoning level, and pricing structure.

But the bill is for the whole agent trajectory. A model can receive a smaller command result and compensate by running twice as many commands. It can emit a smaller patch after spending more tokens deciding how little to write. It can make repository context cheaper to retrieve and respond by retrieving much more of it.

The observed trajectories suggest that the treatments changed search, delegation, validation, and patch boundaries.

Output compression is not workload compression.

Other studies point to the same gap between local savings and the whole trajectory. A replay of 500 real Claude Code sessions found that the 60–90% savings advertised by tools in this category shrank to under 4% of total spend once resent context was counted. JetBrains found the same pattern in controlled studies of Caveman prompting and RTK. This study adds a comparison across six mechanisms on GPT-5.6.

Caveman: small wins erased by expensive misses

Caveman won 12/20 token pairs. Its workload total flipped from +7% to −11%, while its paired estimate for the typical task was −5% and −9%. Tests tied on every task, while equivalence went 0-4 and code review 2-6.

JetBrains' Caveman study is a useful resource going deeper on Caveman. Across 82 clean pairs with the skill forced on, it found 8.5% fewer output tokens with no detectable SkillsBench score change, and JetBrains notes that normal opt-in use can only save less. But the raw full-run cost rose 11.6% because one long-context outlier dominated the bill. Different harness, model, and tasks, same warning: central tendency and the actual bill can disagree.

For our test, the 12 wins averaged −20% tokens; the 8 losses averaged +23%. The losses also came from different failure modes:

• On a replay fidelity bug fix where files hidden by .gitignore were not being staged into Stet's synthetic base commit, Caveman won big and slipped small. Both arms shipped the same production fix in one line and passed tests, equivalence, and review. Caveman used 21 tool calls versus the baseline's 41 and 2.39M tokens versus 5.96M, but added a standalone regression test of 89 lines instead of extending the existing fixture. Footprint moved from low to medium, and the run still took longer.
• The biggest miss (+56% tokens) was a CLI task hardening --json output: stdout isolation, error handling, contract tests. Caveman ran 86 tool calls to the baseline's 49, including roughly twice the git archaeology and a stack of rg searches the baseline never issued, and produced a 907-line patch against the baseline's 230. Terse dialogue, four times the diff.

The mechanism was not “do less” in a uniform sense. In these examples, Caveman compressed conversation or exploration while expanding a patch, a validation loop, or a test artifact. That variance can explain how paired point estimates fell while one workload total rose.

Caveman compressed the interaction, not reliably the total work.

Ponytail: shorter output, incomplete patches

Ponytail was the only mode that reduced output tokens in both repetitions: −3% and −11%. Workload tokens moved +16% and −1%; typical task tokens moved +8% and −3%.

Output is expensive: under current pricing it costs an order of magnitude more than fresh input and roughly fifty times as much as a cached read, so Ponytail targeted the most valuable stream. The rest of the trajectory erased the benefit. It lost tests 1-4, equivalence 0-2, and code review 0-4 while improving footprint 15-4.

Output tokens fell on 11 of 20 pairs while total tokens fell on only 9. The same task swung from +46% total tokens in one run to +126% in the other. The review findings suggest a possible quality mechanism:

• A UI task asked for CTA blocks with separate analytics tracking for each placement. Ponytail delivered the biggest output saving in the arm (−51% output tokens, −41% total) and lost tests, equivalence, and review at once. It appended a decorative arrow that broke an accessibility test requiring an exact match, and merged two distinct analytics placements into one even though placement tracking was the point of the task.
• Another task, the offline prompt context helper, passed tests in both arms. Review found that Ponytail's version never reconstructed context from persisted artifacts, which was the function's whole purpose. It merely reformatted what callers handed it. Tests were a weak signal, and the smaller patch was structurally incomplete.

To be fair, the footprint wins were sometimes genuinely clean. On the ignored file staging task in run 2, Ponytail shrank a bloated test fixture from 124 lines to 4 while preserving the implementation and every quality verdict. When the fat is real, this mode trims it.

One hypothesis follows from the injected behavior. Ponytail says “the best code is the code never written,” adds a seven-rung YAGNI ladder, and caps validation. That may reward the shortest diff before semantic adequacy is established, making error paths, adjacent callers, and edge cases easy to classify as unnecessary.

In the observed failures, Ponytail optimized the visible patch without covering the full semantic boundary.

RTK: smaller command output, more queries

RTK filters shell output before it reaches the model. That sounds like direct token savings. However, it was not reliable: workload tokens moved +18% and −8%, while tokens for the typical task moved +18% and −5%, with 8 token wins against 12 losses.

JetBrains reached a similar conclusion with Claude Code. They estimated that even at 70% compression, RTK's hook could move only about 3% of input tokens in their setup. At low effort, median cost rose 7.6% as turns and cache reads increased; a single high-effort run showed a median delta near zero. Task quality was tied under their sign tests. RTK's own counter reported 96.2 million tokens saved while the measured bill rose, which is JetBrains' larger lesson in one number: a tool's self-reported savings describe its counterfactual, not your bill.

During our testing, in run 1, RTK averaged 60 tool calls per task to the baseline's 49. Call counts rose on nine of ten tasks, and RTK was slower on all ten.

• The clearest case was a small frontend feature: add a TrackLink analytics CTA component to Stet's leaderboard page. Both arms passed tests, equivalence, and review. RTK produced a slightly more explicit implementation and three focused tests instead of two, but used 60 tool calls versus 30, 6.29M tokens versus 3.06M, and took 3.7× longer. The exploratory robustness score rose from 2.2 to 2.8; coherence fell from 3.2 to 2.3.
• The single worst loss (+7.6M tokens, roughly 2× the baseline) was the offline prompt context helper task. The extra commands were not new kinds of work. RTK ran thirteen sed reads to the baseline's three and eleven git invocations to its ten, retrieving the same territory again in compressed slices. It spent twice as much to leave the task incomplete: the baseline's patch was judged a precise solution while RTK's missed the core requirement to reconstruct artifacts.
• The wins are instructive too. RTK's clearest token win in run 1, on the task that hardened --json output, happened despite running 20 more tool calls than the baseline. The run leaned on subagent delegation and achieved a 94% cache hit rate, so the extra calls were nearly free. The saving came from cache economics, not from the output filter doing its job.

The wrapper succeeded at its narrow job: each command returned less raw output. The agent then issued more commands, performed more validation, and read the same files again in smaller pieces. The savings rate of an individual rg, git, or test command says nothing about the number of commands the model will choose to run afterward.

Saving tokens per command did not help when the agent issued twice as many commands.

Context Mode: compressed retrieval, more context

Context Mode had the opposite effect, it actually increased token consumption. Workload tokens rose 94% and 46%; typical task estimates rose 102% and 51%. It had 19 losses in 20 paired tasks and lost tests 2-5.

In run 1, Context Mode averaged 74 tool calls and 79 assistant turns per task to the baseline's 49 and 51:

• The worst case (+200% tokens) was the TrackLink CTA component task, the same small frontend feature that tripped up RTK. Context Mode's session log shows 22 retrieval calls, at least ten of which ask nearly identical versions of the same question: what are TrackLink's exact props, exports, and analytics semantics? The index made each lookup cheap, so the agent repeated the question instead of reasoning from what it already had. Tests went from pass to fail in both runs on this task.
• The next task cuts the opposite way and provides the study's only token pair win for this mode. The task asked the agent to automatically resolve which evaluator provider the CLI should use and pin the model for the AI runtime, a change touching configuration, provider lookup, and authentication. Context Mode flipped the focused verifier and code review from fail to pass in both runs, while winning footprint both times. In run 1 that correctness cost 2× the tokens and 2.37× the time. In run 2 the same behavior came in 26% under baseline. Both patches still missed Claude authentication propagation and remained not equivalent.

One explanation is that cheaper, structured repository search encouraged more retrieval and revisiting. The observed arm spent longer integrating that context.

On this task, more searchable context coincided with better localization, incomplete obligations, and much higher context use.

Mandarin: translated prompts, a different work plan

There's a claim that prompting in Chinese saves tokens based on the idea that denser characters require fewer tokens. In our testing, Mandarin increased workload tokens by 86% and 15%, and its typical task estimates rose 70% and 12%, losing 18/20 token pairs. The observed execution strategy also changed.

In run 1, the baseline varied its subagent count by task. The Mandarin arm used exactly four on all ten tasks, a fixed decomposition, and issued about 38% more tool calls.

• The single biggest loss in the whole study (+21.7M tokens, on the CLI provider resolution task) involved a different failure mode. The subagent counts were identical, but Mandarin used 126 tool calls to the baseline's 60 even though its patch was smaller and narrower. That was all process bloat, with no extra product.
• The two token wins, both in run 2, are just as telling. On the task to forward Claude setup tokens, the Mandarin run happened to spawn one subagent instead of four and came in 5.8M tokens under baseline. It had lost the same task by 10.6M in run 1 with four subagents. When the fixed decomposition relaxed, so did the cost.

The descriptive diff minimality gain was concentrated in the setup token task, where the score rose by more than 3.0 in both runs. The translated arm also showed different decomposition and delegation.

The Mandarin arm differed in decomposition and delegation, not just tokenization.

Terra xhigh: cheaper, not more token efficient

Terra is not a compression trick. It is the most realistic way to save dollars: switch to a cheaper model and turn the reasoning dial up. Its workload totals used 6% and 5% more tokens, while its paired point estimates were 23% and 12% lower, within the noise band. Its recorded cost fell about 49% in both repetitions.

Terra improved deterministic footprint 16-4, but lost equivalence 0-4 and review 2-4. Its clearest descriptive grader deltas were coherence −0.37, robustness −0.33, and diff minimality −0.33.

Terra was cheaper on 19 of 20 task pairs. The token deltas were bimodal: roughly half used 20% to 90% fewer tokens, while the other half used 56% to 185% more. The 6% workload increase nets out those opposing populations.

• The side with token growth is the most interesting. On the offline prompt context task, Terra made fewer tool calls but consumed 56% more tokens. It expanded the solution into tarball extraction, cleanup ownership, error propagation, and more than 100 lines of tests. Review flipped from pass to fail because the new failure path for repository context could block offline rendering. Terra still cost less in run 1, but this was also the task that produced its only cost loss in run 2.
• The quality losses, however, did not come from scope expansion. All three tasks that lost equivalence failed because Terra satisfied only part of a requirement with several components. On the flake rerun serialization fix, Terra serialized the batch that moved from fail to pass but missed the batch that already passed the gold test, repeating the same omission in both runs. On the setup token task, it forwarded authentication to some clients but not all. On the Gradle flag task, it handled --info as a special case instead of implementing the broader recognition requested.
• On the clean side of the bimodal split, Terra beat the baseline on both tokens and cost for the CLI provider resolution task in both runs (−29%/−63% in run 1, −71%/−83% in run 2), with about 21% less diff churn across the same four files.

In run 1, Terra averaged fewer tool calls (42 versus 49) with more output tokens per turn. But the task did not predict whether a run would land in the cheap and tight population or the sprawling one. The quality failures came from omitted components, not from overbuilding.

Terra demonstrates exactly why cost reduction and token reduction are different measurements.

What this is and isn't

This is ten tasks, run twice, on one repo and model family. Reusing the tasks measures repetition, not generalization. Baseline trajectories for run 2 were not retained, so tool call and turn comparisons come from run 1. Token, cost, and verdict data are complete for all runs. Small runs also exaggerate: JetBrains watched a −29.5% saving in a ten-task smoke run shrink to −8.5% at full scale, which is why nothing here rests on a single repetition.

The canonical reports are inconclusive and recommend inspection. Grader discrimination authority and runtime strictness evidence were unavailable, and the test basis was mixed. The resource results and patch-level evidence are useful diagnostics. The quality tables are exploratory, not a leaderboard.

Bottom line

The one intervention that reliably cut the bill was not a token trick at all. Terra, a cheaper model at higher reasoning effort, cut recorded cost roughly in half in both repetitions while using slightly more tokens. Every mode that compressed something locally, dialogue, output, command results, retrieved context, the prompt language, actually failed to reduce the recorded workload's tokens twice, and Context Mode and Mandarin made it dramatically worse. If your goal is a smaller bill, the reliable lever was simply using Terra.

These modes did not just shrink text - they measurably changed how the agent worked. Ponytail's shorter patches skipped parts of the requirement. RTK's compressed command output was followed by more commands. Context Mode's cheap retrieval invited more retrieval. Mandarin changed how the agent delegated work. And several of the visible savings arrived together with patches that failed review or were not semantically equivalent: the saving and the quality loss were often the same event seen from two sides.

Adopting one of these modes is less like installing a compressor and more like hiring a different engineer who happens to be cheaper per word. That is also why the number to trust least is the one a tool reports about itself: a compressor's counter measures the text it deleted, not what your agent does.

Do not adopt a token saving mode from its local savings claim. I recommend testing and validating that the agent actually behaves as expected, and actually saves tokens, before adopting these fully.

Disclosure: I am building Stet.sh, the local eval tool I used to run this. You can ask your coding agent to improve its own setup, adopt or reject a skill for saving tokens, tune AGENTS.md, or change models. It then uses Stet to test candidate changes against historical repo tasks before you commit. I am looking for a few teams that already use coding agents heavily and face a concrete decision, such as whether an output compression wrapper saves money, whether to use high or xhigh reasoning, or whether to use Codex or Claude Code. Stet runs entirely locally using your LLM subscriptions. Reach out to me directly.