Ayush Gupta - AI Agent Engineer

The cheapest way to manage an agent's context is also one of the most accurate. That is the result I took from An Empirical Study of Harness Design for Coding Agents (Fan, Zhang, Ma, Hu, Wang, Song, Liu, Zamani, Wang). A staged strategy, which first trims old tool output and only summarizes when it has to, had success comparable to the alternatives and the lowest cost in 7 of 8 model-benchmark panels.

And one popular idea did nothing. Giving the agent a tool to fetch back what was trimmed did not help, and most agents never used it.

I build agents that sit in Slack and call many MCP servers for many users. So I wanted to know how much of this carries over. This post covers what the paper measured, what the wider research says about tool overload, and the design I would use.

01 What the paper measured

The authors ran 176 matched settings across four models (Nemotron-3 at 30B, 120B and 550B, and Mistral-Medium-3.5-128B) on two benchmarks: SWE-Bench Verified (500 issues) and Terminal-Bench 2.1 (89 tasks). They varied three parts of the harness: planning, the action space, and context management.

For context they tested five strategies at four budgets: 32k, 64k, 96k and 128k tokens.

Tier	What it does
T0	Nothing. When the trajectory outgrows the window, the run ends with an error.
T1	Elision. Stale tool observations become short stubs. Originals are discarded.
T2	Elision plus a `recall_event` tool to fetch an elided observation back.
T3	Summarization. The oldest middle events are folded into a running summary by a separate, tool-free call to the same model.
T4	Staged. Elision at a soft threshold, summarization at a hard threshold.

T4 is the interesting one. Elision starts at 0.6 of the usable window. Summarization starts at 0.85. The system prompt, the task, and a recent window (budgeted at 0.3, never fewer than two turns) are always kept verbatim.

The logic is simple once you see it. Elision is cheap: no model call, just replacing a long observation with a stub. Summarization costs a model call and loses detail. So you do the cheap thing first and only pay for the expensive one when the cheap one is not enough.

T4 had comparable success to the other managed strategies, the lowest cost in seven of eight model-benchmark panels, and the lowest mean cost at every budget.

02 The recall tool nobody called

T2 seems like it should win. You trim aggressively, but the agent can always ask for the original back. Nothing is truly lost.

But against plain elision, T2 won 15 settings, lost 14, and tied 3. The mean difference was -0.36 points. And in 36 of the 64 settings that had the tool (T2 and T4), the agent never called recall_event at all.

So the safety net is mostly decoration. A model that does not know what it is missing does not ask for it. And the tool's description still costs tokens on every turn.

03 The other two findings

Planning helped weak and strong models in different ways. For the 30B model, planning raised success by 11.6 points on SWE-Bench and 4.5 on Terminal-Bench. For the stronger models it mainly cut cost: Mistral's SWE-Bench cost fell about 32% while success moved by 0.4 points.

The action space showed the same split. The largest Nemotron did better with bash alone, and cost 53% less on SWE-Bench and 30% less on Terminal-Bench. A predefined tool set helped the weaker models more.

The pattern across all three: scaffolding is an accuracy aid for weak models and a cost lever for strong ones. That is worth remembering whenever you add structure to a harness.

04 The bigger context problem: too many tools

A coding agent's context fills up with tool output. An agent connected to many MCP servers has a second problem: the tool definitions fill it up before the user has said anything.

Anthropic gives a concrete example in its advanced tool use post: five MCP servers, 58 tools, about 55K tokens of definitions. Internally they had seen tool definitions reach 134K tokens.

More context also does not mean better use of it. Lost in the Middle (Liu et al.) showed models are most accurate when the relevant information is at the start or end of the input, and noticeably worse when it sits in the middle. A hundred tool descriptions put most of them in the middle.

The fix that keeps showing up is to retrieve tools instead of listing them all. RAG-MCP (Gan and Sun) retrieves relevant MCP tools before the model sees them. It cut prompt tokens by over 50% and raised tool selection accuracy from 13.62% to 43.13%. Anthropic's Tool Search Tool, with tools marked defer_loading, cut startup context by 85% and improved Opus 4 on their MCP evals from 49% to 74%, and Opus 4.5 from 79.5% to 88.1%.

But retrieval is harder than it looks. ToolRet (Shi et al.), with 7.6k tasks over 43k tools, found that retrievers that do well on normal search benchmarks do poorly on tools, and that bad retrieval lowers the agent's pass rate. So you cannot assume an off-the-shelf embedding model will pick the right tool.

05 Techniques that fit together

A per-user tool index. Index each tool's name, description and parameters with both BM25 and embeddings. BM25 catches exact names like "jira" or "create_issue". Embeddings catch paraphrases. Filter to the tools this user has connected and is permitted to use before ranking, not after. A tool the user cannot call should never reach the prompt.

Top-k with a token budget. Do not pick a fixed k. Add tools in rank order until a budget for definitions is spent. Five small tools and two huge ones cost different amounts.

Usage priors. If a user calls the same three tools every day, boost them. This is cheap and covers the cases where retrieval is weakest: short, vague requests.

A search_tools meta-tool. Retrieval will sometimes miss. Give the model one small tool that searches the full index and loads more definitions on demand. This is the deferred loading pattern. It differs from the recall tool in the paper: the model knows when it lacks a tool, because it cannot do the task. It does not know when it lacks an old observation.

Code for chained calls. When a task chains several calls, let the model write code instead. Cloudflare's Code Mode turns MCP tools into a TypeScript API run in a sandbox, on the argument that "LLMs are better at writing code to call MCP, than at calling MCP directly." Anthropic's code execution with MCP post reports one workflow going from 150,000 tokens to 2,000. Their programmatic tool calling cut average usage by 37% on complex research tasks.

Big results behind handles. Store a large tool result outside the context and give the model a short handle plus a preview. This is elision done at write time instead of later.

Cache the tool list. The 2026-07-28 MCP revision made the protocol stateless and added ttlMs and cacheScope to tools/list responses. So you can keep the index fresh without re-fetching every server on every request.

One more reason to keep the tool set small: tool poisoning, where malicious instructions hide in tool descriptions the model reads but the user never sees. Every description you load is text you are trusting. Loading fewer is a security choice too.

06 The design for a shared Slack agent

Here is how I would put this together for one Slack agent used by many people.

T4 is the default. Always kept verbatim: the user's user.md, the original request, and the last few turns. At 60% of the window, old MCP results are trimmed to their handle. At 85%, the old middle is summarized. There is no recall tool, since the paper suggests it would rarely be used. Full outputs go to S3, for audit only, not for the model.

def build_context(user, request, history, window):
    tools  = index.search(request, allowed=user.permitted_tools,
                          prior=user.usage, token_budget=TOOL_BUDGET)
    tools += [search_tools]                      # deferred loading
    pinned = [system_prompt, user.md, request]
    recent = last_turns(history, budget=0.3 * window, min_turns=2)
    middle = history - recent

    if tokens(pinned, tools, middle, recent) > 0.6 * window:
        middle = [handle_stub(e) if e.is_tool_result else e
                  for e in middle]               # cheap: no model call
    if tokens(pinned, tools, middle, recent) > 0.85 * window:
        middle = [summarize(middle)]             # one tool-free call

    return pinned + tools + middle + recent
    # full tool outputs already written to S3 by handle, audit only

Now the caveat. The paper tested coding agents on coding benchmarks with four models. A Slack agent serving many users over MCP is a different workload. Conversations are shorter and more scattered, tool results look different, and the tools change per user. The thresholds 0.6, 0.85 and 0.3 are a starting point, not a law.

So validate it the way the paper did: change one part of the harness at a time and measure success and cost together. If you change the tool retrieval and the context policy in the same week, you will not know which one helped.

The lesson I keep is a plain one. Do the cheap thing first, and do not add features the model will not use.