Harness-design findings from arXiv 2609.20804, measured against JP

Done2026-09-18jp

domain: conversation, llm, toolingtype: task

Promoted toW-00e1dvx

"An Empirical Study of Harness Design for Coding Agents" (https://arxiv.org/abs/2609.20804, Fan et al., September 2026) ablates three coding-harness components while holding the execution loop fixed: planning, action space, and context management. 176 matched settings, four models (Nemotron-3 30B/120B/550B, Mistral-Medium-3.5-128B), two benchmarks (SWE-Bench Verified, Terminal-Bench 2.1), four context-window budgets from 32k to 128k.

This ticket records what the paper establishes, how JP's harness compares, and where the comparison says JP should spend effort. The proposals it produced are filed separately and listed at the bottom.

What the paper establishes

1. Context management's value scales inversely with the window budget, and almost all of it comes from preventing overflow termination. Every managed tier overflowed on zero tasks at every budget. The unmanaged tier overflowed on 78.7% of SWE-Bench tasks at 32k and 8.7% at 128k. Accuracy differences between managed tiers are small; the difference between managed and unmanaged is not.
2. Staging cheap elision before expensive summarization wins on cost at equal accuracy. Their T4 (elide at a soft threshold, summarize at a hard one) had the lowest mean cost in seven of eight model/benchmark panels and the lowest peak-context ratio at all four budgets.
3. Model-facing recall is dead machinery. 56% of the settings exposing a recall_event tool never called it; the mean falls to 0.007 calls per task at 128k; all 16 ablation settings recorded zero. No accuracy gain over elision alone.
4. Planning and the action space are model-conditional. Planning is an accuracy scaffold for weak models (+11.6 points for the 30B) and a cost saver for strong ones (roughly 30% cheaper, about 2 points less accurate). Predefined tools scaffold bash-weak models; bash-only is cheaper and more accurate for bash-capable ones.

What it does not establish

• The action-space intervention is bundled: tool availability, interface prompts, file-state tracking, and post-edit diagnostics all vary together. The paper says so in its own limitations. It is not evidence that read-before-write or post-edit diagnostics help.
• One run per setting, and Terminal-Bench has 89 tasks, so most Terminal-Bench contrasts are not significant. Direction is usable; magnitude is not.
• Every crossover point sits at the weak end of its capability axis. JP's users run frontier models, where the paper's own data shows planning and predefined tools helping least.

How JP compares

Context management

Paper	JP	State
T0 (no management, overflow terminates)	the jp query path	shipped behavior
M1 elision (stub bulky stale tool observations)	ToolCallPolicy::Strip { request, response } with a PolicySpec over bound	exists, manual only
M2 recall (recall_event tool)	absent; --reset plus always-preserved raw events	correct by design
M3 summarization (running summary)	SummaryPolicy	exists, manual only
Soft / hard thresholds (0.6 / 0.85), verbatim recent window (0.3, floor 2)	none	RFD D50 proposes a single trigger_ratio
Per-result cap (24k chars)	none	RFD 115
Truncation fallback	window::truncate_to_fit	exists; two call sites, neither on the query path

JP has every mechanism the paper tested. It has no trigger.

Action space and safety

Paper	JP	Verdict
Compile-time choice: predefined set or bash-only	every tool is a TOML declaration under conversation.tools; one built-in (describe_tools)	JP stronger
Full description in the tool schema, always	summary in the schema, full description and examples via describe_tools	JP stronger
Permission: allow / ask / deny	RunMode::{Ask, Unattended, Edit, Skip} plus access grants (RFD 076)	JP stronger
Workspace guard, pre- and post-symlink	jp_tool::AccessPolicy, pre- and post-canonical	comparable
Read-before-write with a content hash	none	absent
Post-edit diagnostics	none	absent
Stuck detection (5 identical warn, 8 identical fail abort)	none	absent
Step budget (300 per task)	none; TurnState::request_count is incremented and never read	absent
Parallel read-only tools, capped at 8	all tools, in parallel, uncapped	different
Cost and token accounting	provider usage parsed into wire types and discarded	absent
Trajectories reconstructed from logs by an LLM judge	durable event stream, stable event IDs (RFD 097), turn markers	JP stronger

SWOT

Strengths

• The action space is a config axis, not a code axis. The paper's action-space finding is that the right tool set depends on the model, and their harness bakes the choice in at compile time. JP expresses either condition as config, per conversation and per persona. What the paper reports as a finding, JP already treats as a knob.
• Recall is a human operation, not a tool. RFD 064 made raw events recoverable by the user and never by the model. The paper's clearest negative result is that the model-facing version goes unused. JP put the boundary in the right place.
• Truncation is prompt-cache-aware. truncate_to_fit rounds its drop to 10% of target to keep the prefix stable across calls. The paper does not model caching at all, which makes its cost figures optimistic in shape for any harness that thrashes the cache.
• The trajectory is a first-class durable artifact. The paper spent an appendix and an LLM judge reconstructing what JP records natively.
• Human-in-the-loop is a designed axis. Inquiries (RFD 005, RFD 028), the interrupt ladder (RFD 045, RFD 092), the permission model. The paper's harness has one escape hatch: kill the run.

Weaknesses

• No context management on the query path. The failure the paper measures most directly, and T-0de0hry is in-repo evidence it already bites.
• Unbounded tool output. RFD 115 records a 1,293,623-token request against a 1,000,000 limit, durably persisted.
• No loop bound and no repetition detection. The turn loop cycles streaming to executing with no cap. The guards are a per-response byte ceiling, an idle timeout, and Ctrl-C.
• No usage accounting. Every proposal below is unfalsifiable without it. The paper's contribution is that it measured; JP cannot.
• The harness knows nothing about what its tools do. No concept of "a file was edited", so no read-before-write, no post-edit diagnostics, no per-turn touched-file set. This is the price of the declarative action space, and it is a real price.

Opportunities

• Adopt the paper's tuned constants instead of deriving them: 0.6 and 0.85 thresholds, a 0.3 recent-window budget floored at two turns, 5 and 8 streak thresholds.
• The recall result retires a question and saves the work.
• Per-model action-space presets are config in JP and impossible in the paper's harness.

Threats

• Model mismatch. Every crossover point in the paper sits at the weak end of its capability axis. In its own strong-model cells, planning reduces accuracy slightly and predefined tools add cost.
• Autonomy mismatch. The paper optimizes for finishing without a human. Importing its shapes wholesale imports that assumption. Stuck detection that kills a run is wrong for JP; stuck detection that asks the user is right.

What not to build

• A model-facing recall tool. The evidence against it is strong and JP's boundary is better. Worth checking RFD D39 is not recreating it under another name.
• A benchmark suite. RFD D65 is a test harness, which is a different thing and worth having. A SWE-Bench-style evaluation is not: effects are conditional on model capability, and JP's users change models faster than a suite stays calibrated. Instrument real sessions instead.
• A hard-coded file tool set to obtain read-before-write and post-edit diagnostics. The paper does not establish that those help, and hard-coding them collapses JP's best structural property. If the gap ever bites, the orthogonal move is a declarative effect annotation on tool configs, not a built-in tool set.

Proposals

Ordered by leverage per unit of cost. The first two attack the failure the paper measures most directly; the third is what makes any of them checkable.

1. T-0n0pcr0: promote 115 and rank bounded tool output. The cheapest change, and it sits upstream of the rest: a single oversized response poisons a conversation permanently, and no compaction trigger rescues one that has already happened.
2. T-0n0pjkk: stage elision before summarization in D50's automatic compaction. JP has both policies already; this is about which fires when, and about borrowing the paper's thresholds rather than guessing new ones.
3. T-0n0pgw6: record per-response token usage as a conversation event. Build alongside 2, not after. Without it, tuning a threshold is guesswork and no claim about the harness can be checked.
4. T-0n0pmyh: bound a turn's tool-call cycles and notice repeated identical calls. Small, and the vestigial TurnState::request_count names the missing check. JP should route a streak to the user rather than nag the model.
5. T-0n0ppbn: decide whether a plan belongs in conversation history. Weakest-supported of the set, and explicitly gated on 3. Filed to keep the question findable.

One gap fell out of 2 and is filed on its own, because it holds independently of whether automatic compaction ever lands:

• T-0n0pex2: the query path never fits a conversation to the model's context window. truncate_to_fit has two call sites and neither is query. D50 assumes this backstop exists.