Jean-Pierre

Appearance

View Markdown
ReferencesD12066

RFD D24: Bounded Tool Output ​

Summary ​

Cap the size of a tool call response before it is delivered to the assistant and persisted to the conversation stream: a configurable per-tool size_threshold resolved through conversation.tools, plus a non-configurable hard ceiling that applies to every response regardless of source.

Motivation ​

A tool call response is unbounded today. Nothing between a tool's stdout and the provider request enforces a size limit — not the tool, not the executor, not the coordinator.

A single oversized response does more than waste tokens. commit_tool_responses writes it into the stream and calls conv.flush(), so it is durably persisted. Every subsequent turn re-sends it and gets the same prompt is too long error from the provider. The conversation is unusable until someone edits the stream by hand.

This is not hypothetical. A panicking #[derive(Config)] produced one diagnostic per expansion site; cargo_test embedded the whole stderr in its error message and the resulting request was 1,293,623 tokens against a 1,000,000 limit.

Bounding output inside each first-party tool (already done for the cargo_* tools) does not solve this. It covers only the tools we wrote. MCP servers, user-defined local tools, and the responses the coordinator synthesizes itself remain unbounded.

Design ​

Configuration ​

One new key, resolved per-tool then from the '*' defaults, exactly like run, result, and cancellation_response:

toml
[conversation.tools.'*']
# Maximum size of a single tool response delivered to the assistant.
# Accepts human-readable sizes ("512KB", "1MB"), a bare byte count, or
# "unlimited".
size_threshold = "256KB"

[conversation.tools.some_mcp_tool]
# Raise it where the large payload is the point of the tool.
size_threshold = "1MB"

size_threshold is an upper bound, not a floor. A tool that caps its own output lands below the configured value and raising the threshold does not recover what the tool already discarded — so cargo_expand's local MAX_EXPANDED_BYTES is raised or dropped when this lands, and first-party tools keep local caps only where they are tighter than any threshold a user would set.

Oversized content is cut at a UTF-8 character boundary and a marker is appended:

... [truncated, 623 KB → 256 KB]

size_threshold accepts a new ByteSize type in jp_config::types, and ToolConfigWithDefaults::size_threshold() resolves it.

Two layers ​

The cap is applied at two seams, for two different reasons.

SeamScopeConfigurable
ToolCoordinator, on every write to results[index]per-tool size_threshold, per responseyes
commit_tool_responseshard ceiling, whole batchno

The coordinator is where ResultMode already dispatches: it has tools_config in hand and is on the path that MockExecutor takes, so it is reachable from the turn-loop integration tests.

A response reaches results[index] by two routes, and the cap belongs on both:

  • handle_tool_result stores it directly for ResultMode::Unattended and Skip.
  • ResultMode::Ask and Edit prompt in a spawned task and return the final value as ExecutionEvent::ResultModeProcessed, whose handler assigns results[index] itself.

A single cap_response(&ToolConfigWithDefaults, ToolCallResponse) -> ToolCallResponse helper called from both routes keeps the two in step. Naming only the first route leaves Edit as a bypass: the user pastes 5 MB into the editor and it lands in the stream uncapped.

Because the cap sits on the store rather than earlier, render_result and the edit prompt both receive the raw response. The terminal's full-content temp file and the user's editor keep the tail the assistant does not get, which is the whole point of calling this a delivery limit.

The commit_tool_responses seam is the last stop before the write. Without it the configurable cap is a suggestion: it would not hold for size_threshold = "unlimited", for an MCP server that ignores everything, or for the responses the coordinator builds itself (unavailable tool, orphan synthesis, inquiry failure) which have no tool config to read.

This ceiling is a budget over the whole batch, not a per-response limit. commit_tool_responses persists a vector whose length is whatever the model emitted, so a per-response ceiling leaves the total unbounded — two 2 MB responses already exceed the context window that produced the reported failure. The budget (MAX_TOOL_RESPONSE_BATCH_BYTES, 2 MB) is spent across the batch by truncating the largest responses first, so one oversized response does not starve its siblings, and a warn! names each tool that was cut.

This is a delivery limit, not a display limit ​

style.inline_results already truncates results, and its documentation promises that "the full tool call results will be sent back to the assistant, regardless of this setting." That is a terminal-rendering axis. size_threshold is a delivery axis. The two stay separate keys: collapsing them would break the reader who sets inline_results = 20 for a quiet terminal and expects the model to still see everything. That documented promise becomes "up to size_threshold" once this lands.

Bytes, not tokens ​

Tokens are what actually bind, but a per-provider tokenizer is a real dependency for a guard that only needs to be approximately right. Byte size is deterministic, free to compute, and provider-agnostic; for prose and code it tracks token count within roughly a factor of two. The draft attachment size policy reached the same conclusion for the same reason.

The naming (size_threshold, human-readable sizes, byte-based comparison) is shared with that draft deliberately. Tool output and attachment content are the same problem — untrusted content of unknown size entering the context — and should not end up with two parallel vocabularies. Whichever ships first defines ByteSize; the other adopts it.

Drawbacks ​

  • A truncated result can be a broken result. Cutting a JSON or XML payload mid-structure yields something the assistant may fail to parse, where the untruncated response would have worked. The marker tells it what happened, but the turn is still degraded.
  • Head-only truncation discards the conclusion. Test output and logs often put the useful summary last. This cut keeps the beginning.
  • Two caps to reason about. A user hitting the ceiling with a raised size_threshold gets truncation they explicitly configured against, and has to find the warning to understand why.
  • The batch budget makes one tool's size depend on its siblings. The same tool returning the same output is truncated in a busy cycle and not in a quiet one. That is the price of bounding the total, but it does make the ceiling non-deterministic from any single tool's point of view.
  • Bytes are the wrong unit for the actual constraint. A response under the threshold can still overflow a small context window; a base64 blob costs far more tokens per byte than prose.

Alternatives ​

Cap inside each tool only. What the cargo_* tools do now. Best truncation quality, because the tool knows which end matters — but it cannot cover MCP servers or user-defined tools, which is where the risk actually lives. Kept as a complement, not a substitute.

Cap in ToolDefinition::execute (jp_llm). The single convergence point for local, mcp, and builtin sources, and it already receives ToolConfigWithDefaults, so the cap would need no new plumbing. Rejected because MockExecutor and TestExecutorSource bypass it entirely: none of the turn-loop integration tests would exercise the cap.

Token-based limits. Correct unit, disproportionate cost. See "Bytes, not tokens."

A size_policy with an ask variant, mirroring the attachment size policy. Prompting mid-turn to approve an 800 KB result is a poor interaction, and ResultMode::Ask already covers "let me look before this goes to the model."

No configuration, ceiling only. Simpler, and it does fix the reported bug. Rejected because the useful thresholds differ by an order of magnitude between tools: a diagnostic dump wants tens of kilobytes, cargo_expand legitimately wants megabytes.

Non-Goals ​

  • Spilling the full output somewhere the assistant can read it. Handing back a path the assistant can fs_read_file is the natural follow-up and is orthogonal to the cap. The renderer already writes results to a temp file, but only for the terminal's OSC 8 link. Doing it properly needs a stable, content-addressed location — RFD 066's territory.
  • Head-and-tail or content-aware truncation. A plain head cut first; a second strategy needs a case behind it.
  • size_policy variants (ask, reject, allow). Deferred, not rejected — the threshold is the same concept if they arrive.
  • Bounding request arguments, attachments, or assistant messages. This RFD covers tool responses only.
  • Unifying the marker text with .config/jp/tools. That crate is project maintenance tooling, not part of the main codebase; its in-tool markers stay as they are. Phase 4 changes one cap value there, not the wording.

Risks and Open Questions ​

  • What default? 256 KB (roughly 64k tokens) is generous enough that no first-party tool reaches it and tight enough to stop the reported failure. A tighter 64 KB would catch more real waste but would silently start truncating cargo_expand and large git_diff_commit output — a behavior change users would notice. Tightening later is easy; loosening after complaints is not.
  • Is 2 MB the right ceiling? It has to be well above any plausible size_threshold while staying below the smallest context window we support. Worth checking against the actual per-provider limits before settling.
  • Largest-first apportioning is a guess. Truncating the largest responses until the batch fits is the obvious rule, but an even split or a proportional one may read better in practice. Worth deciding against a real multi-tool cycle rather than in the abstract.
  • Conversations already carrying an oversized response are not repaired by this RFD. They still need manual stream editing.
  • Hyrum's Law on the marker text. Once the marker string is in tool responses, assistants and user scripts will match on it. It should be settled before the first release, not iterated on.

Implementation Plan ​

Phase 1 — config types. ByteSize in jp_config::types (parses "512KB", "1MB", a bare integer, "unlimited"), size_threshold on ToolsDefaultsConfig and ToolConfig with the usual AssignKeyValue, PartialConfigDelta, FillDefaults, and ToPartial impls, and a resolver on ToolConfigWithDefaults. Pure config, no behavior change. Mergeable alone.

Phase 2 — the configurable cap. Add cap_response and call it from both writes to results[index]: the tracked_response assignment in handle_tool_result, and the assignment in the ExecutionEvent::ResultModeProcessed handler. Two turn-loop integration tests: a MockExecutor returning an oversized result under ResultMode::Unattended, and an oversized result edited under ResultMode::Edit, each asserting the persisted response is capped and carries the marker. The second test is the one that fails if only the first route is capped. Depends on phase 1.

Phase 3 — the batch ceiling. MAX_TOOL_RESPONSE_BATCH_BYTES in commit_tool_responses, apportioned largest-first across the response vector, with a warn! per truncated tool. Two tests: one response over the budget with size_threshold = "unlimited" configured, and several responses individually under it that exceed the budget together. Independent of phase 2, but reviewed after it so the interaction between the two caps is visible in one place.

Phase 4 — raise the tool-local caps. Drop or raise MAX_EXPANDED_BYTES in cargo_expand so the central threshold is the binding limit for that tool. Small, and only meaningful once phase 2 has landed.

Phase 5 — documentation. Correct the InlineResults doc comment, which promises full delivery to the assistant, and document size_threshold in docs/configuration.md.

References ​

  • RFD 066 — content-addressable blob store, the eventual home for full-output retrieval.
  • D12-large-attachment-size-policy (draft) — the attachment-side size policy this RFD shares its vocabulary with.