RFD 117: Transparent Tool Call Fan-Out

• Status: Implemented
• Category: Design
• Authors: Jean Mertz git@jeanmertz.com
• Date: 2026-09-18
• Summary: A per-tool fan_out setting lets one tool call carry several independent operations, which JP's MCP server runs separately and folds into one response.

Summary

This RFD adds fan-out: a per-tool configuration flag that lets a single tool call carry several independent operations, which JP executes as if the assistant had issued them separately. The tool's own implementation is unchanged.

Fan-out lives entirely in JP's in-process MCP server (RFD 109), the one place every tool call passes through regardless of who submits it. The server advertises the tool with an envelope schema, expands a call's envelope into one child invocation per operation, runs them under the tool's concurrency and error policy, and folds their results into the one response the caller is waiting for. No LLM provider, and no MCP client, needs to know fan-out exists.

Motivation

An assistant that already knows which files it wants pays for one request-response cycle per call:

fs_read_file(path: "docs/pitch.md", start_line: 58, end_line: 68)
fs_read_file(path: "scripts/migrate.sh", start_line: 1928, end_line: 1940)
fs_read_file(path: "scripts/migrate.sh", start_line: 2072, end_line: 2082)
... 11 more

Each cycle re-sends the accumulated context. Cached input is roughly a tenth the price of uncached input, so ten cached cycles cost about one uncached request, but fourteen sequential cycles also cost fourteen units of latency and fourteen turns of the agent loop.

The obvious fix is to give each tool a multi-argument schema. That fix has two problems.

It does not reach MCP tools. jp_mcp::server::resolve_tool builds a ToolDefinition uniformly for local, built-in, and MCP sources; for MCP it takes the server's own inputSchema and applies configured overrides. JP does not own those schemas and cannot edit them. A per-tool fix helps only the tools JP wrote.

It entangles two axes. "What this tool does" and "how many operations fit in one call" want to vary independently. Hand-writing the second into every tool's schema means every new tool re-decides it, and every existing tool that wants it needs a schema change, a validation change, and a result-format change.

Fan-out makes the second axis a property of the tool configuration rather than the tool implementation, so it applies uniformly to every source, including MCP servers JP does not control.

Design

What the user sees

Nothing changes. Fan-out exists on the wire and is invisible in the terminal.

A call carrying three operations renders as three tool calls, prompts for permission three times unless the tool runs with run = "allow", and shows three results:

Calling tool fs_read_file(path: "docs/pitch.md", start_line: 58, end_line: 68)
Calling tool fs_read_file(path: "scripts/migrate.sh", start_line: 1928, end_line: 1940)
Calling tool fs_read_file(path: "scripts/migrate.sh", start_line: 2072, end_line: 2082)

A call carrying one operation, in the envelope or bare, renders as one tool call, indistinguishable from a call made without fan-out.

This invariant settles the permission question: whatever is rendered as a distinct call is approved as a distinct call. Collapsing three rendered calls into one approval prompt would ask the user to approve something other than what they were shown. The savings this RFD delivers are entirely in round-trips to the provider, not in user interaction, and the tools that dominate the cost (run = "allow" readers) prompt zero times either way.

The provider boundary

Fan-out is a property of the tool as JP's MCP server advertises it, not of any provider or client.

• Every caller learns a tool's shape from one place, Service::definitions(): JP's own provider request, MCP tools/list for Claude Code on the ACP route (RFD 110) or any other client, and the describe_tools built-in.
• Every call reaches the server the same way, whoever submits it, and the server expands it.

A provider sends the schema it is given, so jp_llm has no fan-out code, and jp_tool::ToolDefinition has no fan-out field. A route that bypassed the server for a fanned-out tool would be a bug in that route, not a case this design has to handle.

The envelope

When a tool opts in, the schema the server advertises is wrapped:

{
  "type": "object",
  "properties": {
    "ops": {
      "type": "array",
      "minItems": 1,
      "description": "The operations to perform. Each element is one complete set of this tool's arguments.",
      "items": { "<the tool's own schema>": "..." }
    }
  },
  "required": ["ops"],
  "additionalProperties": false
}

ops rather than arguments or args: each element is one operation, and the tool's existing parameter object is already "the arguments".

A $defs or definitions block moves from the tool's schema to the envelope's root. Same-document references are anchored at the root (#/$defs/Name), so leaving the block nested under properties.ops.items would point every reference at a root that no longer holds it. MCP servers routinely declare such blocks, and they are the tools fan-out exists to reach. A schema that refers to its own root ($ref: "#") is not rewritten.

The advertised description gains one sentence asking the assistant to batch every operation it already knows it needs into one call. It does not explain the tool's examples: each shows one bare operation, which is itself a valid call (see below).

Bare calls

A call to a fan-out tool whose arguments are not an envelope runs as an ordinary call, exactly as if the tool had no fan_out: no child invocations, no framing. What counts as an envelope depends on whether the tool's own schema declares a top-level ops parameter.

The tool does not declare ops (the common case). An ops key is always the envelope, and its absence is a bare call. An envelope that is not a non-empty array of objects, or whose operation fails validation, answers with an error naming the operation and what is wrong with it. Falling back to a bare call here would replace "operation 2: unknown argument pth" with "unknown argument ops".

The tool declares ops. The arguments are an envelope only if every check passes: ops is a non-empty array, every element is an object, and every element validates against the tool's own schema, with coercion and defaults applied to a copy. If any check fails, the ops is the tool's own and the call runs bare. When both readings validate (possible when the tool requires nothing, as in {"ops": [{"dry_run": true}]}), the envelope wins: it is the shape every caller was shown.

The advertised schema still requires ops. Strict tool use on OpenAI and Anthropic rejects a schema whose root is an anyOf, so a schema permitting both shapes at the root cannot be advertised to them, and merging the tool's properties into the envelope as optional would drop every required the tool declares. Bare calls are accepted, not advertised: a model or client that sends one gets a working call, and one that follows the schema batches.

Where expansion happens

Service::start_call receives a call. For a fan-out tool carrying an envelope it becomes a parent invocation, and each operation becomes a child invocation:

• A child is an ordinary invocation with its own InvocationId, its own cancellation token (a child of the parent's), its own attempts, and its own stderr progress stream. It runs the same run_call state machine a plain call does, against the tool's own schema, so coercion, defaults, validation, the argument formatter, and questions work per operation unchanged.
• The parent runs no tool. It starts the children, applies the concurrency and error policy, waits for every child to settle, folds the results, and owns the one Record barrier and the one MCP response.

The server re-executes a tool after an answered question. Because each operation is its own child, that re-runs only the child that asked; operations that already finished are never recomputed.

Two Host-side mechanisms carry over without modification:

• Inquiry ids do not collide. TurnState::next_inquiry_attempt increments per (tool_id, question_id), so two operations asking the same question under one tool call id get distinct attempts.
• "Answer once for the whole call" already works. Before prompting, the Host consults remembered_tool_answers, keyed by (tool_name, question_id). An answer given at PersistLevel::Turn resolves every later operation silently, each still recorded as its own inquiry pair.

Child invocations on the Host channel

A child talks to the Host through the same private channel and interactions as any call, with three additions.

Every interaction says which operation it is for. CallInfo gains operation: Option<Operation>, where Operation carries the parent's InvocationId, the operation's zero-based index, and the operation count. CallInfo::request stays the caller's original request, envelope included, so the Host's correlation check sees the same request for every child. The operation's own arguments arrive where they always do, in Prepare and Release.

A child ends with Settled, not Record. The conversation records one response per tool call, so a child is never recorded on its own. Its last interaction is a new Interaction::Settled, carrying a Recording of the operation: its executed arguments, its unedited result (absent if it never executed), and the result approved for delivery. The Host takes it and replies at once, so the parent can fold without waiting on the terminal.

Every child the service concludes sends exactly one Settled, including one a stop policy ruled out. A child the Host resolves itself with complete_call (as it does for Stop & respond) sends none, because the Host already holds its result.

The parent ends with Record, holding the folded body. Its result is also what the MCP caller receives, so what JP records and what it delivers are the same text. Each operation's own result already reached the Host in its Settled.

A plain call, including a bare call to a fan-out tool, has operation: None on every interaction and sees no change.

Scheduling

Every child prepares as soon as the parent starts it: it asks for argument rendering and for admission immediately, so the user approves every operation of a call before any of them runs, as with separate calls in one turn.

The policy sits between release and execution. Once the Host releases a child, the child waits for a concurrency slot before its tool runs. Under a limit, children take slots in the order the assistant wrote them, so concurrency = 1 runs them in that order whatever order the Host released them in. Once a slot is free, the parent checks the error policy: under on_error = "stop", a child behind a failed operation settles as not run instead of executing.

A child released to run reports its failure before it gives up its slot, so the next child cannot slip through. That covers a tool that ran and reported an error, and one that could not be run at all (a command that fails to spawn, an upstream MCP error). The tool's own result decides, before result-mode policy applies, so result = "skip" or a declined result = "ask" review cannot hide the failure. A child that ended without running (a Host decision, an argument the tool rejects) reports a failure from the result it ended with.

The parent waits on its children, not on the Host, so a stop that rules out every remaining operation ends the call instead of leaving it waiting for work that will never start.

What the Host does

The Host (jp_cli::cmd::query::tool) keeps its role: prompt, render, route questions, review, record. It neither schedules nor folds.

• The Host submits a fan-out call like any other tool call. It runs the same split the server does, only to open one executor per operation before the first interaction arrives, and refuses to route an operation the two sized differently.
• It demultiplexes child interactions by operation.index into one executor per operation. An operation's display state is keyed by <tool call id>#<index>, so each prompt, running state, and result belongs to its own line.
• Each operation is a call of its own in the coordinator's batch: announced, prompted for, and released in the order the assistant wrote it, like any call.
• Settled completes an operation's executor; the parent's Record completes the tool call and is what the conversation stores. The Host asks for that Record once every operation has a response, and counts the batch as settled only once it arrives.
• Every operation, whoever decided its outcome (the tool, the user at its prompt or result review, a cancellation), is resolved with Executor::settle, which answers whatever barrier the child is parked on. An operation cannot wait for the call's acknowledgement the way a plain call does, because the call is only recorded once every operation is resolved. Settling first makes sure the call's MCP request was sent: an operation settled before any was prepared would otherwise wait on a child the service never started.
• An operation waiting behind a concurrency limit is an executor whose release has not executed yet; the Host needs no queue of its own.

Folding results

The parent folds its children in the order the assistant wrote them, not the order they finished.

A fan-out call carrying one operation is recorded bare, success or failure, so at N=1 the envelope leaves no trace in the result. Otherwise the folded body frames each operation and states what did not run:

[1/5] ok
File deleted.

[2/5] error
File has uncommitted changes. Please stage or discard first.

[3/5] not run (stopped after operation 2 failed)
[4/5] not run (stopped after operation 2 failed)
[5/5] not run (stopped after operation 2 failed)

Without the "not run" lines the assistant assumes all five were attempted and reasons from a false premise.

The call is recorded as an error when no operation succeeded, such as one whose first operation failed and stopped the rest. One success is enough for it to count as a success; the framed sections name the operations that failed. The provider sees the same distinction it would for a bare call to the tool, and replay picks the error or the success style by it.

Interrupts

The Host addresses a fan-out call through its parent and each operation through its child:

• Stop & respond holds every unfinished child and completes each with the cancellation response, running or waiting for a slot. The parent folds those like any other result, so the call records one response saying which operations were cancelled.
• Restart pauses the unfinished children. Settled children keep their results; the others re-prepare on resume, and the Host re-renders and re-prompts only those.
• Escalating past the menu cancels the Host's attempts; each unfinished child ends as cancelled and nothing further starts.

Rendering and replay

Replay re-expands a recorded envelope for a tool whose turn configuration has fan_out, using the same expand the server does, so jp conversation print shows what was shown live. A recorded call without an ops key renders as one call. Replay has the configuration but not the tool's schema, so a bare call to a tool that declares its own ops parameter replays as operations.

Custom-formatter output is stored on the ToolCallRequest event, which carries the whole call. Each operation's rendered chunk is joined with a newline into one string, in the order the assistant wrote the operations, which keeps the stored value a string for conversations recorded before fan-out existed. An operation announced again after a restart replaces its chunk. Replay therefore shows every operation's header and then every description, where the live output put each description under its own header.

Configuration

fan_out accepts a bool or a table, following the precedent enable already sets in the same config tree:

# reads: unbounded concurrency, report every failure
[conversation.tools.fs_read_file]
fan_out = true

# writes: one at a time, in order, stop at the first failure
[conversation.tools.fs_delete_file.fan_out]
concurrency = 1
on_error = "stop"

# rate-limited remote calls
[conversation.tools.github_issues.fan_out]
concurrency = 4

Key	Default	Meaning
concurrency	unbounded	Maximum operations in flight. 1 runs them in order.
on_error	continue	stop starts no further operations after the first failure.

concurrency is one integer rather than a mode = "parallel" | "sequential" enum plus a later max_concurrency. Two knobs that can contradict each other (mode = "sequential", max_concurrency = 4) are one axis wearing two names. A future delay key for rate-limited endpoints slots in beside these without disturbing either.

on_error is genuinely independent of concurrency: independent reads want unbounded concurrency and continue, ordered writes want concurrency = 1 and stop, and both other combinations are reachable.

With concurrency above 1, stop means no further operations are started. In-flight operations finish; nothing is aborted.

The table form turns fan-out on without naming enabled; enabled = false turns it off. A concurrency of 0 reads as unbounded rather than "never run anything". fan_out is a per-tool key with no counterpart on the conversation.tools.* defaults block, so no tool inherits it: whether a tool's calls may be batched depends on what the tool does.

Where the safety boundary sits

An earlier version of this design gated fan-out on tools that never return NeedsInput, on the grounds that a question arriving mid-batch leaves earlier side effects committed.

That gate is unnecessary, and the property it was reaching for is better expressed by the config above. A tool configured with concurrency = 1 and on_error = "stop" behaves exactly as separate calls would: when operation 2 asks a question, operation 1 has committed and operations 3 through 5 have not started. That is the same state the assistant would have reached by issuing two calls.

The question is not "can this tool ask?" but "is this tool's fan-out ordered?", and the tool's configuration answers it.

Drawbacks

A second way to do the same thing. fs_modify_file already accepts many targets in one call, and bash accepts many commands. Fan-out does not replace those and does not subsume them (see Non-Goals), so the project carries two mechanisms that both mean "more than one thing per call".

The advertised schema is not the tool's schema. jp_tool::schema states that a tool's parameters are held exactly as the source declared them. That stays true of ToolDefinition, but what the server advertises for a fan-out tool is its own construction.

The Host protocol grows. CallInfo::operation and Interaction::Settled are new, and the Host has to demultiplex one MCP call into several executors. Every Host implementation carries that, not only JP's terminal one.

Replay loses operation boundaries in custom-formatter output. The chunks are joined into one string, so replay reproduces the printed text but cannot tell which chunk belonged to which operation.

Alternatives

Per-tool multi-argument schemas. Give fs_read_file a reads[] parameter, fs_create_file a files[] parameter, and so on. Rejected: it does not reach MCP tools, and it re-decides the same question in every tool. It also requires each tool to grow its own result-framing and partial-failure handling, which fan-out provides once.

Expansion in the Host. Show the provider the envelope, and have the Host expand a call into one MCP call per operation before submitting anything. This was the first implementation. Rejected: it depends on who submits calls. When an agent submits its own calls, the Host never sees them before the server does, so the agent can only be shown the tool's plain schema and fan-out is silently off on that route. It also records a folded body no MCP call delivered, and needs per-operation acknowledgement to reconcile the two.

One loop over operations inside a single invocation. Rejected: the server re-executes a tool after an answered question, so a loop inside one invocation re-runs committed side effects, and one invocation carries one Host approval, where fan-out needs one per operation. Child invocations keep both per operation.

Gate fan-out on tools that cannot ask questions. Rejected in favour of the concurrency and on_error configuration, which expresses the same safety property without excluding every write tool from the feature.

Non-Goals

Replacing batch operations. fs_modify_file is one operation over a set of targets, not N independent operations: its patterns apply in order across files and each sees the previous one's output, it shows a single diff for one approval, and it validates the whole set before writing anything. Tools shaped that way opt out by never setting fan_out.

Shared parameters across operations. Every operation carries its own complete argument object. Hoisting a common value to the call level (the way fs_modify_file has a call-level path default) is a possible later extension and is not designed here.

A delay knob. Rate-limited endpoints will want one. The config shape above leaves room for it; this RFD does not build it.

Changing what any tool does. No tool implementation changes. A tool that gains fan-out behaves identically per operation.

Risks and Open Questions

Does the assistant use it? A wrapped schema is a more complex schema. Some models may keep issuing one operation per call, which costs an extra envelope of tokens per call and delivers nothing. Enabling it on fs_read_file first and measuring the operation-count distribution answers this before the feature spreads; T-0yavtsr tracks that.

Permission fatigue on write tools. Fan-out plus a write tool that asks before running means N prompts. That is the honest behaviour, but it may make fan-out unattractive on exactly the tools where ordering matters most. The mitigation is the permission decision a user can remember for the rest of the turn, which approves the remaining operations without prompting. Whether that is enough is an observation to make in use.

Do clients validate against the advertised schema? An MCP client that checks arguments against inputSchema before sending would refuse a bare call the server accepts. That costs nothing a client following the schema would miss, but it means bare calls are a leniency of the server, not a promise to every caller.

Implementation Plan

Phase 1: Configuration

Add FanOutConfig to jp_config::conversation::tool with bool-or-table parsing, concurrency, and on_error. No behaviour yet.

Depends on: nothing. Mergeable: yes.

Phase 2: Advertisement

Add jp_mcp::server::fan_out with envelope, split, and fold, where split tells an envelope from a bare call as described under Bare calls. Have Service::definitions() advertise the envelope and the batching sentence, and have JP's provider request take its tool definitions from the same place. jp_tool and jp_llm carry no fan-out code.

Depends on: Phase 1. Mergeable: no; a tool advertised with the envelope needs Phase 3 to run.

Phase 3: Child invocations

Add CallInfo::operation and Interaction::Settled. Expand an envelope into child invocations under a parent, apply concurrency and on_error between release and execution, and fold into the parent's Record. Accept bare calls as plain calls.

Depends on: Phase 2. Mergeable: with Phase 4.

Phase 4: Host

Demultiplex child interactions into one executor per operation, key display state by operation, complete operations on Settled and the tool call on the parent's Record, and resolve every operation with Executor::settle. Route pause, resume, and cancel to the right invocation.

Depends on: Phase 3. Mergeable: with Phase 3.

References

• RFD 082 records every tool question round-trip as an inquiry pair, which is what keeps N per-operation questions individually auditable under one tool call id.
• RFD 109 runs every tool call through JP's in-process MCP server, which is where fan-out lives.
• RFD 110 lets an external agent submit tool calls itself; those calls reach the same server and fan out the same way.