Jean-Pierre

Appearance

View Markdown
References078096D10D31
Referenced byD31

RFD D71: Provider-Hosted Tools ​

Summary ​

Some LLM providers execute tools on their own infrastructure: Anthropic's web_search and code_execution, OpenAI's web_search, Google's Search grounding. This RFD exposes them as provider-hosted tools, configured inside each provider's own config block and translated entirely by that provider's adapter. Anthropic web search is the first vertical slice.

JP core gains no vocabulary for hosted tools.

Motivation ​

Users want current information in a conversation without wiring up a search MCP server, and the provider already offers it — Anthropic charges $10 per 1,000 searches and returns cited sources.

The obvious approach is to make these tools appear as another tool source alongside builtin, local, and mcp. That is wrong, and the reason matters enough to state up front.

Definition ownership and execution ownership are separate axes ​

ExampleSchema ownerExecutorAgent loop
Local JP toolJP / userJP subprocessJP
MCP toolMCP serverMCP serverJP
Anthropic bashAnthropicJPJP
Anthropic web_searchAnthropicAnthropicAnthropic
OpenAI web_searchOpenAIOpenAIOpenAI
Google SearchGoogleGoogleGoogle

Anthropic's own documentation splits these: bash and text_editor are client tools whose schema Anthropic supplies but which the caller executes; web_search and code_execution are server tools Anthropic runs itself.

A ToolSource::Llm variant would therefore mean opposite things about who dispatches, depending on which tool it named. RFD D10 already rejects this class of change — its Alternatives section rejects folding execution mechanics into ToolSource on the grounds that ToolSource answers "where does the definition come from?" and mixing in execution makes the enum a product of sources times runtimes.

The bash row is a legitimate future ToolSource case: provider-supplied schema, JP execution, every ToolConfig field meaningful. This RFD does not cover it, and does not foreclose it.

The web_search row is not a Tool Call at all.

Two execution planes ​

Client tool — the existing Tool Call:

text
assistant emits ToolCallRequest
    -> JP resolves policy and permissions
    -> JP dispatches a runtime
    -> JP records ToolCallResponse
    -> JP returns the result to the provider

Provider-hosted tool:

text
JP enables the facility on a request
    -> provider emits activity
    -> provider executes it
    -> provider continues generating
    -> JP records activity and assistant output

JP observes the second. It never dispatches it, and never fabricates a ToolCallResponse for it.

Design ​

Configuration lives in the provider block ​

toml
[providers.llm.anthropic.hosted_tools.web_search]
enable = true
max_uses = 5
blocked_domains = ["reddit.com"]

Typed, defined in jp_config::providers::llm::anthropic, validated by the Anthropic adapter, invisible to jp_llm core.

OpenAI's block has its own fields under its own names. No shared capability vocabulary is negotiated, because the knobs are not shared: max_uses has no OpenAI equivalent and search_context_size has no Anthropic one.

Consequences of this placement:

  • No source, no aliases, no fallback ordering. The provider serving the request is already selected by assistant.model.id, and the config key is scoped to that provider.
  • No run, result, access, questions, command, or parameters. None of them apply: there is no local execution and no approval point, because the work has already happened by the time JP sees it.
  • ChatQuery is unchanged. Anthropic is built via TryFrom<&AnthropicConfig> and convert_events already reads per-event provider config, so hosted-tool settings arrive through the existing path with no new plumbing.
  • A plugin provider needs nothing from core. Its config subtree is already opaque to JP; its hosted tools ride along inside it.

Hosted tools default to disabled.

The provider executes the work before JP can prompt, so per-call approval is impossible. enable = true is therefore blanket consent for that facility: its network access, its spend, and its provider-side execution behavior for every matching request.

This matters more for code execution and provider-side MCP connectors than for web search, and it is why the default is off.

The Anthropic adapter owns everything else ​

Request. convert_tools gains a branch emitting types::Tool::WebSearch(ToolWebSearch::WebSearch20250305(..)) with the configured knobs. The adapter selects the native wire version (web_search_20250305 through web_search_20260318) and records that selection in the payload, so a JP upgrade does not continue an activity under a contract it did not begin under. The native version is not user-configurable.

Validation. Enabling a hosted tool the active model or endpoint cannot serve is a hard error raised by the adapter before the request goes out — web search is unavailable on Bedrock, and dynamic filtering requires Claude 4.6 or later. Silently dropping it would let an A/B run compare a model with search against one without while claiming to compare models.

Response. map_content_start maps a server_tool_use block to a hosted-activity part. The result arrives as one complete web_search_tool_result block rather than deltas, so the adapter holds the activity open until the result lands and then emits it with the native payload attached. The user sees the activity line stay up while the search runs, which is the correct display.

Replay. convert_events reconstitutes both native blocks from the persisted payload — server_tool_use and web_search_tool_result, with encrypted_content byte-identical, since Anthropic rejects a modified or missing value. An activity whose payload is absent has neither block emitted.

Continuation. pause_turn joins the existing max-token chaining inside the adapter: resend the paused assistant message unchanged until the turn reaches a JP-visible boundary.

Every one of these lives in anthropic.rs. Core sees only the normalized hosted-activity event from RFD D31.

Mixed client and hosted calls ​

When Claude calls web search and a client tool in the same parallel batch, the API returns stop_reason: "tool_use" and does not run the search yet:

  1. JP persists the hosted activity in started state with its native payload.
  2. JP emits ToolCallRequest events for the client tools only.
  3. JP executes those normally.
  4. On the next request the adapter replays the pending activity and the client results in the native order Anthropic requires.
  5. Anthropic runs the deferred search and resumes.

The core invariant holds throughout: every ToolCallRequest is work JP must dispatch.

Continuation is stateless; abandonment is explicit ​

Step 4 reads from the conversation stream, not adapter memory. The payload is persisted, so the adapter reconstitutes it on every request — exactly as it does for thinking signatures today. Retry, restart, and resumption after interrupt all rebuild the request from the stream, and adapter-held state would break all three.

RFD 078 lets a tool change config between cycles, including the active model, so a pending activity can meet a config that cannot continue it. Two cases:

  • Same provider, declaration changed. The adapter keeps declaring the tool for as long as it holds a pending activity, reading its own persisted payload. The freshly resolved config does not silently drop a declaration the transcript depends on.
  • Provider switched. The new adapter preserves and ignores a payload it did not write (RFD D31's contract), so the request is valid and the activity stays started with no terminal transition — an honest record that a search began and never finished.

In both cases the adapter abandons explicitly, recording the abandonment, rather than erroring mid-turn or silently corrupting the provider transcript. No config delta is rejected, and no capability RFD 078 promises is blocked to protect a transcript detail.

tool_choice must not broaden silently ​

Hosted tools go into the same native tools array as client tools, which is the wire shape. Two existing sites in anthropic.rs change behavior as a result, and both must be handled:

  • convert_tool_choice maps ToolChoice::Required to Anthropic's any, meaning "use one of the provided tools." Once web_search is in that array, a user who asked the model to call one of their tools can have that satisfied by a search instead.
  • The documented single-tool quirk workaround (tools.len() == 1 && matches!(tool_choice, ToolChoice::Function(_)) downgrading to Required) stops firing when a hosted tool makes the array length 2.

Policy: assistant.tool_choice stays scoped to JP-dispatched tools. The adapter ensures the native request does not extend it to hosted tools, and forcing a hosted tool is unsupported.

Activation ​

Hosted tools sit outside conversation.tools, so -T, --no-tool, and tool groups do not apply to them — correctly, since they are not tools.

For a single query, --cfg carries the override. The full path is long, so the documented form uses a config fragment resolved against config_load_paths:

sh
jp query --cfg providers.llm.anthropic.hosted_tools.web_search.enable=true "..."
jp query --cfg websearch "..."   # a websearch.toml fragment

No new flag. If the pattern proves common enough, a dedicated selector can be added later.

Rendering ​

Terminal output shows the activity while it runs and its outcome afterward:

text
⚒ web search: "rust async trait object safety"
  3 results

Citations are a release gate, not a follow-up. Anthropic, OpenAI, and Google all impose attribution or display obligations on search-backed output. Rendering search-backed assistant text without its citations is not an incomplete feature, it is a non-compliant one. The typed attributions from RFD D31 are rendered inline or as a source list, and web rendering in the serve-web plugin follows the same rule.

Hosted-activity spend is shown as the number of searches in the turn, counted from the recorded HostedActivity events. RFD D31 persists no usage counters, so the provider's own count (usage.server_tool_use.web_search_requests) is a diagnostic, not a display source.

Drawbacks ​

Two configuration idioms for things users think of as tools. Someone who enabled web search will look for it under conversation.tools and not find it. The split is honest — they are different execution planes — but it is a real learning cost, and the ubiquitous-language entries are the mitigation rather than a fix.

Switching provider silently changes capability. Configure Anthropic's web search, run a query on OpenAI, and that query has no search. This is deliberate: under this placement it is an absent capability on an unconfigured provider, not a degraded explicit setting, which matches how reasoning support already behaves. adaptive_effort warns when it clamps an explicitly requested level; nothing warns when a model simply does not reason.

--cfg providers.llm.anthropic.hosted_tools.web_search.enable=true is long. A direct cost of provider-scoped placement, mitigated by config fragments rather than removed.

Per-provider configuration duplicates intent. Wanting web search on both Anthropic and OpenAI means two config blocks. The knobs genuinely differ, so some duplication is inherent, but the enable flag is duplicated for no reason other than placement.

Alternatives ​

ToolSource::Llm plus a no-op executor. Rejected. llm.bash and llm.web_search would differ in who executes, so one source variant cannot describe both; and it requires exceptions in the turn loop, the execution plan, and orphan sanitization. RFD D10 rejects this class of change explicitly.

assistant.request.hosted_tools.<capability>.<provider>, with a provider-agnostic capability key and per-provider sub-blocks. Genuinely attractive: one enable per capability, symmetric A/B arms, and the tool stays enabled across a model switch. Rejected because core would have to hold a capability namespace and a ChatQuery.hosted_tools field, and a plugin provider's hosted tools would need names core recognizes. Provider-scoped placement is a step toward providers-as-plugins; this is a step away. The earlier arguments for it — that provider config is not conversation-scoped, and that grants would sit next to credentials — do not hold: PartialAppConfig::delta covers providers, the tree holds only api_key_env rather than a secret, and RFD 078 grants are per-path rather than per-section.

A provider_alias map on the tool entry, or a provider-side tools.<native> = "<jp name>" map. Rejected with the tool-entry model itself. The provider-side form also reads backwards: understanding a tool would mean grepping every provider block for a reverse mapping, and it puts JP-namespace names inside a block that should describe the provider.

Ordered fallback variants — one tool name resolving to a provider-hosted search, else an MCP search, else a local one. Rejected here. Their schemas, invocation protocols, pricing, security posture, and citation obligations all differ, and one runs inside the provider's loop while the other stops it for JP execution. Ordered fallback would hide those differences behind a single name. Capability routing is a separate feature; issue #208 covers adjacent multi-exposure requirements and should not be mixed in.

Automatic discovery via a Provider::executed_tools() capability list. Rejected as premature — it exists to serve a discovery caller this design does not have. Typed fields inside each provider's config block give in-tree providers better diagnostics, and a plugin provider's own schema covers the open case without core arbitrating.

Silently dropping an unsupported hosted tool with a warning, mirroring how an MCP tool whose server is down is skipped. Rejected: see Validation above.

Non-Goals ​

  • Provider-supplied client tools (Anthropic bash, text_editor, computer_use). Provider-owned schema, JP execution — a ToolSource question, deliberately separate.
  • Forcing a hosted tool via assistant.tool_choice.
  • Fallback or capability routing across hosted, MCP, and local implementations of one capability.
  • A shared cross-provider capability vocabulary. Each provider names its own.
  • Provider-hosted tools beyond web search in the first slice. OpenAI or Google follows as the test that the boundary generalizes.

Risks and Open Questions ​

  • The boundary is unproven until a second provider lands. If adding OpenAI web search forces changes to RFD D31's normalized activity shape, the normalization was derived from too few examples. Finding that with one implementation in hand is the point of doing them in this order.
  • Google may not fit started | completed | failed. Search grounding may expose only completed metadata with attribution ranges and required Search Suggestion content, with no call/result pair.
  • Deferring the flush of a hosted activity until its result arrives relies on the result being a single complete block. If Anthropic ever streams result content in deltas, the adapter needs a different accumulation strategy.
  • Whether pause_turn interacts with max-token chaining in ways the existing should_chain logic mishandles needs testing against a long search turn.
  • Prompt injection. Web search pulls untrusted content into context. RFD 096 covers terminal sanitization; whether hosted results need handling beyond that is open.

Implementation Plan ​

Requires RFD D31 phases 1 through 4.

Phase 1: Typed configuration ​

hosted_tools on AnthropicConfig with the web-search fields, defaulting to disabled. Schema, delta, fill, and partial support. Independently mergeable; nothing reads it yet.

Phase 2: Request translation and validation ​

convert_tools branch, native version selection, model and endpoint validation, and the two tool_choice corrections. At this point the tool is declared and the model can call it, but the response path drops the blocks.

Phase 3: Response path ​

server_tool_use mapping, deferred flush with payload attachment, convert_events replay of both native blocks, pause_turn continuation, and the abandonment rules. This is the slice that makes web search work end to end.

Phase 4: Rendering and usage ​

Activity display, citation rendering in terminal and serve-web, and the per-turn search count. Gates the release, per Rendering above.

Phase 5: Second provider ​

OpenAI or Google, as the test that provider-owned translation generalizes without touching core.

References ​