# RFD 033: Cache-Preserving Inquiry via Tool Use - **Status**: Superseded - **Superseded by**: [RFD 034] - **Category**: Design - **Authors**: Jean Mertz - **Date**: 2026-03-07 ## Summary This RFD replaces the structured output mechanism (`output_config.format`) used by the inquiry system ([RFD 028]) with a strict tool use approach. A built-in `answer_inquiry` tool is always present in the tool list and is invoked via `tool_choice: Function("answer_inquiry")` during inquiries. This eliminates two sources of prompt cache invalidation that currently cause complete cache misses on every inquiry request. ## Motivation The inquiry system ([RFD 028]) makes a separate LLM call when a tool needs input from the assistant. These calls use structured output (`output_config.format`) to guarantee the response matches a JSON schema. In practice, every inquiry request triggers a **complete prompt cache miss** on Anthropic, rewriting ~95k tokens at 125% cost instead of reading them at 10% cost. Two issues cause this: ### 1\. Empty tool list The inquiry backend sends `tools: vec![]` while normal requests include the full tool definitions. Anthropic's prompt cache prefix follows the hierarchy `tools → system → messages`. Removing the tools section changes the prefix at the top level, invalidating everything. This is a straightforward bug, fixed by passing the same tool definitions to the inquiry backend (merged separately). But fixing it alone is not sufficient because of issue 2. ### 2\. Structured output invalidates the system cache Anthropic's structured output feature injects an additional system prompt describing the expected output format. From the [Anthropic docs][cache-docs]: > When using structured outputs, Claude automatically receives an additional > system prompt explaining the expected output format. > Changing the output\_config.format parameter will invalidate any prompt cache > for that conversation thread. Even with matching tool definitions, the inquiry request gets a system-level cache miss because the system content differs (injected structured output prompt). With ~95k tokens of conversation context, each inquiry wastes roughly: ``` Cache write: 95,000 × $5.00/MTok × 1.25 = $0.59 Cache read: 95,000 × $5.00/MTok × 0.10 = $0.05 Waste per inquiry: ~$0.55 ``` For turns with 3 inquiries (common with multi-file modifications), that is ~$1.65 in avoidable cost per turn, plus the latency of reprocessing the full context. ### Cross-provider applicability While the data above is from Anthropic, other providers with prefix-based prompt caching (OpenAI, Google) are likely to exhibit the same behavior: structured output changes the effective request shape, busting the cache. A solution that works within the existing tool use mechanism avoids this class of problem across all providers. ## Design ### Overview Replace `output_config.format` with a built-in `answer_inquiry` tool that uses `strict: true` for schema validation. The tool is always present in the tool definitions — even when no inquiry is active — so it never changes the tools prefix. During an inquiry, the backend sets `tool_choice: Function("answer_inquiry")` to force the LLM to call it. Cache impact per the [Anthropic docs][cache-invalidation]: | Component | Changes during inquiry? | Cache impact | | ---------------- | ------------------------------- | ------------------- | | Tool definitions | No (always present) | ✓ Cache hit | | System prompt | No (`output_config` not used) | ✓ Cache hit | | `tool_choice` | Yes (`Auto` → `Function(...)`) | Message blocks only | | Messages | Yes (inquiry question appended) | Expected miss | The only cache miss is on the new message content, which is unavoidable and minimal. ### The `answer_inquiry` tool A generic tool with a deliberately simple, stable schema: ```json { "name": "answer_inquiry", "description": "Answer a question from a tool that needs additional input. Call this tool when the system indicates a tool requires your input. Use the inquiry_id from the question and provide your answer as a JSON string.", "strict": true, "input_schema": { "type": "object", "properties": { "inquiry_id": { "type": "string", "description": "The inquiry ID from the question prompt." }, "answer": { "type": "string", "description": "Your answer, formatted as instructed in the question prompt." } }, "required": ["inquiry_id", "answer"], "additionalProperties": false } } ``` The `answer` field is always a string. Type-specific constraints (boolean values, select options) are communicated in the question prompt and validated after extraction. ### Why `answer` is a string A per-inquiry typed schema (boolean, enum, etc.) would require changing the tool definition per inquiry, which invalidates the tools cache — the exact problem we're solving. A stable, generic schema means the tool definition never changes. The tradeoff is that validation moves from the provider (constrained decoding) to our code (post-hoc parsing). For the three answer types: | Type | Prompt instruction | Validation | | ------- | ----------------------------------- | ---------------- | | Boolean | "Answer exactly `true` or `false`." | Parse as bool | | Select | "Answer with one of: A, B, C." | Check membership | | Text | "Answer with free-form text." | Accept as-is | ### Inquiry flow The revised flow in `LlmInquiryBackend::inquire`: ``` 1. Build ChatRequest with question text + formatting instructions (no schema attached — schema field is None) 2. Build ChatQuery with: - tools: self.tools (same as parent turn — includes answer_inquiry) - tool_choice: ToolChoice::Function("answer_inquiry") 3. Send to provider via collect_with_retry 4. Extract tool call arguments from response 5. Parse answer.answer as the expected type 6. On validation failure: retry with error feedback (up to N attempts) ``` ### Answer extraction and validation The response is a `ToolCallRequest` instead of a structured text response. Extraction pulls `inquiry_id` and `answer` from the tool call arguments: ```rust fn extract_answer( tool_call: &ToolCallRequest, expected_id: &str, answer_type: &AnswerType, ) -> Result { let args = &tool_call.arguments; // Validate inquiry_id let id = args.get("inquiry_id") .and_then(Value::as_str) .ok_or(InquiryError::AnswerExtraction { reason: "missing inquiry_id".into() })?; if id != expected_id { return Err(InquiryError::AnswerExtraction { reason: format!("id mismatch: expected '{expected_id}', got '{id}'") }); } // Parse answer string into expected type let raw = args.get("answer") .and_then(Value::as_str) .ok_or(InquiryError::AnswerExtraction { reason: "missing answer field".into() })?; match answer_type { AnswerType::Boolean => match raw { "true" => Ok(Value::Bool(true)), "false" => Ok(Value::Bool(false)), _ => Err(/* retry-eligible error */) }, AnswerType::Select { options } => { if options.contains(&raw.to_string()) { Ok(Value::String(raw.to_string())) } else { Err(/* retry-eligible error */) } }, AnswerType::Text => Ok(Value::String(raw.to_string())), } } ``` ### Retry on validation failure Since the schema is generic (string answer), the LLM might return a malformed answer — e.g., `"yes"` instead of `"true"` for a boolean. The backend retries up to 2 times, appending the error as a user message: ``` Turn 1 (inquiry): User: "A tool requires input. Answer true or false: Create backup?" Assistant: answer_inquiry(id="...", answer="yes") ← invalid Turn 2 (retry): User: "Invalid answer 'yes'. Must be exactly 'true' or 'false'." Assistant: answer_inquiry(id="...", answer="true") ← valid ``` Each retry reuses the same cached prefix (tools + system + prior messages), so only the new error message is a cache write. ### Tool registration `answer_inquiry` is registered as a `BuiltinTool` alongside `describe_tools`. It is always included in the tool definitions when any tools are enabled. The tool's `execute` method is never called during normal tool execution — it only serves as a schema carrier for the LLM. If the LLM calls it outside an inquiry context, the executor returns an error message. ```rust // In BuiltinExecutors setup: executors.register("answer_inquiry", AnswerInquiryTool); // AnswerInquiryTool::execute always returns an error — it should // only be called via the inquiry backend, not the normal tool loop. ``` The tool definition is constructed in `jp_llm::tool::builtin` and added to the tool list by the query command alongside `describe_tools`. ### Changes to `LlmInquiryBackend` The backend no longer needs to set a schema on the `ChatRequest`. Instead: 1. The `ChatRequest.schema` field is `None`. 2. The `ChatQuery.tool_choice` is `ToolChoice::Function("answer_inquiry")`. 3. The response is processed as tool call events instead of structured text events. The `InquiryBackend` trait and `MockInquiryBackend` are unchanged — the trait returns `Value` regardless of the underlying mechanism. ## Drawbacks - **Weaker type safety.** Structured output guarantees the response matches the schema via constrained decoding. Tool use with a generic string answer relies on prompt instructions and post-hoc validation. In practice, boolean and select answers are simple enough that LLMs get them right on the first attempt the vast majority of the time, and the retry mechanism handles the rest. - **Extra retries on malformed answers.** A structured output response never needs a retry for schema violations. The tool use approach may occasionally need 1 retry (estimated \<5% of inquiries based on the simplicity of the answer types). Each retry is cheap (cache hit + small message delta). - **Always-present tool.** `answer_inquiry` appears in every request even when no inquiry is happening. This adds a small constant to the tool definitions token count (~50-100 tokens). The LLM may occasionally try to call it unprompted, though this is mitigated by the description making its purpose clear and the executor returning an error. ## Alternatives ### Keep structured output, pass same tools This is the partial fix already implemented: pass the same tool definitions to the inquiry backend to avoid the empty-tools cache bust. However, the structured output system prompt injection still causes a system-level cache miss on every inquiry. For conversations with ~95k tokens of context and multiple inquiries per turn, the cost is substantial. ### Provider-specific strategy selection Choose between structured output and tool use based on provider caching heuristics — e.g., use structured output for providers without prefix caching, tool use for those with it. Rejected for now: adds complexity with unclear benefit. All major providers (Anthropic, OpenAI, Google) use some form of prefix-based caching, and structured output is likely to bust the cache on all of them. If a provider is found where structured output doesn't affect caching, this can be revisited. ### Per-inquiry dynamic tool schema Define the `answer` parameter with the exact type for each inquiry (boolean, enum, string) instead of a generic string. Rejected: changing the tool schema per inquiry changes the tools prefix, which invalidates the tools cache — the same problem as the current approach, just at a different level. A stable, generic schema is the whole point. ## Non-Goals - **Batching multiple inquiry questions into a single tool call.** The `answer_inquiry` tool answers one question at a time. Batching is orthogonal and can be layered on later. - **Rendering inquiry tool calls in conversation output.** The inquiry remains invisible to the user (same as today). - **Replacing structured output for non-inquiry uses.** The `schema` field on `ChatRequest` and `output_config.format` support remain for other features (e.g., scriptable structured output via `jp query --schema`). ## Risks and Open Questions - **LLM calling `answer_inquiry` unprompted.** If the LLM calls the tool outside an inquiry context, the executor returns an error and the turn continues normally. The tool description should be clear enough to prevent this in practice, but it should be monitored. - **Answer parsing edge cases.** The LLM might return `"True"` or `"TRUE"` instead of `"true"`. The parser should be case-insensitive for booleans. For selects, exact match is required (the prompt lists the exact options). - **Interaction with `ToolChoice::Function` and reasoning.** Anthropic does not support extended thinking when `tool_choice` forces a specific tool. The inquiry backend currently does not enable reasoning, so this is not a problem today. If reasoning is later enabled for inquiries, this constraint will need to be handled (the existing soft-force fallback in `create_request` already covers this case). ## Implementation Plan ### Phase 1: `answer_inquiry` builtin tool Add `AnswerInquiryTool` to `jp_llm::tool::builtin`. Implement the `BuiltinTool` trait (executor returns an error — it's only called via the inquiry backend). Add the `ToolDefinition` constructor. Register it in the builtin executors alongside `describe_tools`. Can be merged independently. ### Phase 2: Always include `answer_inquiry` in tool definitions Wire the `answer_inquiry` definition into the tool list construction in `jp_cli::cmd::query`. Ensure it's present whenever tools are enabled. Verify that the tool appears in the API request and doesn't change token counts unexpectedly. Can be merged independently. ### Phase 3: Rewrite `LlmInquiryBackend` to use tool calls - Remove `schema` from the inquiry `ChatRequest`. - Set `tool_choice: ToolChoice::Function("answer_inquiry")`. - Process the response as tool call events instead of structured text. - Add answer parsing (string → bool/select/text) with validation. - Add retry loop (up to 2 retries) with error feedback messages. - Update unit tests. Depends on Phase 2. ### Phase 4: Cleanup - Remove the `tools: vec![]` workaround from the previous fix (tools are now always passed through and `answer_inquiry` is always present). - Update [RFD 028] status to Superseded by this RFD. - Verify cache behavior with Anthropic API logs: inquiry requests should show cache reads matching normal requests. Depends on Phase 3. ## References - [RFD 028: Structured Inquiry System for Tool Questions][RFD 028] — the current implementation this RFD supersedes. - [Anthropic prompt caching docs][cache-docs] — cache prefix hierarchy and invalidation rules. - [Anthropic structured outputs docs][structured-docs] — documents the system prompt injection that causes cache invalidation. [RFD 028]: 028-structured-inquiry-system-for-tool-questions.md [RFD 034]: 034-inquiry-specific-assistant-configuration.md [cache-docs]: https://platform.claude.com/docs/en/build-with-claude/prompt-caching [cache-invalidation]: https://platform.claude.com/docs/en/build-with-claude/prompt-caching#what-invalidates-the-cache [structured-docs]: https://platform.claude.com/docs/en/build-with-claude/structured-outputs