RFD 068: Forced Tool Retry Without Reasoning

ImplementedDesign2026-03-27Jean Mertz

Summary

When the Anthropic API cannot combine forced tool use with extended thinking, JP falls back to a soft-force system prompt directive. If the model ignores that directive and finishes without calling the required tool, JP retries the request with reasoning disabled (or escalating nudges on always-thinking models) so the tool call is enforced.

Motivation

Anthropic's API does not support tool_choice: {type: "tool", name: "..."} (or any) when extended thinking is enabled. JP currently works around this by switching to tool_choice: auto and injecting an IMPORTANT: You MUST call... system prompt. This works most of the time, but the model sometimes ignores the directive and responds with text instead of a tool call.

This breaks workflows that depend on a specific tool being called on the first turn. The stager persona, for example, sets tool_choice = "git_list_patches" so the model immediately lists patches before doing anything else. When the model skips this, the downstream commit workflow operates on stale or missing data.

Design

NOTE

As implemented, the retry lives in the Anthropic provider (crates/jp_llm/src/provider/anthropic.rs), not the turn loop this section describes. The provider already holds the two facts the detection needs — the original tool_choice and whether thinking was downgraded — so keeping the retry there avoids surfacing an API restriction of one provider to the turn loop.

The implementation also went further than this RFD: models that cannot turn thinking off (claude-fable-5) get escalating soft-force nudges with thinking left on, because forced tool_choice stays rejected for them and the retry described below is unavailable.

The change lives in the turn loop (turn_loop.rs), not the provider layer.

Detection

Before starting the streaming phase, record two things:

1. The original tool_choice value (before the provider downgrades it).
2. Whether reasoning was active for this request.

If both are true, the turn is in "soft-force mode."

After the streaming phase completes with FinishReason::Completed (not interrupted, not tool calls), check:

• Was this a soft-force turn?
• Did the model produce zero tool calls?
• (If Function(name): did it not call the named tool?)

If all conditions hold, trigger a retry.

Retry

1. Keep the conversation as-is. The model's reasoning and text response from the first attempt are already in the conversation history, giving it full context.
2. Build a new ChatQuery with:
   • The original tool_choice (e.g. Function("git_list_patches"))
   • Reasoning disabled (override the config for this single request)
3. The provider layer now sees a forced tool call with no reasoning config, so it sends the real tool_choice parameter to the API.
4. Continue the normal turn loop from the streaming phase.

Limits

Retry at most once. If the second attempt also fails to call the tool (which should not happen with real tool_choice), proceed normally and let the user see the response.

Drawbacks

• Extra API call and latency when the soft-force fails. In practice this should be rare.
• The retry turn has no reasoning, so the tool arguments may be lower quality than a reasoned call. Mitigated by the fact that the model already reasoned in the first attempt and that context is in the conversation.

Alternatives

• Structured output workaround: Use --schema to force JSON output of the tool's expected result, then manually invoke the tool. This works for specific workflows (e.g. staging) but doesn't generalize.
• Always disable reasoning for forced tool calls: Loses the benefit of reasoning entirely. The current soft-force works most of the time; this retry is a safety net for when it doesn't.

Implementation Plan

Single phase:

1. Add soft-force tracking state to the turn loop (original tool_choice + whether reasoning was active).
2. After the streaming phase, check if a retry is needed.
3. If so, re-enter the streaming phase with reasoning disabled and the original tool_choice.
4. Add a test with SequentialMockProvider that verifies the retry fires when the first response has no tool calls.