# RFD 007: Llamacpp Reasoning Support - **Status**: Implemented - **Category**: Design - **Authors**: Jean Mertz - **Date**: 2025-07-18 - **Summary**: Replaces the `openai` crate in the Llamacpp provider to extract reasoning content across all three `llama.cpp` formats. ## Summary The Llamacpp provider silently drops reasoning content when the `llama.cpp` server uses its default `--reasoning-format deepseek` setting. This RFD replaces the `openai` crate with a direct HTTP implementation that extracts reasoning content across all three `llama.cpp` reasoning formats. ## Motivation The llama.cpp server exposes an OpenAI-compatible `/v1/chat/completions` endpoint. When models produce thinking tokens, the server can return them in three ways depending on the `--reasoning-format` flag: | Format | `delta.content` | `delta.reasoning_content` | | -------------------- | -------------------------------- | ------------------------- | | `deepseek` (default) | Regular content only | Reasoning content | | `none` | Everything (with `` tags) | Not set | | `deepseek-legacy` | Everything (with `` tags) | Reasoning content | Our Llamacpp provider uses the `openai` Rust crate, which only reads `delta.content` and `delta.tool_calls`. The `reasoning_content` field is a non-standard DeepSeek extension that the crate ignores. The existing `ReasoningExtractor` parses `` tags from `delta.content`, which works for `none` and `deepseek-legacy` but does nothing for the default `deepseek` format — there are no tags to extract because the server already stripped them. This is the same class of bug we fixed in the Ollama provider, where `message.content` and `message.thinking` were never processed. The Ollama fix was straightforward because `ollama-rs` exposes the `thinking` field natively. For Llamacpp, the `openai` crate does not expose `reasoning_content`. ## Design From a user's perspective, reasoning content from llama.cpp models appears in conversations the same way it does for Ollama or Anthropic — no configuration changes, no new flags. The fix is entirely internal to the Llamacpp provider. ### Drop the `openai` Crate Entirely The `openai` crate's `ChatCompletionMessageDelta` struct does not include `reasoning_content`, and the field is unlikely to be added upstream since it's not part of the OpenAI spec. Rather than forking or patching the crate, we replace all of its usage — both streaming and request building — with our own implementation using `reqwest`, `reqwest_eventsource`, and `serde_json`. The `openai` crate is used for four things, all replaced: 1. **`ChatCompletionDelta::builder()` + `create_stream()`** — replaced with manual JSON body construction via `serde_json::json!` and SSE streaming via `reqwest_eventsource::EventSource`. 2. **`ChatCompletionMessage` / `ChatCompletionMessageRole`** — conversation history is serialized directly to `serde_json::Value` messages. 3. **`ChatCompletionChoiceDelta` / `ChatCompletionMessageDelta`** — replaced with local serde types (`StreamChunk`, `StreamChoice`, `StreamDelta`) that include the `reasoning_content` field. 4. **`Credentials`** — removed; base URL comes from `LlamacppConfig` directly. The `Llamacpp` struct simplifies to just `reqwest_client` and `base_url`. Model listing already uses `reqwest` directly and is unaffected. ### SSE Streaming The llama.cpp server sends SSE events in this shape: ```json data: {"choices":[{"delta":{"reasoning_content":"Let me think..."},"index":0}]} data: {"choices":[{"delta":{"content":"The answer is..."},"index":0}]} ``` We parse each SSE event's JSON ourselves, reading both `reasoning_content` and `content` from the delta object. The llama.cpp server source (`common_chat_msg_diff_to_json_oaicompat` in `common/chat.cpp`) is the authoritative reference for the streaming delta format. The SSE loop handles: - `Event::Open` — ignored - `Event::Message` with `data: [DONE]` — finalize and flush - `Event::Message` with JSON — parse as `StreamChunk`, route fields to events - Errors — delegated to the existing `From for StreamError` classifier in `error.rs`, which handles retry-after extraction, `x-should-retry`, and status code classification ### Reasoning Routing The implementation auto-detects which llama.cpp format is active based on which fields are present in each delta, with no user configuration required: | Format | `reasoning_content` | `content` | Routing | | ----------------- | ------------------- | -------------------- | --------------------------------------- | | `deepseek` | Present | Regular text | `reasoning_content` → reasoning events | | | | | directly | | `none` | Absent | Everything with tags | `content` → `ReasoningExtractor` parses | | | | | `` | | `deepseek-legacy` | Present | Everything with tags | `reasoning_content` → reasoning; | | | | | `content` → text | For `deepseek-legacy`, when both fields are present, `reasoning_content` goes directly to reasoning events and `content` is treated as regular text (bypassing the extractor, since the server already separates them). ### Event Indexing Following the Ollama provider's convention: - **Index 0**: reasoning content - **Index 1**: message content (or structured output) - **Index 2+**: tool calls The `ToolCallRequestAggregator` is reused unchanged to accumulate partial tool call JSON chunks across multiple SSE events. ### Flush Strategy Same as Ollama — reasoning (index 0) flushes when the first content or tool call chunk arrives. This guarantees reasoning events precede content and tool call events in the conversation history. On `finish_reason` or `[DONE]`, all remaining indices flush. ### Request Building Conversation history serialization produces `serde_json::Value` messages directly. Reasoning responses from previous turns are wrapped in `` tags so the model can pick up its own chain-of-thought. Tool definitions use `parameters_with_strict_mode` from the OpenAI provider module (shared utility, not the `openai` crate). ## Alternatives - **Patch the `openai` crate** to add `reasoning_content` to `ChatCompletionMessageDelta`. Rejected because the field is a non-standard DeepSeek extension unlikely to be accepted upstream, and maintaining a fork for a single field is more burden than owning the SSE loop directly. - **Keep the `openai` crate for request building, replace only streaming.** Rejected because the crate's builder, message types, and credential handling are tightly coupled — replacing streaming alone leaves dead code and a confusing split of responsibilities. ## Non-Goals - Adding user-facing configuration for `--reasoning-format`. The routing auto-detects the format from the delta fields. - Changing other providers. The Ollama and OpenRouter providers already handle reasoning correctly. ## Risks - **Owning the SSE loop**: Edge cases like malformed events and the `[DONE]` sentinel are our responsibility. Mitigated by keeping the loop thin and delegating error classification to the existing `reqwest_eventsource` classifier. - **`ReasoningExtractor` buffering latency**: The extractor holds a small tail buffer (< 8 bytes) to detect split `` tags. Only affects the `none` format path; latency is negligible. ## Implementation 1. Define local serde types for the OpenAI Chat Completions streaming delta format: `StreamChunk`, `StreamChoice`, `StreamDelta`, `ToolCallDelta`, `FunctionDelta`. The key addition is `reasoning_content` on `StreamDelta`. 2. Replace request building with `build_request()` producing `serde_json::Value`. Conversation history, tool definitions, and structured output schemas serialize to JSON directly. 3. Replace `openai` crate streaming with `reqwest_eventsource::EventSource`. The `handle_sse_event` function processes each SSE event and routes reasoning/content/tool-call fields to provider-agnostic `Event` values. 4. Wire up reasoning routing: `reasoning_content` is used directly when present, `ReasoningExtractor` as fallback when only `content` is available. 5. Remove all `openai` crate imports from `llamacpp.rs`. 6. Add unit tests for delta parsing across the three formats, request building (tool call merging, reasoning wrapping), and tool choice conversion. 7. Re-record VCR cassettes against a live llama.cpp server. ## References - llama.cpp server reasoning format docs: `--reasoning-format` flag in `tools/server/README.md` - llama.cpp source: `common_chat_msg_diff_to_json_oaicompat` in `common/chat.cpp` - Ollama provider (this codebase): reads `message.thinking` and `message.content` directly; uses the same index convention (0/1/2+) and flush strategy that this implementation mirrors. - [yoagent `openai_compat.rs`]: Reference implementation of OpenAI-compatible SSE streaming with `reqwest_eventsource`. Provides the serde type patterns and SSE loop structure adapted for this implementation. [yoagent `openai_compat.rs`]: https://github.com/yologdev/yoagent/blob/main/src/provider/openai_compat.rs