# RFD 012: Typed LLM Streaming Events - **Status**: Implemented - **Category**: Design - **Authors**: Jean Mertz - **Date**: 2026-03-15 - **Required by**: [RFD 043] ## Summary This RFD replaces `ConversationEvent` inside `jp_llm::Event::Part` with a purpose-built `EventPart` enum that models what LLM providers actually stream: message chunks, reasoning chunks, structured data chunks, and tool call lifecycle events. This decouples the streaming transport layer from the persistence layer, eliminates invalid states, and prepares the infrastructure for incremental tool call argument streaming. ## Motivation `Event::Part` currently carries a `ConversationEvent`: ```rust pub enum Event { Part { index: usize, event: ConversationEvent }, Flush { index: usize, metadata: IndexMap }, Finished(FinishReason), } ``` `ConversationEvent` is a persistence type. It carries a timestamp, a metadata map, and an `EventKind` that can be any of seven variants: `TurnStart`, `ChatRequest`, `ChatResponse`, `ToolCallRequest`, `ToolCallResponse`, `InquiryRequest`, `InquiryResponse`. In the streaming context, only a subset is valid: | EventKind variant | Used in `Event::Part`? | | -------------------------- | ----------------------------------------------------- | | `ChatResponse::Message` | Yes — message content chunks | | `ChatResponse::Reasoning` | Yes — reasoning content chunks | | `ChatResponse::Structured` | Yes — but hacked as `Value::String(chunk)` | | `ToolCallRequest` | Yes — but only as a start signal with empty arguments | | `TurnStart` | No | | `ChatRequest` | No | | `ToolCallResponse` | No | | `InquiryRequest` | No | | `InquiryResponse` | No | This creates several problems: 1. **Invalid states are representable.** The `EventBuilder` must silently ignore `ChatRequest`, `ToolCallResponse`, `InquiryRequest`, `InquiryResponse`, and `TurnStart` in its `handle_part` method. These should be unrepresentable at the type level. 2. **Structured data is a hack.** Structured response chunks piggy-back on `ChatResponse::Structured { data: Value::String(chunk) }` — a persistence type abused as transport. This has caused bugs where consumers process structured data chunks as regular message content. 3. **Tool call starts are overloaded.** When a tool call begins, the provider emits a `ToolCallRequest` with empty arguments. This is a persistence type pretending to be a "streaming has started" signal. 4. **Wasted allocations.** Every streaming chunk allocates a `ConversationEvent` with a timestamp and metadata map that nobody reads during streaming. 5. **Growing `ConversationEvent`.** As the persistence type grows (new event kinds, richer metadata), every streaming chunk pays the cost even though it only uses a small fraction of the type. ## Design ### New Event structure ```rust pub enum Event { /// Streaming data for a given index, accumulated until Flush. Part { index: usize, part: EventPart, /// Metadata accumulated during streaming (e.g. thinking signatures). /// Usually empty. metadata: Map, }, /// Flush accumulated Parts for this index into a ConversationEvent. Flush { index: usize, metadata: IndexMap, }, /// The response stream is finished. Finished(FinishReason), } ``` ### EventPart ```rust /// A chunk of streaming data from an LLM provider. /// /// Each variant maps to a distinct content type that providers /// differentiate between. The EventBuilder accumulates these into /// ConversationEvents on Flush. pub enum EventPart { /// A chunk of assistant message content. Message(String), /// A chunk of reasoning/thinking content. Reasoning(String), /// A chunk of structured response JSON. Structured(String), /// Tool call streaming data. ToolCall(ToolCallPart), } ``` ### ToolCallPart ```rust /// Streaming events for a single tool call. pub enum ToolCallPart { /// Tool call identity. First non-empty value wins per field when /// multiple Start events arrive for the same index. Start { id: String, name: String, }, /// A raw JSON chunk of tool call arguments. ArgumentChunk(String), } ``` ### EventBuilder translation The `EventBuilder` becomes an explicit translator between the streaming domain (`EventPart`) and the persistence domain (`ConversationEvent`): | EventPart received | Buffer action | ConversationEvent on Flush | | ------------------------------ | -------------------------------- | ------------------------------------------------ | | `Message(s)` | Append to string buffer | `ChatResponse::Message { message }` | | `Reasoning(s)` | Append to string buffer | `ChatResponse::Reasoning { reasoning }` | | `Structured(s)` | Append to JSON string buffer | Parse JSON → `ChatResponse::Structured { data }` | | `ToolCall(Start { id, name })` | Initialize tool call buffer | — | | `ToolCall(ArgumentChunk(s))` | Feed `ToolCallRequestAggregator` | `ToolCallRequest { id, name, arguments }` | The `ToolCallRequestAggregator` remains unchanged — it buffers raw JSON argument chunks and parses them on finalize, as today. This RFD only changes the transport; it does not change how arguments are parsed. ### Metadata handling Streaming metadata (e.g. Anthropic's thinking signatures via `SignatureDelta`) currently piggy-backs on `ConversationEvent`'s metadata map. With `EventPart`, metadata moves to a field on `Event::Part`: ```rust Event::Part { index: 0, part: EventPart::Reasoning("thinking...".into()), metadata: Map::new(), // usually empty; populated for signature deltas } ``` The `EventBuilder` accumulates Part-level metadata the same way it does today — merging it into the final `ConversationEvent`'s metadata on Flush. ### Provider changes Each provider's stream mapper changes from constructing `ConversationEvent`s to constructing `EventPart`s. This is a mechanical transformation: **Before (Anthropic example):** ```rust // Message content Event::Part { index, event: ConversationEvent::now(ChatResponse::Message { message: text.clone(), }), } // Tool call start Event::Part { index, event: ConversationEvent::now(ToolCallRequest { id: id.clone(), name: name.clone(), arguments: Map::new(), }), } // Structured data chunk Event::Part { index, event: ConversationEvent::now(ChatResponse::Structured { data: Value::String(chunk), }), } ``` **After:** ```rust // Message content Event::Part { index, part: EventPart::Message(text.clone()), metadata: Map::new(), } // Tool call start Event::Part { index, part: EventPart::ToolCall(ToolCallPart::Start { id: id.clone(), name: name.clone(), }), metadata: Map::new(), } // Structured data chunk Event::Part { index, part: EventPart::Structured(chunk), metadata: Map::new(), } ``` ### TurnCoordinator changes The `TurnCoordinator` currently inspects `ConversationEvent` variants inside `Event::Part` to decide what to render. With `EventPart`, it matches directly on the streaming types: ```rust Event::Part { index, part, metadata } => { match &part { EventPart::Message(text) => { self.chat_renderer.render_message(text); } EventPart::Reasoning(text) => { self.chat_renderer.render_reasoning(text); } EventPart::Structured(chunk) => { self.structured_renderer.render_chunk(chunk); } EventPart::ToolCall(ToolCallPart::Start { id, name }) => { self.chat_renderer.flush(); // register tool call for preparing display } EventPart::ToolCall(ToolCallPart::ArgumentChunk(_)) => { // forwarded to EventBuilder only; no rendering yet } } self.event_builder.handle_part(index, part, metadata); } ``` ### Backwards compatibility - `ConversationEvent` is unchanged — it remains the persistence type used by `ConversationStream`. - The `ConversationStream` is not affected. - `Event::Flush` and `Event::Finished` are unchanged. - The `Event::Part` variant changes its inner type. All consumers of `Event::Part` must be updated (providers, EventBuilder, TurnCoordinator). This is a breaking change within `jp_llm`. ### What does NOT change - `ConversationEvent` and `EventKind` (persistence types) - `ConversationStream` (persistence layer) - `ToolCallRequestAggregator` (argument parsing — see [RFD 048] for its replacement) - The query stream pipeline's Part \> Flush \> Finished lifecycle ## Drawbacks - **Breaking change to `Event`.** Every provider and every consumer of `Event::Part` must be updated. This is a mechanical transformation but touches many files. - **Two type hierarchies for the same concepts.** `EventPart::Message` and `ChatResponse::Message` represent the same concept (assistant message content) at different layers. This is intentional — streaming and persistence have different concerns — but it means the EventBuilder translates between parallel hierarchies. ## Alternatives ### Keep ConversationEvent in Event::Part, add ValueChunk variant Add `Event::ValueChunk { index, chunk }` alongside the existing `Event::Part` for raw JSON chunks (tool call arguments, structured responses). This is the minimal change needed for incremental argument streaming. Rejected because: - `Event::Part` continues to carry invalid variants (`ChatRequest`, `TurnStart`, etc.) that the EventBuilder silently ignores. - Structured data still piggy-backs on `ChatResponse::Structured` with `Value::String(chunk)` — the hack that has caused bugs. - Tool call starts still overload `ToolCallRequest` with empty arguments. - The streaming and persistence layers remain coupled. ### EventPart with a generic Json variant instead of Structured + ToolCall Use `EventPart::Json(String)` for both structured response chunks and tool call argument chunks, routing by index buffer type in EventBuilder. Rejected because: - The EventBuilder already routes by type (message vs reasoning vs structured vs tool call). Having typed variants at the EventPart level is consistent with this routing. - `Json(String)` loses the tool call identity information that `ToolCall(Start { id, name })` provides. - Two different streaming use cases (structured data, tool call arguments) have different downstream handling — typed variants make this explicit. ## Non-Goals - **Incremental tool call argument streaming.** This RFD keeps the `ToolCallRequestAggregator` in the EventBuilder for argument parsing. The `ArgumentChunk` variant forwards raw JSON to the aggregator, same as today's behavior but with cleaner transport. - **Changing ConversationEvent.** The persistence type is unchanged. This RFD only affects the streaming transport layer. ## Risks and Open Questions - **Provider-specific streaming patterns.** Some providers may stream tool call `id` and `name` across multiple deltas rather than in a single start event. The `ToolCallPart::Start` design handles this via first-non-empty-wins semantics (same as today's `merge_tool_call`), but this needs validation with all supported providers (Anthropic, OpenRouter, Llamacpp, Ollama, Google). - **Metadata edge cases.** Moving metadata from `ConversationEvent` to `Event::Part` should be straightforward, but any provider-specific metadata patterns need testing. ## Implementation Plan ### Phase 1: Define EventPart types Add `EventPart`, `ToolCallPart` to `jp_llm`. These are pure types with no behavioral changes. ### Phase 2: Update EventBuilder Change `EventBuilder::handle_part` to accept `EventPart` instead of `ConversationEvent`. Update the `IndexBuffer` creation and accumulation logic. The output (flushed `ConversationEvent`s) is identical. ### Phase 3: Update providers Update Anthropic, OpenRouter, Llamacpp, Ollama, and Google providers to emit `EventPart` variants instead of constructing `ConversationEvent`s. Remove `ConversationEvent` construction from the provider layer entirely. Move the `ToolCallRequestAggregator` from the provider layer into the EventBuilder — providers now emit `ToolCall(ArgumentChunk(chunk))` instead of buffering chunks themselves. ### Phase 4: Update TurnCoordinator Update `TurnCoordinator::handle_streaming_event` to match on `EventPart` instead of `ConversationEvent`. Update the chat renderer, structured renderer, and tool renderer integration points. ### Phase 5: Clean up Remove `Event`'s `as_conversation_event()`, `into_conversation_event()`, and related helper methods that assume `Part` carries a `ConversationEvent`. Add equivalent helpers for `EventPart` if needed. ## References - `crates/jp_llm/src/event.rs` — current `Event` type - `crates/jp_conversation/src/event.rs` — `ConversationEvent`, `EventKind` - `crates/jp_conversation/src/event_builder.rs` — `EventBuilder` - `crates/jp_cli/src/cmd/query/turn/coordinator.rs` — `TurnCoordinator` - `crates/jp_llm/src/stream/aggregator/tool_call_request.rs` — `ToolCallRequestAggregator` [RFD 043]: 043-incremental-tool-call-argument-streaming.md [RFD 048]: 048-four-channel-output-model.md