# RFD D21: Interactive Conversation Stream Editing - **Status**: Draft - **Category**: Design - **Authors**: Jean Mertz - **Date**: 2026-04-13 ## Summary This RFD introduces `jp conversation edit --interactive`, an `$EDITOR`-based workflow for destructively editing the raw event stream of a conversation. The editor opens a temporary directory containing a plan file (the manifest of events) and individual event files (markdown or TOML). Users delete or reorder lines in the plan to restructure the conversation, and edit individual event files to modify content. On editor exit, JP validates the result and rebuilds the `ConversationStream`. ## Motivation JP users already edit conversations by opening `events.json` directly — tweaking context, removing noisy tool calls, editing responses, trimming history. This works but is painful: the JSON is base64-encoded in places, events are interleaved with config deltas, and the structure is fragile (orphaned tool responses, broken request/response alternation). [RFD 064] introduced non-destructive compaction as an overlay — the right approach for routine context reduction. But compaction is a *projection*: it changes what the LLM sees without changing what's stored. Sometimes you need actual surgery: fix a wrong tool result, reword the user's request, delete an entire tangent, inject a clarifying message. Compaction can't do this. `jp conversation fork --last N` is the blunt instrument for this today. It discards everything before the last N turns. There's no way to selectively keep turn 1, drop turns 2-4, and keep turn 5 onward. Users need a way to make precise, destructive edits to the conversation stream in a format they can read and manipulate with their existing editor. ## Design ### User-Facing Behavior #### The `--interactive` Flag ```sh jp conversation edit --interactive [ID] ``` Opens an interactive editing session for the active conversation (or the specified one). This is a destructive operation — it modifies the stored event stream. The short flag is `-i`: ```sh jp conversation edit -i ``` #### What the Editor Sees JP creates a temporary directory with this structure: ``` /tmp/jp-edit-/ ├── CONVERSATION ├── 000-request.md ├── 001-message.md ├── 002-tool-call-fs_create_file.md ├── 003-tool-result-fs_create_file.md ├── 004-config-delta.toml ├── 005-request.md ├── 006-reasoning.md ├── 007-tool-call-fs_read_file.md ├── 008-tool-result-fs_read_file.md ├── 009-message.md ├── 010-compaction.toml └── ... ``` The editor is invoked with this directory as its argument. Most editors (VS Code, Vim, Neovim, Emacs) open a directory as a file browser or project root, giving the user a tree view of all files. #### The Plan File `CONVERSATION` is the manifest. It lists every event file, grouped by turn with comment headers. The all-caps name sorts to the top in file explorers (uppercase before lowercase in most locales) and gives editors a recognizable filename for syntax highlighting — the same pattern as `Makefile` or `Dockerfile`. ```txt # Conversation Edit Plan # - Delete lines to remove events. # - Reorder lines to reorder events. # - Edit event contents in the corresponding files. # Turn 0 000-request.md 001-message.md 002-tool-call-fs_create_file.md 003-tool-result-fs_create_file.md # Turn 1 004-config-delta.toml 005-request.md 006-reasoning.md 007-tool-call-fs_read_file.md 008-tool-result-fs_read_file.md 009-message.md # Turn 2 010-compaction.toml 011-request.md 012-message.md ``` The `CONVERSATION` file controls **structure**: which events survive and in what order. Users delete lines to drop events and reorder lines to reorder events. Comment lines (`#`) are ignored during parsing. The `CONVERSATION` file is authoritative for structure. If a line is removed, the event is dropped — regardless of whether the file still exists on disk. #### Event Files Each event is written to an individual file. The file format depends on the event type: **Conversation events** use markdown with YAML frontmatter: ```markdown --- type: request --- Set up the project with error handling and logging. ``` ```markdown --- type: message --- I'll create the project structure for you. ``` ```markdown --- type: reasoning --- The user wants a Rust project with error handling. I should use anyhow for the error type and set up a basic main.rs... ``` ````markdown --- type: tool-call tool: fs_create_file id: call_abc123 --- ```json { "path": "src/main.rs", "content": "fn main() {\n println!(\"hello\");\n}" } ```` ```` ```markdown --- type: tool-result id: call_abc123 is_error: false --- File created successfully. ```` **Metadata events** use TOML: ```toml # config-delta.toml [assistant] model = "anthropic/claude-sonnet" ``` ```toml # compaction.toml from_turn = 0 to_turn = 5 [reasoning] policy = "strip" ``` The frontmatter carries the metadata needed to reconstruct the `ConversationEvent`: event type, tool name, call ID, error status. The body carries the user-editable content. #### File Naming Files are named with a zero-padded numeric prefix followed by a descriptive suffix: | Event type | Suffix | Extension | | -------------- | ------------------------- | --------- | | `ChatRequest` | `request` | `.md` | | `ChatResponse` | `message`, `reasoning`, | `.md` | | | `structured` | | | `ToolCallReq` | `tool-call-{tool_name}` | `.md` | | `ToolCallResp` | `tool-result-{tool_name}` | `.md` | | `InquiryReq` | `inquiry-request` | `.md` | | `InquiryResp` | `inquiry-response` | `.md` | | `ConfigDelta` | `config-delta` | `.toml` | | `Compaction` | `compaction` | `.toml` | The numeric prefix establishes original ordering for orientation — it is an address, not a sort key. The `CONVERSATION` file determines final ordering. `TurnStart` events are not represented as files. They are internal markers inferred from the event sequence during reconstruction (a `ChatRequest` starts a new turn). The turn comments in the plan file (`# Turn N`) provide visual grouping but have no semantic effect. #### The Edit Cycle 1. JP creates the temporary directory and writes all files. 2. JP hashes every file. 3. JP opens the editor with the directory path. 4. The user edits the plan and/or event files. 5. The editor exits. 6. JP re-reads the plan file and all referenced event files. 7. JP validates the result (see [Validation](#validation)). 8. If valid: JP rebuilds the `ConversationStream` and persists it. 9. If invalid: JP writes error annotations to the top of `CONVERSATION` and re-opens the editor. The user can fix the issue or clear the file to abort. **Abort:** If `CONVERSATION` is empty (all lines deleted or cleared) when the editor exits, the edit is aborted with no changes. If the editor exits with a non-zero status code, the edit is also aborted. **Unchanged files:** If a file's hash matches its original, JP uses the original event data. Only files with changed hashes are re-parsed. This avoids round-trip fidelity issues for events the user didn't touch. #### New Events Users can create new event files in the temporary directory and add their filenames to the plan. JP parses new files the same way as modified files — the frontmatter must contain valid metadata for the event type. For tool calls, the user must provide a call ID in the frontmatter. If omitted, JP generates one. Tool results must reference an existing call ID. This enables injecting events: adding a clarifying user message, inserting a corrected tool result, or adding a config delta. ### Integration with `fork` `jp conversation fork` gains an `--edit` flag: ```sh jp conversation fork --edit jp conversation fork --last 5 --edit ``` This forks the conversation first (with any applicable filtering), then opens the interactive editor on the forked conversation. The original conversation is untouched. This is the safe workflow for destructive edits — you always work on a copy. The `--edit` flag is incompatible with `--compact` (compaction is non-destructive and additive; editing is destructive). ### Validation When the editor exits, JP validates the rebuilt stream. Validation enforces structural invariants that providers require: 1. **Tool result follows its tool call.** A `tool-result` must appear after the `tool-call` with the matching call ID. 2. **Request/response alternation.** A `ChatRequest` (user role) must not be immediately followed by another `ChatRequest` without an intervening assistant response. 3. **Orphaned references.** A `tool-result` whose call ID doesn't match any `tool-call` in the plan is rejected. Similarly for inquiry responses. 4. **Missing references.** A `tool-call` without a matching `tool-result` in the plan gets a synthetic error response injected (consistent with `sanitize_orphaned_tool_calls`). 5. **Non-empty stream.** The plan must contain at least one `ChatRequest`. Validation reuses and extends the existing `ConversationStream::sanitize()` logic. The difference: `sanitize()` silently fixes issues (it's designed for automatic recovery), while the editor validation reports errors and re-opens the editor so the user can fix them intentionally. When validation fails, JP prepends error messages to `CONVERSATION`: ```txt # ERROR: tool-result call_abc123 appears before its tool-call (line 5) # ERROR: orphaned tool-result call_xyz789 has no matching tool-call # # Fix the errors above and save, or clear this file to abort. # Turn 0 000-request.md ... ``` ### Round-Trip Fidelity The editing format must preserve all data needed to reconstruct events exactly. Key concerns: - **Tool call arguments.** Complex nested JSON must survive the round-trip through the markdown frontmatter format. Arguments are stored as a JSON code block in the file body, not inlined into YAML. - **Timestamps.** Original event timestamps are preserved in the frontmatter but hidden from casual editing (they appear as a `timestamp` field). If the user modifies a timestamp, the new value is used. If omitted from a new event, the current time is used. - **Metadata.** The `metadata` map on `ConversationEvent` (cache breakpoints, rendered arguments, etc.) is serialized into the YAML frontmatter. Fields the user doesn't touch are preserved via the hash-based change detection. - **Base64 encoding.** The storage layer base64-encodes certain fields (tool arguments, response content). The editor files contain decoded (plain text) content. Re-encoding happens during reconstruction. - **Base config.** The conversation's `base_config.json` is not part of the edit session. It is immutable and preserved as-is. ## Drawbacks - **Destructive by nature.** Unlike compaction, this modifies the actual stored events. There is no undo. Mitigation: `fork --edit` is the recommended workflow, and we document it prominently. The original conversation is untouched. - **Format complexity.** The markdown-with-frontmatter format for events is a new serialization format that must be maintained alongside the JSON storage format. It adds code surface for parsing and round-tripping. - **Editor compatibility.** The design assumes the editor can open a directory. Most modern editors support this, but some minimal editors (e.g. `ed`, `nano`) do not. For these editors, the experience degrades — the user would need to open `CONVERSATION` directly and navigate to event files manually. - **Large conversations.** A conversation with 500 events produces 500+ files in the temporary directory. This is fine for filesystems and editors, but the plan file becomes long. Mitigation: the turn-grouped comments help navigation, and users typically edit recent portions of the conversation. ## Alternatives ### Single-file editing (git rebase model) Present all events in a single plan file with inline content. The user edits everything in one file, using `pick`/`drop`/`edit` verbs like `git rebase -i`. Rejected because conversation events are contextual — when editing event 7, you want to see events 6 and 8 simultaneously. A single-file model either inlines all content (making the file enormous and hard to navigate) or forces a sequential `edit` workflow where you're walked through events one at a time. The directory model lets you open multiple files side-by-side in your editor. ### Git rebase `pick`/`drop` verbs Add `pick` and `drop` verbs to the plan file lines, matching git rebase's interface. Rejected as unnecessary complexity. Since the plan file only supports two structural operations (delete and reorder), the simpler model works: lines present = kept, lines absent = dropped, line order = event order. No verbs to learn. ### Edit `events.json` directly The status quo. Users open the raw JSON file and edit it. This already works but is error-prone: base64-encoded fields, interleaved config deltas, fragile structural invariants. The interactive editor provides a human-readable format with validation on save. ### Non-destructive editing via overlay Extend the compaction overlay model to support content replacement — store "event X should have this content instead" as a projection event. Rejected because it conflates two different concerns. Compaction reduces what the LLM sees while preserving history. Content editing changes history itself. Overlaying content edits would make the projection layer significantly more complex and would still not support structural changes (reordering, deletion of arbitrary events). ## Non-Goals - **Automatic conflict resolution.** If two users edit the same conversation simultaneously, the last writer wins. Interactive editing acquires the conversation lock, so concurrent edits are prevented during the session. - **Undo/redo.** There is no undo for destructive edits. Use `fork --edit` to work on a copy. The original conversation is the "undo." - **TUI-based editing.** This RFD uses `$EDITOR` exclusively. A built-in terminal UI for event editing (with live validation, drag-and-drop reordering, etc.) is a separate feature. - **Partial stream editing.** The editor session covers the entire conversation stream. Editing a subset (e.g. "only turns 5-10") can be achieved by forking with `--last` first, then editing the fork. - **Custom editor invocation.** The editor is resolved using `EditorConfig` (the existing `JP_EDITOR` / `VISUAL` / `EDITOR` chain). Adding a conversation-edit-specific editor config (e.g. for opening a directory vs. a file) is deferred. ## Risks and Open Questions - **Frontmatter parsing robustness.** YAML frontmatter in markdown is a de-facto standard but has edge cases (content that looks like YAML, `---` in code blocks). We need a robust parser that handles these correctly. The existing `jp_md` crate does not parse frontmatter today. - **Inquiry event editing.** `InquiryRequest` and `InquiryResponse` have complex structures (answer types, select options, default values). The frontmatter representation needs to be both human-readable and round-trippable. This may require a more structured format for inquiry events specifically. - **Structured response editing.** `ChatResponse::Structured` contains a `serde_json::Value`. The editing format needs to present this as editable JSON while preserving the value's structure on round-trip. - **Editor directory support.** If the configured editor cannot open a directory, the experience breaks. We should detect this and fall back to opening `CONVERSATION` directly, with a note about how to edit individual event files. - **Conversation lock duration.** The conversation lock is held for the entire editor session. If the user leaves the editor open for hours, other operations on that conversation are blocked. This matches the behavior of `git rebase` holding a lock, but may need a warning or timeout. ## Implementation Plan ### Phase 1: Event Serialization Format 1. Define the markdown-with-frontmatter format for each `EventKind` variant. 2. Define the TOML format for `ConfigDelta` and `Compaction` events. 3. Implement `serialize_to_edit_file()` and `deserialize_from_edit_file()` for each event type. 4. Add round-trip unit tests: serialize an event, deserialize it, assert equality. Can be merged independently. No behavioral changes. ### Phase 2: Plan File and Directory Builder 1. Implement the temporary directory builder: given a `ConversationStream`, produce the file tree (`CONVERSATION` + event files). 2. Implement the file naming scheme (numeric prefix + descriptive suffix). 3. Implement turn-grouped comments in the plan file. 4. Add unit tests for directory generation from sample streams. Depends on Phase 1. ### Phase 3: Plan Parser and Stream Reconstruction 1. Implement the plan file parser: read `CONVERSATION`, produce an ordered list of filenames. 2. Implement the reconstruction pipeline: parse the plan, read event files (using originals for unchanged files), rebuild the `ConversationStream`. 3. Implement hash-based change detection for event files. 4. Add unit tests for plan parsing, reconstruction, and change detection. Depends on Phase 2. ### Phase 4: Validation 1. Implement structural validation on the rebuilt stream (tool call pairing, request/response alternation, orphaned references). 2. Implement error annotation in `CONVERSATION` for re-opening. 3. Add unit tests for each validation rule and the error-annotation format. Depends on Phase 3. Can partially reuse `ConversationStream::sanitize()`. ### Phase 5: CLI Integration 1. Add `--interactive` / `-i` flag to `jp conversation edit`. 2. Implement the edit cycle: create directory, hash files, open editor, read back, validate, rebuild, persist. 3. Implement abort detection (empty plan, non-zero exit). 4. Add `--edit` flag to `jp conversation fork`. 5. Integration tests with `MockEditorBackend`. Depends on Phase 4. ## References - [RFD 047] — Editor and Path Access for Conversations - [RFD 064] — Non-Destructive Conversation Compaction (defers interactive editing as a non-goal) - [Issue #57] — Make conversation management more powerful - `crates/jp_conversation/src/stream.rs` — `ConversationStream` and `InternalEvent` definitions - `crates/jp_conversation/src/event.rs` — `EventKind` variants - `crates/jp_editor/src/lib.rs` — `EditorBackend` trait - `crates/jp_config/src/editor.rs` — `EditorConfig` and editor resolution [Issue #57]: https://github.com/dcdpr/jp/issues/57 [RFD 047]: ../047-editor-and-path-access-for-conversations.md [RFD 064]: ../064-non-destructive-conversation-compaction.md