RFD 031: Durable Conversation Storage with Workspace Projection

• Status: Draft
• Category: Design
• Authors: Jean Mertz git@jeanmertz.com
• Date: 2026-03-05

Summary

This RFD changes conversation storage so that user-local storage is the source of truth for all conversations. Workspace storage (.jp/conversations/) becomes a projection — a copy that exists for git visibility. Non-local conversations are written to both locations; local conversations are written only to user-local. The local property becomes a runtime-derived indicator (whether the workspace copy exists), not a stored flag.

This makes conversations durable across workspace directory deletion - the primary pain point for git worktree users; without requiring JP to be aware of git.

Motivation

JP stores conversations in two locations: workspace storage (.jp/conversations/ inside the project directory) and user-local storage (~/.local/share/jp/workspace/<name>-<id>/conversations/). Today, these are mutually exclusive — a conversation lives in one place or the other, controlled by the conversation.user flag (exposed as --local).

This creates a data loss problem for git worktree users. A typical worktree workflow looks like:

cd /path/to/project
git worktree add my-feature
cd my-feature
# ... work, use jp query, accumulate conversations ...
cd ..
git worktree remove my-feature   # conversations in .jp/conversations/ are gone

Any non-local conversation that was not committed to git is permanently lost. The user-local storage for that worktree also becomes orphaned, because the directory name is encoded in the user-local path (my-feature-<workspace-id>), and no future worktree will reconnect to it unless it happens to have the same directory name.

This is not a theoretical problem — it's a regular occurrence in active development with multiple worktrees. The current mitigations (commit conversations to git, or remember to --local every time, or manually move conversations before deleting the worktree) are all fragile and rely on the user remembering to act before destruction.

The underlying issue is that workspace storage is the only copy for non-local conversations. If the workspace directory disappears, the data is gone.

Design

Storage Model

Every conversation is always persisted to user-local storage. This is the durable copy. For non-local conversations, a second copy is additionally written to the workspace .jp/conversations/ directory. This workspace copy is a projection — it exists so that conversations are visible to git status and can be committed alongside code.

Conversation type	User-local	Workspace
Non-local (default)	✓ (durable copy)	✓ (projection)
Local (--local)	✓ (durable copy)	—

When both copies exist and their contents differ, the most recently modified copy takes precedence (see Manual Editing and Conflict Resolution).

User-local storage remains optional in jp_storage. When it is None (e.g., a headless server setup, or tests that don't need durability), the dual-write logic is skipped and JP falls back to single-write workspace storage — the same behavior as today. The --local flag and local derivation are unavailable without user-local storage; all conversations are workspace-only.

Shared User-Local Storage

Today, user-local storage is keyed by both the worktree directory name and workspace ID: ~/.local/share/jp/workspace/<name>-<id>/. This means each worktree gets its own user-local silo, and removing a worktree orphans its silo.

This RFD changes the user-local path to be keyed by workspace ID only:

~/.local/share/jp/workspace/<workspace-id>/conversations/

All worktrees for the same repository share a single user-local store. This aligns with RFD 020, which already places session mappings and locks at ~/.local/share/jp/workspace/<workspace-id>/.

The with_user_storage method on Storage drops the name parameter. The rename-on-mismatch logic in with_user_storage is replaced by a one-time migration that moves existing <name>-<id> directories to <id>.

The local Property

The conversation.user field (serialized as "local" in metadata.json) is removed from the Conversation struct. The local indicator shown in jp conversation ls becomes a runtime-derived property:

• local N: The conversation exists in the current workspace's .jp/conversations/. It is projected.
• local Y: The conversation exists only in user-local storage. It is not projected into this workspace.

This derivation is a filesystem check: does a directory matching this conversation's ID exist in <workspace>/.jp/conversations/?

Persistence Behavior

When JP persists a conversation:

1. Always write to user-local storage.
2. Check if a workspace copy exists (directory present in .jp/conversations/).
   • If yes, also write to workspace storage (update the projection).
   • If no, do not create a workspace projection.

For new conversations:

• jp query --new → create in both user-local and workspace.
• jp query --new --local → create in user-local only.

This means the workspace projection is created once (at conversation creation time for non-local conversations, or explicitly via jp conversation edit), and then maintained as long as the workspace directory survives. If the workspace directory is deleted and recreated, the projection is gone, and the conversation appears as local Y until explicitly re-projected.

Toggling Projection

jp conversation edit --local toggles the workspace projection:

• local Y → local N (project into workspace): Copy the conversation from user-local to workspace .jp/conversations/.
• local N → local Y (remove from workspace): Delete the workspace copy. The user-local copy remains.

Conversation Origin

When JP creates a conversation, it stores the worktree directory name as an origin field in the conversation metadata. This is purely informational — it has no behavioral impact — but provides context in jp conversation ls for identifying which worktree a conversation came from, especially when viewing conversations that were created in a worktree that no longer exists.

{
  "origin": "feature-a",
  "last_activated_at": "2025-07-20T10:00:00.000Z"
}

The origin field is set once at creation time and never updated.

Manual Editing and Conflict Resolution

JP uses plain, pretty-printed JSON files (events.json, metadata.json) specifically so that users can edit them by hand. Tweaking a conversation's context window — removing a noisy tool call, editing a response, trimming history — is a supported workflow. The dual-write model must not break this.

The rule is last-write-wins based on file modification time (mtime). When JP loads a conversation that exists in both user-local and workspace storage, it compares the mtime of the two events.json files (and separately, the two metadata.json files). Whichever was modified more recently is the version JP loads into memory.

On persist, JP writes to both locations, bringing them back into sync. After a successful persist, both copies are identical with fresh mtimes.

The full load-persist cycle:

1. Load: For each conversation file (events.json, metadata.json), if both copies exist, compare mtimes. Load the newer one.
2. Run: Execute the query, tool calls, etc. The in-memory state reflects the user's edits.
3. Persist: Write to user-local, then write to workspace (if projected). Both copies are now identical.

If only one copy exists (local-only conversation, or workspace copy was deleted by a worktree removal), there is nothing to compare — JP loads what's there.

This means a user can edit either copy between JP runs:

• Edit .jp/conversations/<id>/events.json in the workspace (the common case — the file is right there in the project directory, visible in the editor's file tree).
• Edit ~/.local/share/jp/workspace/<id>/conversations/<id>/events.json in user-local (less common, but works the same way).

In both cases, JP picks up the edit on the next run and propagates it to the other copy on persist.

Loading Conversations

When loading the conversation list (e.g., jp conversation ls), JP reads from both user-local and workspace storage:

1. Load all conversation IDs from user-local (authoritative).
2. Load all conversation IDs from workspace .jp/conversations/.
3. Merge: conversations in user-local are the primary set. Conversations that exist only in the workspace (not in user-local) are included but marked as workspace-only (see External Conversations).
4. Derive local for each conversation: check whether a workspace copy exists.

External Conversations

Conversations can appear in the workspace that do not exist in user-local. This happens when another contributor commits a conversation to git and you pull it.

These conversations are shown in jp conversation ls with a distinct indicator (e.g., origin: <their-worktree-name> from the stored metadata, or simply the absence of a user-local copy). They are read-only from JP's perspective until the user interacts with them.

On first write interaction (jp query --id=<id>, jp conversation edit, or any operation that modifies the conversation), the conversation is imported: copied from workspace to user-local. From that point on, it follows the normal dual-write persistence rules.

Non-write interactions (jp conversation show, jp conversation print, jp conversation ls) read directly from the workspace without importing.

Drawbacks

Double disk I/O for non-local conversations. Every persist operation writes to two locations. For JSON conversation files this is negligible, but it is strictly more work than the current single-write model.

Divergence risk. If JP crashes between writing to user-local and workspace (or vice versa), the two copies can diverge. The mtime-based resolution (see Manual Editing and Conflict Resolution) handles this gracefully — whichever copy was written last is used on next load, and the next successful persist re-syncs both copies. However, if the crash happens after writing user-local but before writing workspace, the workspace copy may be stale until the next JP run. Users who inspect workspace files directly between JP runs could see outdated content.

Conversation accumulation in user-local. Because user-local is the durable store, conversations accumulate there indefinitely (across worktree lifetimes). The existing ephemeral conversation cleanup (expires_at) helps, but users who create many conversations across many worktrees may eventually want a dedicated cleanup command or policy. This is not new — user-local conversations already accumulate today — but the volume increases when all conversations go through user-local.

Alternatives

Keep current model, add evacuation command

A jp conversation evacuate command that moves all workspace conversations to user-local before worktree removal. This requires the user to remember to run it (or configure a git hook).

Rejected because it doesn't solve the problem — it shifts it from "remember to use --local" to "remember to run evacuate." Forgetting either one results in data loss.

Store conversations at the bare repo level

For worktree setups, store conversations in a shared location above the worktrees (e.g., /path/to/project/.jp/conversations/). All worktrees read from and write to this shared directory.

Rejected because it requires JP to understand git worktree topology (resolving the common git dir), which violates the goal of keeping JP git-unaware. It also introduces write contention between concurrent worktrees.

Sync conversations across workspaces

Keep per-worktree storage but replicate conversations bidirectionally across all workspaces with the same ID.

Rejected because bidirectional sync is the wrong model. Conversations are contextual to a branch or task — syncing a feature-a conversation into the feature-b worktree is undesirable. The problem is durability, not distribution.

.jp as a file (like .git in worktrees)

Make .jp a pointer file that redirects to shared storage, similar to how git worktrees use a .git file pointing to the common git dir.

Rejected because .jp/conversations/ needs to contain real files for git status visibility. A pointer file would make conversations invisible to git, defeating one of the two core requirements.

Default all conversations to --local

Make Conversation::default() set user: true, so all conversations go to user-local by default. Users who want git-visible conversations opt in explicitly.

Rejected because it inverts the current expectation that conversations are visible in the workspace by default. The dual-write approach preserves the default behavior (workspace-visible) while adding durability.

Non-Goals

• Git awareness. JP does not detect whether it is running inside a git worktree, does not resolve the common git directory, and does not read or write git-specific metadata. The design works for any workflow where the workspace directory might be deleted — worktrees, temporary clones, or manual cleanup.

• Three-way merge conflict resolution. If both copies are edited between JP runs (e.g., a user edits the workspace copy while a background process edits the user-local copy), the older edit is silently overwritten by the newer one. JP does not attempt to merge concurrent edits. In practice, users edit one copy (almost always the workspace one), so this is unlikely to cause issues.

• Cross-machine sync. User-local storage is local to the machine. This RFD does not address synchronizing conversations across machines.

• Conversation garbage collection policy. This RFD does not define when or how old conversations in user-local should be cleaned up. The existing expires_at mechanism continues to work. A dedicated cleanup UX is future work.

Risks and Open Questions

Migration of existing user-local directories

Renaming <name>-<id> directories to <id> needs to handle the case where multiple worktrees have already created separate user-local directories (e.g., main-otvo8 and feature-a-otvo8). The migration must merge their contents without losing conversations. If two directories contain a conversation with the same ID (unlikely but possible if both worktrees were used concurrently), the most recently modified copy should win.

Workspace copy freshness

The workspace copy is updated on every persist, but only if the workspace copy already exists. If a user manually deletes a conversation from .jp/conversations/ (e.g., via git checkout or git clean), JP will not recreate it — the conversation silently becomes local-only. This is arguably correct behavior (the user deleted it), but might surprise users who expect workspace conversations to be self-healing.

Performance of local derivation

Deriving the local indicator requires checking the filesystem for each conversation when listing. For workspaces with many conversations, this could add latency to jp conversation ls. In practice, the number of conversations per workspace is small (tens, not thousands), so this is unlikely to be a problem. If it becomes one, the check can be batched into a single directory listing.

Implementation Plan

Phase 1: Shared User-Local Storage

Change the user-local storage path from <name>-<id> to <id>. Add migration logic to merge existing per-worktree directories. Update with_user_storage to drop the name parameter.

Depends on: nothing. Can be merged independently.

Phase 2: Dual-Write Persistence

Change persist_conversations_and_events to always write to user-local. For conversations where a workspace copy exists (or for new non-local conversations), also write to the workspace. Remove the conversation.user field from the Conversation struct. Derive the local property at runtime from workspace filesystem state.

Depends on: Phase 1. Can be merged independently.

Phase 3: Conversation Origin Metadata

Add the origin field to Conversation, populated from the worktree directory name at creation time. Surface it in jp conversation ls and jp conversation show.

Depends on: nothing (can be done in parallel with Phase 1/2). Can be merged independently.

Phase 4: External Conversation Import

Add lazy-import logic: when JP encounters a conversation in the workspace that does not exist in user-local, and the user performs a write operation on it, copy it to user-local first. Update jp conversation ls to display workspace-only conversations with appropriate indicators.

Depends on: Phase 2. Can be merged independently.

Phase 5: Update jp conversation edit --local

Change --local toggling to copy-to-workspace / delete-from-workspace instead of the current move-between-storage-locations behavior.

Depends on: Phase 2. Can be merged independently.

References

• RFD 020: Parallel Conversations — removes active_conversation_id, introduces per-session conversation tracking and conversation locks. This RFD's shared user-local storage aligns with RFD 020's session and lock storage paths.