RFD 106: Collision-Free Conversation ID Allocation

• Status: Discussion
• Category: Design
• Authors: Jean Mertz git@jeanmertz.com
• Date: 2026-08-05
• Extends: RFD 031, RFD 054, RFD 073

Summary

Conversation IDs encode a decisecond timestamp, so two conversations created within the same 100 ms window get the same ID and one silently destroys the other on disk. This RFD keeps the ID format and replaces Utc::now() with an allocator that claims the first unoccupied slot at or after the current decisecond, and adds an exact created_at to conversation metadata.

Motivation

ConversationId is a DateTime<Utc> truncated to deciseconds, rendered as jp-c17457886043. ConversationId::default() returns Utc::now() truncated, and every creation path goes through it.

jp conversation fork A B loops over its sources in one process (fork_each in fork.rs), calling create_and_lock_conversation_with_projection per iteration. Both land in the same decisecond on typical local storage, where the per-iteration persist is three small JSON writes. Then:

1. The second creation runs entry(id).insert_entry(OnceLock::new()) (create_conversation_with_projection in jp_workspace/src/lib.rs), replacing the first fork's in-memory state; the result is discarded with let _err =.
2. The first fork's ConversationLock dropped at the end of its iteration, so try_lock succeeds and no error surfaces.
3. On persist, reconcile_conversation_dir (jp_storage/src/lib.rs) renames the first fork's directory into the second's name and remove_dir_alls the rest.

The first fork is gone. No error, no warning, exit status 0.

This needs no concurrency and no unusual timing, though it is not guaranteed: fork --compact runs an LLM call per iteration (fork.rs), which separates them by seconds. The cross-process cases are probabilistic, and RFD 050 makes them routine — conversation new and fork exist to be driven from scripts, and fork prints one ID per source, so a script can receive the same ID twice.

The existing test (test_conversation_fork in fork_tests.rs) forks a single source against a stubbed epoch clock, so it always passes; the multi-source path has no coverage.

Design

What the user sees

The ID format is unchanged: jp-c17457886043 stays an eleven-digit decisecond timestamp, directory names are unchanged, and existing conversations keep their IDs.

The contract:

Conversations auto-allocated and persisted through one allocation domain — one machine's pair of storage roots — receive distinct IDs.

Three behavioral changes:

• Every created conversation gets a distinct ID. A burst of N creations yields N distinct IDs in ascending order, skipping occupied slots.
• IDs allocated within one process are strictly increasing. Across process restarts, clock corrections, and conversations arriving through git they stay unique but do not define creation order. created_at records the exact instant of creation, undrifted, but it orders conversations only as far as the clocks that stamped them agree.
• A burst that runs the allocator far ahead of wall-clock time fails with a clear error rather than succeeding with a badly-dated ID.

The ID-ordered interfaces stay approximate by design. --sort created, the --created-since / --created-before thresholds, and the newest target all read id.timestamp() (cmd/target.rs), so a backwards clock correction can leave newest pointing at the earlier conversation. (latest is unaffected: it sorts on last_activated_at.)

metadata.json gains an exact created_at, and jp conversation show a Created row backed by a top-level created_at key in its -F json payload (DetailsFmt::json in format/conversation.rs). jp conversation ls -F json already emits a created_at derived from the ID (payload in ls.rs); it switches to the stored value, falling back to id.timestamp() when the field is absent, so the two commands cannot report different creation times for the same conversation. Ordering and filtering stay on the ID everywhere, --sort created included.

Why the ID format is worth keeping

Seven behaviors read the ID's timestamp: --sort created (ls.rs, grep.rs), the last_event_at fallback for an empty conversation (ls.rs), the created_at key in conversation ls -F json (payload in ls.rs), the --created-since / --created-before thresholds (CreationRange::matches in time.rs), expires_at = id.timestamp() + ttl (create_new_conversation in query.rs), newest resolution (target.rs), and the seed for ConversationStream::created_at (jp_storage/load.rs, jp_workspace/src/lib.rs). Any format change — random suffix, ULID, added precision — invalidates some or all of them plus every directory name on disk. Allocating within the existing encoding fixes the collision without touching the contract.

The allocator

The requirement is uniqueness, not global ordering: across machines IDs never could increase monotonically, since a teammate's conversation can arrive through git carrying any timestamp. So the allocator takes the first slot nobody else has rather than chasing a maximum. It replaces ConversationId::default() in the creation path:

candidate = max(now_decisecond, process_high_water + 1)
while candidate is occupied:
    candidate += 1
claim candidate

A slot is occupied when a conversation directory exists for it in any partition of any storage root, or when its conversation lock is currently held. Neither condition alone covers a slot for its whole life: between the claim and the persist only the lock does, and once the lock is released only the directory does. A ConversationLock persists before releasing its guard, so the two overlap instead of leaving a gap, and the implementation must preserve that ordering. Observing both without a gap is a separate problem, which Exclusion covers. process_high_water is bumped per allocation so a twelve-source fork does not hand out a slot it claimed but has not yet persisted.

Occupancy comes from storage, not from a cursor file. Allocated IDs are already durably recorded as directory names, and scan_conversation_ids (jp_storage/load.rs) already reads them. The scan runs once per allocation, under the allocator lock, into a HashSet. It cannot be hoisted to once per process: a set built before the first allocation goes stale as soon as another process persists, which is the race Exclusion covers. Nor is a per-candidate check cheaper — directories are named {id}-{title}, so testing one candidate is itself a read_dir (conversation_dirs_for_id in jp_storage/src/lib.rs).

The scan covers both roots and both partitions. load_conversation_index filters to the active directory or .archive/, not both (jp_storage/load.rs), so an occupancy set built from the workspace index alone would miss archived IDs and unarchive_conversation could land on an occupied directory.

Occupancy is a read, so it goes on LoadBackend, which already owns reading conversations (RFD 073 made Storage an internal detail behind it): FsStorageBackend answers with a union scan, InMemoryStorageBackend from its own map. Allocation therefore reads through loader and claims through locker (jp_workspace/src/lib.rs), which is sound only when both describe the same conversation set — a requirement worth stating rather than assuming. --no-persist breaks it deliberately: the loader stays filesystem-backed while the locker becomes NullLockBackend, so occupancy reflects durable state the run will never add to.

Because the floor is now rather than the stored maximum, an imported future-dated ID costs one skipped slot instead of blocking creation until local time catches up. After a backwards clock jump the next process reuses the low free slots: IDs stop increasing, uniqueness holds. Within the process that saw the jump the high-water mark still applies, so allocation stays above the last ID it handed out and the drift budget decides what happens next.

Exclusion

A scan is a snapshot, blind to what another process writes afterwards. Given two concurrent multi-source forks, P can claim T while Q claims, persists, and releases T+1; P's second allocation then finds T+1 unlocked and absent from its stale snapshot, and overwrites Q.

The per-conversation lock cannot close that on its own, and not only because of the snapshot: ConversationFileLock::drop releases the flock before unlinking the pathname (jp_storage/src/lock.rs), so two processes can hold exclusive locks on different inodes for the same path.

So each allocation runs under a single stable lock, locks/allocator.lock, held from the scan through the claim:

acquire locks/allocator.lock
    scan occupancy into a set
    walk to the first candidate the set does not hold and no lock covers
    claim it (per-conversation lock, as today)
    re-read occupancy for that one ID
    still free: keep it. occupied: release the claim, advance, repeat
release locks/allocator.lock

The re-read is what makes the snapshot safe to act on. The scan describes the directories at one instant and the lock test describes the locks at a later one, so a writer that persists and releases in between reads as free on both: absent from the snapshot, and unlocked by the time the walk asks. Taking the observations in the other order settles it. Nobody else can claim while this process holds the allocator lock, so a candidate whose lock this process now holds and whose directory is absent from a fresh read was unclaimed at both instants. The up-front scan stays as the cheap skip-list; the re-read costs one more scan per allocation, not one per candidate. It also takes allocation out of reach of the release-then-unlink defect: two processes holding flocks on different inodes for the same path still cannot both keep the candidate, because the loser's re-read finds the winner's directory.

Partition moves are the one transition the scan cannot serialize.archive_conversation holds the conversation lock across the rename (jp_workspace/src/lib.rs), so the lock test covers it. unarchive_conversation takes no lock at all (ticket T-02ynsca), so a rename out of .archive/ can slip between the two halves of a scan and be missed by both. The union therefore reads .archive/ before the active partition — the ordering cleanup_stale_files already uses against the same defect — so whichever read follows the rename finds the directory. An archived ID only becomes a candidate after a backwards clock correction past its creation time, so the ordering is cheap insurance rather than a live hazard.

The lock file is never unlinked, so it is a stable inode and the release-then-unlink hazard does not apply to it. Nothing may prune it while allocation can run, including the reclamation pass that exists today (see Risks). The critical section is two directory scans, so serializing it costs nothing for a single-user CLI and leaves per-conversation lock semantics untouched. Occupancy checking fails closed: a scan that returns an I/O error fails the allocation rather than proceeding on unknown state.

The allocator lock belongs to LockBackend, which already owns exclusion (jp_storage/src/backend/lock.rs). It gains one operation returning a guard that releases the OS lock on drop and leaves the pathname in place: flock for FsStorageBackend, a process mutex for InMemoryStorageBackend, unconditionally granted by NullLockBackend (as try_lock already is, since --no-persist writes nothing and promises no uniqueness). It is not try_lock with a synthetic identifier — that signature accepts "allocator", but its guard unlinks the path on drop (ConversationFileLock::drop in jp_storage/src/lock.rs), the one property this lock must not have.

Contention waits briefly, then fails. try_lock is non-blocking and the creation path turns Ok(None) into an immediate error (lock_new_conversation in jp_workspace/src/lib.rs); inheriting that would fail one of two concurrent forks on a lock held for the duration of a scan. The allocator lock blocks with a bounded retry against a constant, ALLOCATOR_LOCK_WAIT, set to one second, then returns a typed contention error. It does not prompt and does not reuse style.lock_wait, which covers a ten-second human-scale wait on a lock held by another session, ending in an interactive choice (acquire_lock in jp_cli/src/cmd/lock.rs). This lock is held by a scan, and there is nothing for the user to decide.

The release-before-unlink defect still affects write exclusion on existing conversations; that is a pre-existing bug, out of scope here. create_conversation, the unlocked variant, takes no allocator lock and its doc comment says it carries no guarantee; every production caller uses the locking variants (cmd/query.rs, conversation/fork.rs).

Drift budget

The allocator can run ahead of wall-clock time, and the consumers listed above treat the ID's timestamp as a time: an ID 100 seconds ahead is excluded by a --created-before cutoff at the current time and expires 100 seconds late. Drift decays during idle time, but it needs a ceiling.

MAX_ID_DRIFT is a constant, 5 seconds (50 slots). A candidate past now + MAX_ID_DRIFT returns a typed error. Two things push a candidate that far: a local burst, and a backwards clock step inside a process that has already allocated, whose high-water floor stays where the old clock left it. An imported future-dated ID is neither — it occupies one slot rather than moving the floor — so the error can name both causes it has:

Error: cannot allocate a conversation ID: allocation is 5s ahead of wall-clock
time. Either conversations are being created faster than the ID format allows
(10 per second), or the system clock moved backwards during this run. Slow down,
or open an issue describing the workload.

Five seconds covers every realistic burst (RFD 050 designs for N positional fork sources; a dozen is plausible) and keeps the error on derived behavior negligible. A clock step larger than the budget blocks creation in the process that saw it until wall-clock time catches up; a process started afterwards is unaffected, having no high-water mark to carry.

Exact created_at

Conversation gains created_at: Option<DateTime<Utc>>, #[serde(default, skip_serializing_if = "Option::is_none")], so conversations written before this change load unchanged and absent means "fall back to id.timestamp()". It carries no serialize_with, so it serializes as RFC 3339 rather than the "YYYY-MM-DD HH:MM:SS.f" shape the other timestamps use (Conversation in jp_conversation/src/conversation.rs): RFC 3339 is where those fields are headed, and parse_dt already reads both.

The allocating path stamps the field. create_and_lock_conversation* captures one instant, derives the decisecond floor from it, and writes it after the ID is claimed, overwriting whatever the caller passed in. Both production sites need that: jp query passes Conversation::default() (create_new_conversation in cmd/query.rs), and fork_conversation clones the source's metadata, resetting only last_activated_at and expires_at (conversation/fork.rs), so a fork would otherwise report its source's creation time. The same instant seeds ConversationStream::created_at — today derived from id.timestamp() (jp_storage/load.rs) — so the two cannot disagree. The explicit-ID variants, used only in tests, leave an existing created_at alone: restoring or importing preserves creation time, allocating does not.

Sort and filter paths keep reading id.timestamp(). Routing --created-since / --created-before through metadata would turn a free integer comparison into a metadata read per conversation, and their error is bounded by the drift budget. conversation ls -F json is the exception, and only for the value it reports: it already loads every conversation's metadata to build its rows (ls.rs, Workspace::conversations in jp_workspace/src/lib.rs), so reporting the stored created_at there costs nothing.

Drawbacks

• The ID stops being exactly the creation time. Anyone reading a raw ID as a precise timestamp is now slightly wrong, where before they were right whenever they were the only writer.
• IDs are no longer ordered against imported conversations, and a backwards clock jump reorders local ones. Cross-machine ordering was never real; this makes the loss explicit rather than disguised.
• Creation is serialized by a new lock and gains a failure surface if locks/allocator.lock becomes unwritable.
• One directory scan per conversation created. A twelve-source fork scans twelve times over two roots and two partitions. Bounded by the conversation count, and the same scan already runs on most listing paths.
• A new failure mode: creation can fail on drift where it previously succeeded with a colliding ID. A backwards clock step larger than MAX_ID_DRIFT does the same to a process that has already allocated, until wall-clock time catches up.
• created_at has no consumer that strictly needs it today. It exists because once the ID is approximate, something has to hold the exact value.

Alternatives

A persistent cursor file. A next_id counter on disk, incremented under a lock. Rejected: redundant state that desyncs from what it describes. In .jp/ it becomes a committed merge-conflict source; in user storage, wiping it silently re-enables collisions. Either way it has never seen conversations that arrived via git pull. The directory names are the record.

Sleep to bound drift. Rejected: the sleep sits under lock acquisition, serializing concurrent creators, and a silent multi-second stall in jp conversation new is indistinguishable from a hang. An error names the cause and invites the user to describe the workload.

Change the format. Added precision (centiseconds, milliseconds) lowers collision probability without establishing uniqueness, so fork A B stays a coin flip; a ULID or random suffix breaks the six timestamp consumers outright. Both invalidate every directory name on disk and lengthen a deliberately typeable ID.

Process-local counter only. A high-water mark in memory, no storage read. Fixes fork A B and nothing else. Rejected as a final design, but it is a strict subset of the proposed allocator, not a separate implementation path.

Allocate above the highest stored ID. Rejected: it cannot coexist with the drift budget. A conversation projected from a machine whose clock is a minute fast would push the floor a minute ahead, failing every local creation until wall-clock time caught up — an outage from ordinary clock skew, misdiagnosed as a creation-rate problem.

A dedicated allocation backend trait. Rejected: LoadBackend and LockBackend already own reading and exclusion, and a third trait would re-declare both for every backend that implements them. It adds a name, not a decision. (Implementation count is not the argument — it would have as many implementations as the two it wraps.)

Non-Goals

• Cross-machine collisions via git projection. The allocator is per-storage-root and cannot see the other side. Detecting and re-keying duplicate IDs is separate work; RFD 097 solves the same shape one level down for event IDs, and its repair half is the eventual answer here.
• What the storage layer does with a duplicate ID once one exists. Under collision, reconcile_conversation_dir renames one match into the target and remove_dir_alls the rest (jp_storage/src/lib.rs), destroying a live conversation. That is one of several operations that delete conversation data as a side effect; making all of them non-destructive is separate work. Neither effort gates the other: this RFD lowers the rate at which same-ID directories appear, the other makes them survivable.
• Reconciling copies that live one per root. Deciding whether two same-ID directories in different roots are replicas or distinct conversations revises the last-write-wins resolution RFD 031 defines. Out of scope here, and not covered by the displacement work either.
• User-supplied IDs. RFD 050 rejects --id for pre-generated IDs; that stays rejected.
• Changing how conversation directories are named or resolved.
• Raising the 10-per-second ceiling. A workload that needs more gets an error and a conversation, not a format change.

Risks and Open Questions

• Creation must persist for the sequential case to hold. A process leaves a directory behind only when the conversation was made dirty; ConversationMut::drop returns early otherwise (jp_workspace/src/conversation_lock.rs). A jp conversation new that never mutates would write nothing, and once its lock file is unlinked the next process has nothing to skip. RFD 050 owns that command; this is a constraint on it, not something this RFD can verify.
• Scan cost on very large workspaces. Ten thousand conversations across two roots and two partitions is a scan per allocation, inside the critical section, twelve times for a twelve-source fork. Worth measuring before assuming it is free.
• The reclamation that exists today would delete the allocator lock.Workspace::cleanup_stale_files runs at the end of every invocation (jp_cli/src/lib.rs) and removes every unheld *.lock in the user locks/ directory (jp_workspace/src/session_mapping.rs, list_orphaned_lock_files in jp_storage/src/lib.rs). An unlink landing between one process opening the path and another opening it leaves the two holding flocks on different inodes — the hazard the stable lock exists to avoid. Reclamation must skip this file by name; Phase 2 owns the change and the test.

Implementation Plan

Phase 1: Occupancy on LoadBackend

Add the occupancy operation to LoadBackend, implemented for FsStorageBackend as a union scan across both roots reading .archive/ before the active partition, and for InMemoryStorageBackend from its own map. Fails closed on I/O error.

• Test: a workspace holding an archived conversation reports that ID as occupied.
• Test: an ID present only in the user root is occupied.

Phase 2: Allocation

Add the allocator-lock operation to LockBackend and its three implementations, and exclude locks/allocator.lock from list_orphaned_lock_files so the end-of-run reclamation leaves it alone. Then add the allocator to Workspace: acquire the allocator lock, scan occupancy, walk from max(now, process_high_water + 1) to the first unoccupied candidate, claim it via the existing try_lock, re-read occupancy for that candidate and advance if it is no longer free, release. Route create_and_lock_conversation* through it. Document create_conversation as offering no collision guarantee.

• Test: jp conversation fork A B yields two distinct IDs and two surviving conversations. This reproduces the Motivation bug and must fail before this phase; it must not use --compact, whose per-iteration LLM call separates the deciseconds and passes vacuously.
• Test: cleanup_stale_files removes an orphaned conversation lock and leaves locks/allocator.lock in place.
• Test: an ID persisted by another process after this process's first allocation is skipped by its second (the rescan-under-lock case).
• Test: an occupancy answer that changes between the scan and the re-read makes the allocation release that candidate and advance, rather than keeping a slot another writer has published.
• Test: an occupied slot at now, directory present and no lock held, allocates now + 1 (the case a lock-only claim misses).
• Test: a second process holding the lock for T allocates T + 1.
• Test: a held allocator lock past ALLOCATOR_LOCK_WAIT fails the second process with the typed contention error rather than blocking or duplicating.

Phase 3: Drift budget

Add MAX_ID_DRIFT and the typed error.

• Test: an occupancy set filling every slot from now to now + MAX_ID_DRIFT produces the error rather than an ID.
• Test: a single future-dated occupied slot far beyond the budget is skipped, not treated as a floor — allocation succeeds at now.

Phase 4: Exact created_at

Add the field, stamp it in the allocating creation path from the instant that seeds the decisecond floor, prefer it over id.timestamp() when seeding ConversationStream::created_at, add the Created row and the top-level created_at key to conversation show, and switch the created_at key in conversation ls -F json to the stored value.

• Test: a fork of a conversation with a known created_at records its own creation time, not the source's.
• Test: a conversation created through jp query --new has created_at set.
• Test: a conversation written without the field loads and behaves as before.
• Test: a metadata.json round-trip emits RFC 3339 for created_at and the existing space-separated format for last_activated_at.
• Test: conversation show -F json carries a top-level RFC 3339 created_at.
• Test: conversation ls -F json reports the stored created_at when the conversation has one, and the ID's timestamp when it does not.

Phase 5: Vocabulary

Add a Conversation ID entry to docs/architecture/ubiquitous-language.md: an allocated identifier that approximates creation time, distinguished from created_at.

References

• RFD 031 — the storage-root pair that bounds the allocation domain.
• RFD 050 — the scripting surface that makes bursts routine, and the owner of the conversation new persistence constraint in Risks.
• RFD 054 — declared the ID canonical for ConversationStream.created_at; Phase 4 revises that.
• RFD 073 — the backend decomposition that occupancy checking extends.
• RFD 097 — insertion-time uniqueness plus load-time repair for event IDs.