# RFD 066: Content-Addressable Blob Store

- **Status**: Discussion
- **Category**: Design
- **Authors**: Jean Mertz <git@jeanmertz.com>
- **Date**: 2025-07-17

## Summary

This RFD introduces a content-addressable blob store that externalizes all
content payloads from `events.json`. Resource content (attachments and tool
responses) is stored as gzip-compressed blobs keyed by SHA-256 checksum.
`events.json` carries compact blob references instead of inline content, keeping
it a small structural skeleton of conversation metadata. A background task
garbage-collects unreferenced blobs on every JP invocation.

## Motivation

[RFD 065] places resource content inline in `events.json` — both on
`ChatRequest.resources` (attachments) and in `ToolCallResponse` content blocks
(tool results). As conversations grow, this creates several problems.

### Readability and editability of `events.json`

A core goal of JP is that users can read and hand-edit `events.json` files. With
large base64-encoded content payloads inline, the file becomes difficult to
navigate. A single attached source file can add thousands of characters of
base64 to what is otherwise a readable JSON structure. Users regularly
copy-paste parts of a conversation into a new conversation, and inline blobs
make that unwieldy.

### Disk space and duplication

A `--attach ./src` on a modest project can resolve to hundreds of files. Each
file's content is serialized into `events.json`. Multiple snapshots of the same
resource — from re-attachments, refreshes, or the same file read by multiple
tool calls — multiply the stored content. Team members sharing conversations via
Git transfer this bulk content inline, with no deduplication across
conversations that reference the same files.

### Context window pollution for future tools

If JP gains the ability for LLMs to search conversation history (a planned
feature), inline content payloads would be pulled wholesale into the context
window. An LLM searching for a conversation about a design decision does not
need the raw bytes of every file that was attached — it needs metadata (URIs,
MIME types, timestamps) and should fetch content separately only when relevant.

### Parse time

Every `jp query` invocation deserializes `events.json`. JP already uses
`serde_json::RawValue` for fast-path operations (conversation listing, metadata
display), so not all operations pay the full deserialization cost. But
operations that do need the full event stream — building the LLM request,
forking, compaction — are slower with megabytes of inline content.

## Design

### Overview

All content payloads — resource content from attachments, tool response text,
and tool response resource blocks — are stored in a content-addressable blob
store at `.jp/blobs/`. `events.json` carries compact references (checksum +
size) instead of inline content. The blob store is workspace-scoped, enabling
cross-conversation deduplication. Blobs are gzip-compressed on disk. Content is
loaded lazily, only when building the LLM request.

### Blob storage

Blobs are stored with a two-level directory prefix (2 + 2 characters) derived
from the SHA-256 checksum:

```
.jp/
  blobs/
    a1/
      b2/
        c3d4e5f6789...abc.blob.gz
    fe/
      dc/
        0123456789...xyz.blob.gz
```

The two-level prefix provides 65,536 directory buckets (256 × 256), giving
headroom for long-lived workspaces with heavy tool use. Each leaf directory
contains blob files whose checksums share the same 4-character prefix.

Each blob file contains the gzip-compressed raw content bytes. No base64, no
JSON wrapping. The checksum in the filename is the integrity check — to verify,
re-hash the decompressed content and compare.

### Gzip compression

All blobs are gzip-compressed on disk. This serves two purposes:

1. **Search opacity.** Compressed files do not match text searches in `rg`,
   `grep`, or editor search-and-replace. Content payloads are currently
   base64-encoded in `events.json` for this same reason; gzip compression
   preserves the property when content moves to separate files.

2. **Size reduction.** Text content (source code, tool output) typically
   compresses 60–80% with gzip. This is especially valuable since blobs are
   committed to Git for team sharing (see below).

### Always external

All content payloads are externalized to the blob store, with no size threshold.
A tool response of "check succeeded" (16 bytes) is stored as a blob, same as a
500KB source file.

This avoids conditional logic ("is this content big enough to externalize?") and
prevents inline content from appearing in contexts where it causes problems:

- **Copy-pasting conversation history.** Users sharing conversation excerpts
  with colleagues or other LLMs would include inline content, wasting space and
  leaking potentially sensitive file contents.
- **Future search-over-history tools.** An LLM tool that searches conversation
  history would pull inline content into its context window, wasting tokens on
  raw file contents when it only needs metadata.

The filesystem cost of small blobs (a gzip-compressed 16-byte string is ~36
bytes on disk, plus one inode) is negligible. The implementation simplicity of
one code path outweighs the marginal storage overhead.

### Content representation in `events.json`

Content payloads in `events.json` are wrapped in a `content` object that
supports three variants:

**Blob reference** (JP's default for all writes):

```json
{
  "type": "resource",
  "resource": {
    "uri": "file:///project/src/main.rs",
    "mimeType": "text/x-rust",
    "content": {
      "$blob": "a1b2c3d4e5f6789...abc",
      "size": 12345
    }
  }
}
```

The `$blob` field contains the SHA-256 hex digest. The `size` field records
the uncompressed content size in bytes, enabling display and token estimation
without reading the blob.

**Inline text** (for user hand-edits):

```json
{
  "type": "resource",
  "resource": {
    "uri": "file:///project/src/main.rs",
    "mimeType": "text/x-rust",
    "content": {
      "text": "fn main() {}"
    }
  }
}
```

**Inline binary** (base64-encoded, for user hand-edits of binary content):

```json
{
  "type": "resource",
  "resource": {
    "uri": "file:///project/logo.png",
    "mimeType": "image/png",
    "content": {
      "blob": "<base64-encoded data>"
    }
  }
}
```

The same three variants apply to `ContentBlock::Text` tool responses:

```json
{
  "type": "text",
  "content": {
    "$blob": "fe98dc0123456789...xyz",
    "size": 847
  }
}
```

Or after a user hand-edit:

```json
{
  "type": "text",
  "content": {
    "text": "cargo check returned without errors"
  }
}
```

The `content` object always contains exactly one discriminant key: `$blob`,
`text`, or `blob`. Deserialization maps these to `BlobContent::Ref`,
`ResourceContent::Text`, and `ResourceContent::Blob` respectively.

#### Why inline variants matter

Users hand-edit `events.json` to fix conversations — for example, changing a
failed tool call to a success requires editing both `is_error` and the response
content. With only blob references, the user would need to create a
gzip-compressed file with the correct checksum filename. The inline `text` and
`blob` variants let users write content directly in `events.json`.

JP always writes `$blob` references. When JP reads an `events.json` that a
user has edited with inline content, it deserializes the inline data normally.
On the next write (e.g., when the conversation advances), JP re-externalizes
all content to the blob store. The inline forms are a user convenience for
reads and hand-edits, not a format JP produces.

Existing conversations without the `content` wrapper (pre-blob-store format)
are handled by backward-compatible deserialization — bare `text` strings and
base64 `blob` fields are read as inline content.

### What stays in `events.json`

`events.json` remains the source of truth for conversation structure. It
contains all metadata — URIs, MIME types, checksums, tool call IDs, question
blocks, annotations, config deltas, turn boundaries — everything except raw
content payloads. The file stays small and fast to parse regardless of how much
content the conversation references.

### Content addressing

The blob store uses SHA-256 as its content-address key. When JP writes a
resource to the store:

1. Compute SHA-256 of the raw content bytes.
2. Check the blob store — if a blob with that hash already exists, skip the
   write.
3. If new, gzip-compress and write to `blobs/<prefix>/<hash>.blob.gz`.
4. Write the blob reference to `events.json`.

The checksum is computed once on write and exposed via `BlobContent::Ref` for
any consumer that needs content identity.

### Cross-conversation sharing

The blob store is shared across all conversations in a given storage location.
If two conversations attach the same file (same content, same SHA-256), only
one blob exists on disk. This is a natural consequence of content-addressing
— no additional dedup logic is needed at the storage layer.

JP has two storage locations:

- **Workspace storage** (`.jp/` in the project directory): conversations
  shared with the team via Git. The blob store lives at `.jp/blobs/`. Blobs
  are committed to Git alongside `events.json` — team members need blob
  content to read, continue, or fork conversations. Without the blobs,
  `events.json` contains dangling references and conversations are
  unreadable.

- **User-local storage** (`~/.local/share/jp/workspace/<project>/` or
  platform equivalent): conversations private to the user, not committed to
  Git. The blob store lives at the corresponding `blobs/` directory within
  this location.

Each storage location has its own independent blob store. Cross-conversation
dedup works within each store (workspace blobs dedup with other workspace
conversations, user-local blobs dedup with other user-local conversations).
Dedup across the two stores is a non-goal — the storage locations serve
different purposes and sharing blobs between them would complicate the
ownership model.

#### Viewing content

`jp conversation print` resolves content references transparently and renders
the full conversation with actual content inline. The user sees file contents
and tool responses, not checksums. The blob store is invisible during normal
conversation viewing.

#### Selective VCS staging

The shared content store means that staging a single conversation for version
control requires knowing which files it depends on. `jp conversation show
--files <id>` lists all filesystem paths for a conversation (events.json plus
referenced content files). Users pipe this to their VCS of choice:

```sh
git add $(jp conversation show --files <id>)
hg add $(jp conversation show --files <id>)
```

JP remains VCS-agnostic: it outputs paths, the user's toolchain consumes them.

### Lazy loading

Blob content is loaded only when needed — primarily when building the LLM
request that sends content to the provider. Operations that only need
conversation metadata (listing conversations, displaying titles, forking,
reading event structure) work with `events.json` alone and never touch the blob
store.

The deserialization layer produces a lazy wrapper type instead of eagerly
loading content:

```rust
/// Content that may be loaded lazily from the blob store.
pub enum BlobContent {
    /// Content loaded in memory.
    Loaded(Vec<u8>),
    /// Reference to content in the blob store, not yet loaded.
    Ref {
        checksum: String,
        size: u32,
    },
}
```

`ResourceContent` and `ContentBlock::Text` use `BlobContent` internally. Code
that needs the actual bytes calls a `resolve` method that reads from the blob
store on first access. Code that only needs metadata (URI, MIME type, checksum)
never triggers a load.

### Writing blobs

When JP writes a new event containing content (a `ChatRequest` with resources,
or a `ToolCallResponse` with content blocks):

1. For each content payload, compute SHA-256.
2. Check if the blob exists in the appropriate store (workspace or user-local,
   matching the conversation's storage location).
3. If not, gzip-compress the raw bytes and write to a temporary file in the same
   directory, then atomically rename to the final path. The rename is atomic on
   POSIX, so readers never see a partially-written blob.
4. Write `events.json` with `$blob` references.

If JP crashes after writing the blob but before writing `events.json`, the
blob is orphaned — an unreferenced file in the store. The garbage collector
handles this (see below).

### No locking required

Blobs are immutable and content-addressed. This eliminates most concurrency
concerns:

- **Concurrent writes of the same checksum** produce identical bytes. The atomic
  rename means one process wins; the other's temp file is cleaned up. The result
  is correct either way.
- **Reads during writes** are safe because the final path either doesn't exist
  yet (blob not visible) or is complete (rename is atomic). Readers never see
  partial content.
- **GC races** are the one real hazard: the GC sweep can delete a blob between
  the time a write creates it and the time `events.json` records the reference.
  The write path mitigates this by verifying blob existence after writing
  `events.json` and re-creating the blob if it was deleted. Since blobs are
  immutable and content-addressed, re-creation produces the identical file. The
  worst case is a redundant write, not data loss.

### Garbage collection

Unreferenced blobs accumulate when conversations are deleted, forked (old
references dropped), or compacted (turns removed). A background task runs on
every JP invocation to clean up orphans. The sweep runs independently for each
storage location (workspace and user-local):

1. Scan all conversations' `events.json` files in this storage location and
   collect every referenced checksum into a `HashSet`.
2. List all blob files in this location's `blobs/` directory.
3. Delete any blob whose checksum is not in the referenced set.

This is a full sweep, not an incremental scan. It is cheap because `events.json`
files are small metadata-only skeletons (all content is in the blob store). For
a workspace with 100 conversations and 1000 blobs, the sweep reads 100 small
files and scans 1000 filenames — milliseconds of work.

The sweep runs as a background task using JP's existing task infrastructure. LLM
queries take seconds to minutes; the GC sweep completes unnoticed in the
background. No manual `jp gc` command, no refcount state, no cursor tracking.
One task, one `HashSet`, one directory walk.

## Drawbacks

**Every content access requires a filesystem read.** Even a 16-byte "check
succeeded" message requires opening a file, decompressing, and reading. In
practice this is microseconds per blob, and content is only loaded when building
LLM requests — not during conversation listing or metadata operations. The cost
is measurable but not meaningful.

**Inode overhead for small blobs.** Each blob consumes one inode and one
directory entry. A conversation with 500 tool calls produces 500 blob files.
Modern filesystems handle this without issue, but it is more filesystem pressure
than a single `events.json`.

**Git repository size.** Blobs must be committed for team sharing. A workspace
with extensive conversation history accumulates blob files in Git. Gzip
compression helps (text blobs compress 60–80%), and cross-conversation dedup
avoids storing the same content twice. But long-lived workspaces with heavy tool
use will grow their `.jp/blobs/` directory over time. Git LFS is an option for
teams where this becomes a problem, but is not designed in this RFD.

**Full GC sweep scales linearly.** The sweep reads all conversations'
`events.json` on every invocation. With thousands of conversations this could
take noticeable time. For typical workspaces (tens to low hundreds of
conversations), the cost is negligible. If scaling becomes a problem, a refcount
index can be layered on without changing the blob store format.

## Alternatives

### Size threshold for externalization

Externalize only content above a threshold (e.g., 4KB), keeping small payloads
inline in `events.json`.

**Rejected because:** Conditional logic adds two code paths for every
serialization and deserialization site. Small inline content still appears in
copy-pasted history and search-over-history tools. The marginal storage savings
for small blobs do not justify the implementation complexity.

### Uncompressed blob storage

Store blobs as raw bytes without gzip compression.

**Rejected because:** Uncompressed text blobs appear in workspace-wide text
searches (`rg`, `grep`, editor find-and-replace). Gzip makes blobs opaque to
text tools regardless of Git or editor ignore configuration. The compression
CPU cost is negligible for the blob sizes involved.

### SQLite blob store

Store blobs in a SQLite database (e.g., `.jp/blobs.db`) with a
`(checksum, content)` table.

**Rejected because:** The filesystem already provides O(1) lookup by checksum
(the path is deterministic), crash-safe writes (write to temp file, rename),
and Git-compatible storage. SQLite adds a dependency and a binary format that
is harder to inspect, debug, and share via Git. A database would be justified
if we needed richer queries over blob metadata, but we do not — the only
operation is "read blob by checksum."

### Refcount-based garbage collection

Maintain a separate refcount index tracking which conversations reference each
blob. Decrement on conversation delete, delete blobs with zero references.

**Rejected for now because:** With `events.json` being small metadata-only
files, a full sweep is cheap enough that the refcount's added complexity
(crash consistency between refcount file and events.json, handling of orphaned
refcount entries) is not justified. A refcount index can be added later as an
optimization if the full sweep becomes expensive with thousands of
conversations.

### Sidecar file per conversation

Store a single append-only sidecar file per conversation mapping checksums to
content.

**Rejected because:** Linear scan for lookups (O(n)), no cross-conversation
deduplication, base64 encoding still needed for binary content in a text-based
format, and line-based framing requires escaping for binary content.

## Non-Goals

- **Blob encryption.** Blobs are stored as gzip-compressed content without
  encryption. Workspace-level encryption is a separate concern.

- **Git LFS integration.** Large blob files in Git are a potential concern for
  long-lived workspaces. Integration with Git LFS or similar large-file
  storage is deferred.

- **Blob deduplication across workspaces.** Cross-conversation dedup within a
  storage location is handled by content-addressing. Dedup across separate
  workspaces or between workspace and user-local storage is out of scope.

- **Streaming blob access.** Blobs are read fully into memory on access. For
  the content sizes involved (source files, tool output), this is appropriate.
  Streaming access for very large blobs (video, large datasets) is not
  designed here.

## Risks and Open Questions

### Blob file count on resource-constrained filesystems

A workspace with heavy tool use accumulates many small blob files. On
filesystems with limited inodes (e.g., some default ext4 configurations), this
could theoretically exhaust inodes before exhausting disk space. In practice,
default inode counts are generous enough that this is unlikely for typical
development workspaces. If it becomes a problem, a packed blob format
(multiple blobs in one file with an index) could replace individual files.

### Concurrent writes from parallel conversations

[RFD 020] introduces parallel conversations that may write blobs
simultaneously. Content-addressing makes this safe — two processes writing the
same checksum produce the same file. For different checksums, the directory
fanout makes contention unlikely. The write pattern (write to temp file,
rename to final path) is atomic on POSIX filesystems.

### GC race with in-progress writes

The GC sweep could delete a blob between the time a write computes its
checksum and the time it writes the events.json reference. Mitigation: the
write path checks blob existence and re-creates it if missing before writing
events.json. Since blobs are immutable and content-addressed, re-creating a
deleted blob produces the identical file.

### Migration from inline conversations

Existing conversations store content inline in `events.json`. The
deserialization layer handles both formats (inline and blob-ref). On first
access after migration, existing conversations continue to work with inline
content. A migration tool (`jp migrate-blobs` or automatic on next write)
could extract inline content to the blob store and rewrite `events.json` with
blob references, but is not required — both formats coexist.

## Implementation Plan

### Phase 1: Blob store and write path

Create the `.jp/blobs/` directory structure. Implement blob write (SHA-256,
gzip, write to `<prefix>/<hash>.blob.gz`). Implement the `BlobContent` lazy
type. Update `events.json` serialization to write `$blob` references for all
content payloads.

Can be merged independently. Existing conversations with inline content
continue to work via the backward-compatible deserializer.

### Phase 2: Lazy deserialization

Update `events.json` deserialization to detect `$blob` fields and produce
`BlobContent::Ref` values. Implement the `resolve` method that reads and
decompresses from the blob store on first access.

Depends on Phase 1.

### Phase 3: Background garbage collection

Implement the GC sweep as a background task: scan conversations, build
referenced set, delete orphans. Register the task in JP's task infrastructure
so it runs on every invocation.

Depends on Phase 2.

### Phase 4: Migration tooling

Implement optional migration for existing conversations: read `events.json`
with inline content, extract payloads to blob store, rewrite with blob
references. This can be automatic (on next conversation write) or manual
(`jp conversation migrate`).

Depends on Phase 2.

## References

- [RFD 065: Typed Resource Model for Attachments][RFD 065] — defines the
  `Resource` type whose content payloads are externalized by this RFD.
  Identifies inline content bloat as a blocking concern requiring this solution.
- [RFD 058: Typed Content Blocks for Tool Responses][RFD 058] — defines
  `ContentBlock` and `Resource` types whose content payloads are externalized.
- [RFD 036: Conversation Compaction][RFD 036] — compaction drops old turns,
  potentially orphaning blobs that GC cleans up.
- [RFD 020: Parallel Conversations][RFD 020] — parallel writes to the blob
  store are safe due to content-addressing.

[RFD 020]: 020-parallel-conversations.md
[RFD 036]: 036-conversation-compaction.md
[RFD 058]: 058-typed-content-blocks-for-tool-responses.md
[RFD 065]: 065-typed-resource-model-for-attachments.md