Jean-Pierre

Appearance

View Markdown
References064

RFD D17: Per-Tool Compaction Hints and Automatic Compaction ​

Summary ​

This RFD extends the compaction system from RFD 064 with per-tool compaction hints that let individual tools control how their calls are stripped, and automatic compaction that triggers when conversations approach the context window limit.

Motivation ​

RFD 064 delivered compaction as an explicit, user-initiated operation with uniform stripping. Two gaps remain:

  1. Uniform stripping is too coarse. ToolCallPolicy::Strip applies identically to all tools. But fs_read_file arguments (a file path) are cheap to keep, while fs_create_file arguments (full file content) dominate the token count. Without per-tool hints, users choose between stripping too aggressively (losing useful context like file paths) or too conservatively (keeping bulk they don't need).

  2. Manual compaction requires vigilance. Users must notice degradation and run jp conversation compact at the right time. In long-running coding sessions, context windows fill gradually and quality degrades before the user intervenes.

Design ​

Per-Tool Compaction Hints ​

Tools declare how their calls should be compacted via a new compaction section in their tool config:

toml
[conversation.tools.fs_read_file.compaction]
request = "keep"
response = "strip"

[conversation.tools.fs_create_file.compaction]
request = "strip"
response = "keep"

[conversation.tools.fs_modify_file.compaction]
request = "strip"
response = "strip"

Each field accepts "keep" or "strip". When absent, the field inherits from the profile's ToolCallPolicy. A tool with response = "keep" is exempted from response stripping even when the active profile sets response: true.

Config Type ​

A new ToolCompactionConfig struct is added as an optional field on ToolConfig:

rust
pub struct ToolCompactionConfig {
    /// How to handle this tool's request arguments during compaction.
    /// `None` inherits from the active compaction profile.
    pub request: Option<ToolFieldMode>,

    /// How to handle this tool's response content during compaction.
    /// `None` inherits from the active compaction profile.
    pub response: Option<ToolFieldMode>,
}

pub enum ToolFieldMode {
    Keep,
    Strip,
}

Projection Integration ​

The projection layer (stream/projection.rs) currently applies ToolCallPolicy::Strip uniformly via strip_tool_request and strip_tool_response. With per-tool hints:

  1. Before projection, build a map of tool name → ToolCompactionConfig from the stream's resolved config.
  2. When stripping a tool call request, check if the tool has compaction.request = "keep". If so, skip stripping for that request.
  3. When stripping a tool call response, check if the tool has compaction.response = "keep". If so, skip stripping for that response.

The tool name is already available on ToolCallRequest. For responses, the existing tool_names map (built during projection) provides the lookup.

Default Hints ​

JP should ship sensible defaults for its built-in tools in the workspace tool config files:

ToolrequestresponseRationale
fs_read_filekeepstripPath is cheap; file content isn't
fs_grep_fileskeepstripPattern is cheap; matches aren't
cargo_checkkeepstripArgs are cheap; output isn't
cargo_testkeepstripArgs are cheap; output isn't
fs_create_filestripkeepContent is bulk; "created" is cheap
fs_modify_filestripstripBoth carry large diffs
git_commitstripkeepMessage is bulk; hash is cheap

Automatic Compaction ​

Automatic compaction fires when the projected conversation size approaches the model's context window. It evaluates after each turn completes (before persisting).

Trigger ​

A character-based token estimate determines when to compact:

estimated_tokens = character_count / 4
threshold = model.context_window * auto.trigger_ratio

When estimated_tokens > threshold and the conversation has more turns than auto.min_turns, compaction is applied using the configured profile.

Configuration ​

toml
[conversation.compaction.auto]
enabled = false
trigger_ratio = 0.75
profile = "default"
min_turns = 5

Automatic compaction is disabled by default. It must be explicitly opted into because compaction is lossy and the token estimation is approximate.

Behavior ​

When triggered:

  1. Resolve the profile from auto.profile.
  2. Resolve the range: from = AfterLastCompaction, to = FromEnd(keep_last).
  3. If the profile has summary, generate the summary (LLM call).
  4. Append the compaction event.
  5. Log the compaction (turn range, profile, estimated token reduction).

The compaction runs synchronously within the turn loop, between turns. This is acceptable because mechanical strategies are fast, and summary generation (which adds latency) is opt-in via the profile.

Context Window Discovery ​

Automatic compaction needs the model's context window size. This is available from ModelDetails (returned by provider.model_details()). The turn loop already has access to the provider and model details. If the context window is unknown (e.g. a local model without metadata), automatic compaction is silently skipped.

Prompt Cache Interaction ​

Adding a compaction event changes the projected prefix, invalidating cached conversation history. For automatic compaction, this could cause unexpected latency spikes. Mitigation: automatic compaction fires between turns, when the cache is already partially invalidated by the new assistant response. The system prompt cache (a separate prefix) is unaffected.

Drawbacks ​

  • Per-tool hints add config surface. Every tool gains an optional compaction section. Mitigation: hints are optional and inherit from the profile by default. Most users never set them.

  • Automatic compaction is lossy and invisible. Users may not realize their conversation has been compacted. Mitigation: disabled by default, logged when it fires, original events always preserved.

  • Token estimation is approximate. Character-based estimation can be off by 2–3x depending on content. The trigger_ratio provides a safety margin, and a more accurate tokenizer-based approach can replace it later without changing the compaction model.

Alternatives ​

Tokenizer-based estimation ​

Use tiktoken or a model-specific tokenizer for accurate counts. Rejected for now: adds a dependency, requires per-model tokenizer selection, and a conservative trigger_ratio with character-based estimation is sufficient for the trigger decision.

Automatic compaction as a background task ​

Run compaction asynchronously (like title generation) so it doesn't block the turn loop. Rejected: compaction modifies the event stream, and concurrent modification during a turn would require synchronization that doesn't exist today. Between-turn compaction is simpler and safe.

Non-Goals ​

  • Token-accurate estimation. This RFD uses character-based approximation. Precise tokenization is a future refinement that doesn't change the compaction model.

  • Per-tool compaction strategies beyond keep/strip. Custom per-tool compaction functions (e.g. "summarize this tool's output") are interesting but add significant complexity. The keep/strip binary is sufficient for the common cases.

Risks and Open Questions ​

  • What trigger_ratio works in practice? 0.75 is a guess. Conversations with lots of tool calls may need a lower ratio since tool responses dominate token count. Needs experimentation.

  • Should automatic compaction notify the user? A subtle indicator ("context compacted") during the next response could help, but adds UI complexity. The log is sufficient initially.

Implementation Plan ​

Phase 1: Per-Tool Compaction Hints ​

  1. Add ToolCompactionConfig and ToolFieldMode to jp_config.
  2. Add optional compaction field to ToolConfig with trait impls.
  3. Update the projection layer to accept a tool config map and check per-tool overrides before stripping.
  4. Add default hints to the JP tool config files.
  5. Tests.

Can be merged independently. No behavioral change for users who don't configure hints.

Phase 2: Automatic Compaction ​

  1. Add AutoCompactionConfig to jp_config.
  2. Add token estimation function.
  3. Wire the trigger check into the turn loop (after turn completion, before persist).
  4. Reuse existing compaction logic to build and append the compaction event.
  5. Add logging.
  6. Tests (with mock provider for context window size).

Depends on Phase 1 for per-tool aware stripping. Can be merged independently from Phase 1 if per-tool hints are not required for the trigger logic.

References ​

  • RFD 064 — Non-Destructive Conversation Compaction