# RFD D37: Conversation Query DSL

- **Status**: Draft
- **Category**: Design
- **Authors**: Jean Mertz <git@jeanmertz.com>
- **Date**: 2026-05-16

## Summary

Introduce a small predicate language for querying conversations, evaluated by a
new `jp_query` crate and exposed through a `--filter EXPR` flag on `jp c ls`,
`jp c grep`, `jp c rm`, `jp c archive`, and `jp c fork`.
The language supports boolean predicates over conversation metadata, the
configuration tree, and conversation events, with explicit `event(...)` and
`turn(...)` scoping for binding predicates to a single record.

## Motivation

Locating a conversation today means combining several mechanisms:

- `jp c ls` filters by metadata flags (`--archived`, `--local`, `--sort`).
- `jp c grep` searches event content textually.
- `ConversationTarget` keywords (`?`, `?p`, `last`, `+session`) resolve to
  specific IDs.

None of them answer questions like "which conversation called `fs_modify_file`
with `path = crates/jp_cli/src/lib.rs`?", "which archived conversations used a
specific model?", or "which conversations contain a chat response mentioning X
*and* are older than a week?".
Composing those queries today requires post-processing JSON or chaining commands
with brittle text matching, and even then no path gets at structured event
fields (tool name, tool arguments) without exposing the on-disk JSON shape.

Tool call arguments are also base64-encoded on disk specifically to keep them
out of editor and `rg` results, so any structured search must round-trip through
the conversation runtime.
That makes a JP-native query engine the natural — and effectively the only —
place for this capability.

A single predicate language solves these cases with one mechanism, composes
naturally across multiple commands, and gives plugins and scripting workflows a
programmatic surface for finding conversations.

## Design

### Surface

A new `--filter EXPR` flag is added to:

- `jp c ls` — show matching conversations.
- `jp c grep` — search text within matching conversations.
- `jp c rm` — remove matching conversations (with destructive-UX rules below).
- `jp c archive` — archive matching conversations.
- `jp c fork` — fork all matching conversations.

<!-- end list -->

```sh
jp c ls --filter 'archived and assistant.model == "anthropic/claude-sonnet-4-5"'
jp c ls --filter 'tool == "fs_modify_file" and arg.path == "crates/jp_cli/src/lib.rs"'
jp c grep --filter 'tool == "fs_modify_file"' 'TODO'
jp c rm  --filter 'created < "1 month ago" and not pinned'
```

The flag accepts an expression directly, `@path/to/file.qry` to read from a
file, or `-` to read from stdin.

### Grammar

```
expr      := or_expr
or_expr   := and_expr ('or' and_expr)*
and_expr  := not_expr ('and' not_expr)*
not_expr  := 'not' not_expr | primary
primary   := '(' expr ')' | scope | predicate
scope     := ('event' | 'turn') '(' expr ')'
predicate := field (op value)?

field        := dotted_field | bare_field
dotted_field := '.' segment ('.' segment)*
bare_field   := ident ('.' segment)*
segment      := ident | quoted_string
ident        := [a-zA-Z_][a-zA-Z0-9_-]*

op       := '==' | '!=' | 'contains' | '~' | '<' | '>' | '<=' | '>='
value    := string | number | bool
string   := '"' (escape | non_dquote_non_backslash)* '"'
          | "'" non_squote* "'"
number   := signed integer or floating point literal
bool     := 'true' | 'false'
escape   := '\n' | '\t' | '\r' | '\"' | '\\' | '\u{HEX}'
```

Operator precedence: `not` \> `and` \> `or`.
Parentheses override.

A predicate may omit the operator and value when the field is boolean-typed:
`archived` is sugar for `archived == true`.
`not pinned` reads as expected.

### Quoting

Both single and double quotes delimit strings:

- `"..."` — escapes processed: `\n`, `\t`, `\r`, `\"`, `\\`, `\u{HEX}`
  (Rust-style; 1–6 hex digits).
- `'...'` — raw, no escapes.

Both produce semantically identical strings.
The split exists for shell composition:

```sh
jp c ls --filter "title contains '$FOO'"   # shell expands $FOO
jp c ls --filter 'title contains "$FOO"'   # literal $FOO
```

### Field paths

Two equivalent forms:

```
tool                         # canonical, bare
arg.path                     # canonical, bare with chain
arg."request body"           # bare first, quoted later
metadata."x-trace-id".value  # mixed
."weird field"               # quoted first segment requires the dot
."weird field".sub           # same, with continuation
"weird field"                # invalid — parses as string literal
```

The dot prefix is only required when the *first* segment must be quoted; once
past the first segment, a leading `.` already disambiguates a path continuation
from a string literal.

Documentation and error messages use the bare form unless the first segment must
be quoted.
A field name that collides with a reserved keyword (`and`, `or`, `not`, `true`,
`false`, `event`, `turn`) is escaped with a leading dot: `.and`.

### Field namespace

Three top-level roots, declared in a static registry:

| Root                  | Cost                  | Examples                                                                       |
| --------------------- | --------------------- | ------------------------------------------------------------------------------ |
| Conversation metadata | cheap (no event load) | `title`, `archived`, `pinned`, `local`, `created`, `updated`, `messages`       |
| Configuration tree    | cheap (no event load) | `assistant.model`, `assistant.reasoning.enabled`, `conversation.tool.style`, … |
| Event fields          | event load required   | `event`, `tool`, `arg.*`, `content`                                            |

Configuration-tree fields mirror the `AppConfig` schema exactly.
Only primitive-typed config fields (string, number, bool) are queryable in v1;
list and map types fall back to a parse-time error.

Unknown fields are a **parse-time error**, not a silent "no match."
A typo of `archvied` returns a clear `unknown field 'archvied'` with position
information.

### Types

Four primitive types: `string`, `number`, `bool`, `date`.

Numbers are `i64` and `f64`; mixed comparison promotes int to float.

Dates parse from string literals on the right side of a comparison when the
field is date-typed.
Two forms accepted:

- **Absolute** (RFC 3339 / ISO 8601): `"2026-01-01"`, `"2026-01-01T12:00:00Z"`.
- **Relative**: `"N <unit> ago"` where `<unit>` is one of `second`, `minute`,
  `hour`, `day`, `week`, `month`, `year` (plural accepted).
  Also `"now"`.

<!-- end list -->

```
created > "2026-01-01"
created > "1 day ago"
updated > "2 weeks ago"
created < "now"
```

Resolution order on the right side of a date comparison: RFC 3339 →
relative-time → parse-time error.

Type mismatches (`messages > "ten"`, `archived < 3`) are parse-time errors where
statically detectable, runtime errors otherwise.
No implicit string-to-number coercion.

### Semantics

Each field has an **intrinsic record level**: conversation, turn, or event.
Expressions evaluate against a conversation, with these rules:

1. **Same-record-level binding is the default.** Predicates over fields at the
   same record level are joined at that level.
   `tool == "X" and arg.path == "Y"` — both event-level — bind to the *same
   event*.

2. **Cross-level mixing broadcasts the higher-level field.** `assistant.model ==
   "X" and tool == "Y"` reads as "the conversation's model is X *and* there
   exists an event with tool=Y."
   Conversation-level fields act as constants when joined with event-level
   predicates.

3. **Event-level predicates are existentially quantified at the top level.**
   `tool == "X"` matches conversations that contain at least one event with
   tool=X.

4. **`not` over event-level predicates is universal.** `not tool == "X"` means
   "no event in this conversation has tool=X" — the natural reading, and the
   one that makes negation reachable without de Morgan acrobatics.

5. **Explicit scoping with `event(...)` and `turn(...)`.** Force a binding scope
   when the default isn't enough:

<!-- end list -->

```
# Default: same-event binding (tool name + tool arg on the same call)
tool == "fs_modify_file" and arg.path == "crates/jp_cli/src/lib.rs"

# Cross-event: called X in some event AND Y in some (possibly different) event
event(tool == "X") and event(tool == "Y")

# Same-turn binding: both calls happened within one turn
turn(tool == "X" and tool == "Y")
```

A *turn* is defined by `TurnStart` event boundaries in the conversation stream
— the existing ubiquitous-language definition.

Inside `event(...)` or `turn(...)`, conversation-level fields are broadcast as
constants (same rule as at the top level).
Nested scopes (`turn(event(P))`) are accepted; redundant nesting is not
rejected.

### Cost-aware execution

The evaluator computes, from the AST alone, whether an expression references any
event-scoped fields.
Expressions that touch only conversation metadata and configuration **must not
load event streams**.

This is non-negotiable: it is cheap to enforce at design time and expensive to
retrofit.
`jp c ls --filter 'archived'` runs at the same cost as `jp c ls --archived`.

### Destructive command UX

`jp c rm --filter`, `jp c archive --filter`, and similar destructive commands
carry a real risk: a typo or an unexpected operator semantic can affect many
conversations at once.

The rule:

- A destructive command with `--filter` prints a list of affected conversation
  IDs and asks for confirmation, unless `--yes` is passed.
- `-F json` output mode bypasses the prompt; scripts pass `--yes` explicitly to
  suppress prompts.
- A `--dry-run` flag shows what would be affected without doing it.

This makes the safety behavior part of the command's contract from day one,
before scripts accumulate.

### Architecture

A new crate `jp_query` owns the parser, AST, type registry, and evaluator:

- `Expression` — typed AST.
- `parse(s: &str) -> Result<Expression, ParseError>` — produces a
  position-aware parse error on failure.
- `Expression::touches_events(&self) -> bool` — for cost-aware dispatch.
- `Expression::evaluate(&self, ctx: &EvaluationContext) -> bool` —
  `EvaluationContext` carries the conversation metadata, merged config, and an
  optional event stream.

The crate depends on `jp_conversation` (for `ConversationStream`, `EventKind`)
and `jp_config` (for the config schema).
It does not depend on `jp_workspace`, `jp_cli`, or any storage backend.
The evaluator is pure — no I/O.

`jp_cli` adds a single `FilterArg` clap type, flattened into each subcommand
that supports `--filter`.
The arg type accepts `EXPR`, `@FILE`, and `-` (stdin).

## Drawbacks

**A second small grammar in the codebase.** `ConversationTarget` already has a
tiny grammar (`?`, `+session`, etc.).
This adds another.
They live at different layers (selector vs. predicate) and do not compose
textually, but contributors now have two mini-languages to keep in mind.

**Hyrum's Law on field names.** Every config key and every conversation metadata
field becomes queryable, which means renaming any of them is a DSL-breaking
change.
The field registry is the load-bearing surface; rename moves require coordinated
changes to the registry, the docs, and any external scripts.

**Cost-model surprise from the same flag.** `--filter 'archived'` is cheap;
`--filter 'tool == "X"'` is expensive.
The same flag, two cost regimes.
Cost-aware execution hides the regime from users — which is the right tradeoff
for ergonomics, but means a user can't tell from the surface why one query is
fast and another is slow.

## Alternatives

### Flag-based predicates (no DSL)

Add `--tool NAME`, `--arg KEY=VALUE`, etc. to a new `jp c find` command.

Rejected.
Composes poorly: AND/OR/NOT across predicates requires either flag explosion
(`--tool`, `--not-tool`, `--any-tool`, `--all-tool`) or implicit-AND semantics
that can't express the motivating query.
Same-event binding via flags is unnatural.
Path Independence: rebuilding JP today with current scripting ambitions would
not land on flags.

### Reuse jq via `jaq-core`

Accept jq syntax for filter expressions, wrap user input in `select(...)`
internally, feed jq a synthetic per-event JSON view.

Rejected.
The semantic model we want (record-level binding, conversation broadcast,
`turn`/`event` scoping) is not native to jq — we would own the entire semantic
layer anyway, with jq serving as a low-level boolean evaluator.
Cost-aware execution becomes much harder: jq's AST is more permissive than our
predicate AST, validation has to walk it twice.
Two v1 requirements — `"1 day ago"` syntax and shell-friendly single quotes —
are not natural in jq.
Net dependency cost (~5kLoC of `jaq-core`) for a benefit that erodes once the
semantics are layered on top.

### Predicates on metadata-only filters

Skip the event-content angle entirely; `--filter` operates only on conversation
metadata and config.

Rejected.
The motivating use case — "which conversation modified file X?" — needs event
predicates.
A metadata-only filter doesn't solve the problem this RFD exists for.

### Defer relative-time syntax to v2

Originally proposed during design.
Reversed: relative times are a major ergonomic win on time-based queries, cost
~100 lines of isolated parser code, and fit cleanly into the existing
string-literal date rule with no grammar impact.

## Non-Goals

- **Transformations.** Output reshaping (`jp c show -F json | jq ...`) is jq's
  job.
  The DSL is a predicate language, not a pipeline language.
- **Aggregations.** No `count(...)`, `sum(...)`, `min(...)`, `max(...)` in v1.
- **List / map config values.** `conversation.tool.allow` and similar
  collection-typed config keys are not queryable in v1.
  Adding support requires defining `in`, `subset`, `any(...)` operators.
- **Cross-event temporal ordering.** "Called X and *then later* called Y" is not
  expressible.
  Use `turn(...)` for same-turn binding; broader temporal queries are deferred.
- **Custom functions.** No user-defined functions, no built-in scalar functions
  (`upper`, `length`, etc.).
- **Date arithmetic in function form.** `created > now() - "1 day"`-style
  expressions are deferred.
  The string-literal form (`"1 day ago"`) covers v1.
- **`ConversationTarget` keyword integration.** Keywords (`?`, `last`,
  `+session`) remain selectors, separate from `--filter`.
  They compose externally: resolve targets, then filter the set.
- **Saved queries and aliases.** Out of scope for v1.

## Risks and Open Questions

### Final field inventory

The exact v1 list of conversation-metadata and event fields needs ratifying in
the implementation.
The categories are settled; the exact names and shapes are not.

### Shape of the `event` field

Two candidate forms:

- Flat: `event == "tool_call_request"` (string of the kind tag).
- Nested: `event.kind == "tool_call_request"` (struct).

The flat form is simpler and matches the predicate-language style.
Pinned as default unless implementation surfaces a reason to nest.

### Error reporting quality

Position-aware parse errors are a hard requirement; type-mismatch errors should
point at the offending field and surface its registered type.
`ParseError("syntax error")` is not acceptable.
The implementation commits to a clear error story up front.

### Destructive command behavior contract

The dry-run / confirmation / `--yes` rules in the Design section are part of the
public contract once shipped.
Worth one more pass during review to make sure the defaults match user
expectations.

### Long expressions

`--filter @file.qry` and `--filter -` (stdin) are part of v1, but the
file-loading details (encoding, comment syntax, multi-line formatting) are not
yet specified.

## Implementation Plan

### Phase 1: `jp_query` crate

Create `jp_query` with AST types, parser, and evaluator.
Define the static field registry covering conversation metadata, the config tree
(primitives), and event fields.
Implement `Expression::touches_events()` for cost-aware dispatch.
Implement the evaluator over `&EvaluationContext`.
Tests: parser round-trips, semantic correctness on representative streams,
error-message coverage.

Reviewable and mergeable independently.

### Phase 2: `--filter` on read-only commands

Add `FilterArg` to `jp_cli` and integrate `--filter` on `jp c ls` and `jp c
grep`.
Cost-aware dispatch is exercised here; event-loading kicks in only when the
expression references event fields.

Depends on Phase 1.

### Phase 3: `--filter` on destructive commands

Add `--filter` to `jp c rm`, `jp c archive`, and `jp c fork`.
Implement the dry-run / confirmation / `--yes` UX.
End-to-end tests for the safety paths.

Depends on Phase 2 (to share the `FilterArg` type and dispatch wiring).

### Phase 4: Documentation

User-facing documentation under `docs/features/`, with a concentrated examples
section.
Where possible, the field registry is generated from the config schema to avoid
drift.

## References

- [RFD 050]: Scripting Ergonomics for Conversation Management — the broader
  scripting story this DSL plugs into.
- [`docs/architecture/ubiquitous-language.md`][ubiquitous] — definitions of
  Conversation, Turn, Event, and related terms used throughout this RFD.

[RFD 050]: ../050-scripting-ergonomics-for-conversation-management.md
[ubiquitous]: ../../architecture/ubiquitous-language.md