Jean-Pierre

Appearance

View Markdown
Referenced byD64

RFD D02: Lossless Config Editing ​

Summary ​

Replace the lossy deserialize-mutate-serialize round-trip in config set with format-preserving editors for every config format JP supports (TOML, JSON, JSON5, YAML). This eliminates git diff noise when a user changes a single config value.

Motivation ​

JP supports four configuration file formats: TOML, JSON, JSON5, and YAML. The config set command currently reads the file, deserializes it into a PartialAppConfig, merges the delta, and re-serializes the entire struct back to disk. This full round-trip through serde destroys the original file's formatting:

  • Key reordering: serde serializes fields in struct definition order, not the order they appeared in the file.
  • Representation normalization: compact inline values get expanded. For example, a TOML alias anthropic = "anthropic/claude-sonnet-4-6" becomes a multi-line table with provider and name keys because the Deserialize impl accepts both forms but Serialize always produces the struct form.
  • Whitespace and comment destruction: comments in YAML, JSON5, and TOML are discarded entirely. Blank lines used for visual grouping disappear. Indentation style may change.
  • Quote style changes: TOML single-quoted keys become double-quoted. JSON5 unquoted keys gain quotes.

The result is that setting a single value like conversation.default_id=ask produces a diff that touches most of the file. For users who commit their JP config to version control (which we encourage), this creates noisy, unreviable diffs that obscure the actual change.

Example ​

Running jp cfg set --cfg conversation.default_id=ask on a 55-line TOML config produces a 130-line diff. Every alias, every nested key, and the key ordering are rewritten, even though only one value was added.

Design ​

Approach ​

Instead of round-tripping through serde, config set should:

  1. Parse the original file into a format-preserving document tree.
  2. Serialize only the delta (the changed fields) to the same format.
  3. Deep-merge the delta into the original tree, touching only the keys present in the delta.
  4. Write the modified tree back to disk.

Untouched content — comments, whitespace, key order, quote styles, blank lines — is preserved because it was never reparsed through serde.

Deep merge algorithm ​

The merge is recursive and applies to all formats:

rust
fn deep_merge(target: &mut Tree, source: &Tree) {
    for (key, source_value) in source {
        if target[key] is a map AND source_value is a map {
            deep_merge(&mut target[key], source_value)
        } else {
            target.insert(key, source_value)  // add or overwrite
        }
    }
}

For existing keys, the value at that position in the document is replaced. Surrounding whitespace and comments stay intact. For new keys, they are appended to the relevant section/object. The new key inherits the formatting conventions of its siblings (indentation, trailing commas, etc.) as best as the editor library supports.

Format-specific editors ​

Each format needs a library that can parse source text into an editable tree and write it back losslessly.

FormatEditorSource
JSON / JSON5json_editNew crate in crates/contrib
YAMLyaml-editExternal crate
TOMLtoml_edit orSee TOML strategy
json_edit-derived

JSON and JSON5: json_edit ​

No format-preserving JSON editor exists in the Rust ecosystem. We build one as crates/contrib/json_edit, designed for eventual publication to crates.io.

The crate is built on rowan, the lossless syntax tree library created for rust-analyzer. yaml-edit already uses rowan, so this is a shared dependency rather than a new one.

JSON's grammar is LL(1) and trivial compared to YAML. The full crate is estimated at ~900 lines including tests:

ComponentEst. linesNotes
SyntaxKind enum + rowan glue~50~15 token types
Lexer~150String escapes are the hardest part
Parser~150Recursive descent
High-level API (Document, Object,~250get, set, remove, insert
Array)
Tests~300

JSON5 extends JSON with: single-line comments (//), block comments (/* */), trailing commas, unquoted identifier keys, single-quoted strings, and additional number literals. These are handled in the lexer; the parser and high-level API are nearly identical.

The public API:

rust
let doc = json_edit::Document::parse(input)?;
let root = doc.as_object()?;
root.set("key", json_edit::Value::from("value"));
root.remove("old_key");
println!("{}", doc); // lossless output

YAML: yaml-edit ​

yaml-edit is an existing crate that provides exactly what we need: lossless YAML parsing and editing, also built on rowan. It preserves comments, whitespace, and formatting through in-place syntax tree mutations.

Its API maps directly to our deep-merge pattern:

rust
let doc = yaml_edit::Document::from_str(content)?;
let mapping = doc.as_mapping()?;
mapping.set("key", "value");    // preserves surrounding formatting
mapping.remove("old_key");

yaml-edit is at version 0.2.1 with ~1.7k downloads. The API is clean but the crate is young. We should pin to a specific version and monitor for breaking changes.

TOML strategy ​

toml_edit is the established solution for lossless TOML editing (it underpins Cargo's Cargo.toml manipulation). It works well for our use case, but has one documented limitation: it does not preserve the order of dotted keys (e.g., a.b.c = 1 may be rewritten as a nested table).

Two options:

  1. Use toml_edit as-is. The dotted-key limitation is minor — most users write section headers ([section]) rather than deeply dotted keys. This is the pragmatic choice.

  2. Build a toml_edit equivalent on top of json_edit's rowan infrastructure. After building json_edit, we have a lexer/parser framework that could be extended to TOML. TOML's grammar is more complex than JSON but simpler than YAML. This gives us full control over formatting preservation, including dotted keys. The cost is ~1500-2000 additional lines.

This RFD recommends option 1 for the initial implementation. If the dotted-key limitation causes real user complaints, option 2 becomes viable with low marginal effort since the rowan infrastructure already exists.

Integration with ConfigFile ​

ConfigFile::merge_delta becomes the single entry point. It dispatches to the appropriate editor based on self.format:

rust
pub fn merge_delta<T: Serialize>(
    &mut self,
    delta: &T,
) -> Result<(), Box<dyn Error + Send + Sync>> {
    match self.format {
        Format::Json | Format::Json5 => self.merge_json(delta),
        Format::Yaml => self.merge_yaml(delta),
        Format::Toml => self.merge_toml(delta),
    }
}

Each merge_* method follows the same pattern:

  1. Parse self.content into the format's lossless document type.
  2. Serialize delta to a serde_json::Value (a common intermediate representation that all formats can consume).
  3. Deep-merge the intermediate value into the lossless document.
  4. Write the document back to self.content.

Step 2 uses serde_json::Value as the interchange format because all four formats can round-trip through it without loss for the subset of types PartialAppConfig uses (strings, numbers, booleans, arrays, objects).

Interaction with config fmt ​

The config fmt command intentionally normalizes formatting — it runs a full deserialize-serialize round-trip. This RFD does not change config fmt. The distinction is:

  • config set: preserve the user's formatting, change only what was requested.
  • config fmt: normalize the entire file to a canonical format.

format_content continues to use the serde round-trip.

Drawbacks ​

  • New dependency on rowan (~4 crates in its dependency tree). Mitigated by the fact that yaml-edit already brings it in.
  • Maintenance cost of json_edit. We own this crate. JSON's grammar is stable and small, so ongoing maintenance should be minimal, but it is nonzero.
  • Three different editor backends. Each format has its own tree representation and API. The merge_delta dispatcher and the shared deep-merge-via-serde_json::Value pattern keep the integration surface small, but bugs could manifest differently per format.

Alternatives ​

Span-guided string surgery ​

Parse the original file with a span-tracking parser (e.g., json-spanned-value), then use byte offsets to surgically splice replacement values into the raw string. This avoids needing a full lossless tree.

Rejected because: insertion of new keys requires finding the right position, matching indentation, and handling trailing commas — essentially rebuilding the formatting logic that a lossless tree provides for free. Also, json-spanned-value only handles standard JSON (not JSON5) and depends on indexmap v1 (we use v2).

Serde round-trip with preserve_order ​

Use serde_json with preserve_order for JSON, accept the whitespace normalization, and only fix TOML/YAML.

Rejected because: key ordering is only part of the problem. Comment preservation matters for JSON5 and YAML, and indentation normalization creates diff noise even with preserved ordering.

Do nothing ​

Accept the noisy diffs.

Rejected because: users commit config files to version control. Noisy diffs from config set erode trust in the tool and discourage use of the command.

Non-Goals ​

  • Conflict-free concurrent editing. config set reads, modifies, and writes the file non-atomically. This RFD does not add file locking or merge conflict resolution.
  • Schema migration. If a config key is renamed across JP versions, that is handled by the config loading pipeline, not by the editor.
  • Formatting preferences. This RFD preserves existing formatting; it does not allow users to configure a preferred style for new keys (e.g., "always use 2-space indent"). New keys inherit the style of their surroundings as best as the editor library supports.

Risks and Open Questions ​

  • yaml-edit maturity. The crate is at 0.2.1. It may have parsing bugs on edge-case YAML documents. Mitigation: pin the version, add integration tests against realistic JP config files, and contribute fixes upstream.
  • JSON5 comment variants. JSON5 allows // and /* */ comments. The json_edit lexer must handle both, including comments inside objects and arrays. This is straightforward but must be tested thoroughly.
  • toml_edit dotted key reordering. If users rely on dotted keys like a.b.c = 1, toml_edit may rewrite them as nested tables. The toml_edit issue tracker documents this. Monitor for user reports.
  • Empty delta serialization. PartialAppConfig skips None fields during serialization. If the delta is empty (all fields None), the serialized output is "" or {}. The merge must handle this as a no-op without corrupting the file.

Implementation Plan ​

Phase 1: json_edit crate ​

Create crates/contrib/json_edit with:

  • Rowan-based lexer and parser for JSON and JSON5.
  • Document, Object, Array, Value wrapper types.
  • set, remove, insert operations that preserve formatting.
  • Unit tests covering: comment preservation, whitespace preservation, key ordering, nested edits, trailing commas (JSON5), unquoted keys (JSON5).

Can be reviewed and merged independently. No changes to JP's config pipeline.

Phase 2: JSON/JSON5 integration ​

Wire json_edit into ConfigFile::merge_delta for Format::Json and Format::Json5. Add integration tests that verify config set on JSON and JSON5 files produces minimal diffs.

Depends on Phase 1.

Phase 3: YAML integration ​

Add yaml-edit as a dependency. Wire it into ConfigFile::merge_delta for Format::Yaml. Add integration tests with comment-heavy YAML configs.

Independent of Phases 1-2. Can be done in parallel.

Phase 4: TOML integration ​

Add toml_edit as a dependency. Wire it into ConfigFile::merge_delta for Format::Toml. Add integration tests verifying alias strings, key order, and comments are preserved.

Independent of Phases 1-3. Can be done in parallel.

Phase 5 (optional): Rowan-based TOML editor ​

If the toml_edit dotted-key limitation or other formatting issues prove problematic, build a TOML lexer/parser on the same rowan infrastructure as json_edit. This replaces the toml_edit dependency.

Depends on Phase 1 (for the rowan infrastructure) and Phase 4 (for user feedback on toml_edit limitations).

References ​