Jean-Pierre

Appearance

View Markdown
ReferencesT-0yh5frrT-0yh5npc
Referenced byT-0yh5frrT-0yh5npc

Rustqual baselines go stale when main moves ​

  • Status: Todo
  • Kind: Chore
  • Authors: jp
  • Date: 2026-10-06
  • Label: domain=tooling
  • Label: type=follow-up

just qual-ci compares the workspace against counts committed in .config/rustqual/baseline.json and baseline-dry.json. Those counts describe the tree at the moment they were recorded, so they go stale as soon as main moves.

The first real run of the gate demonstrated it. The baselines were recorded on 22 Sep; rebasing the branch onto 6 Oct main put 382 extra findings in the main pass and 116 in the DRY pass, across twelve categories, none of them from the branch — its whole Rust diff was seven comment lines. The gate was right that the numbers had moved and had no way to say whose numbers they were.

Why it matters ​

The author who hits the failure is whoever rebases next, and the documented response (just qual-baseline) makes them re-record a baseline covering two weeks of someone else's code. Refreshing is the only action that turns the job green, it always works, and it takes one command — so the gate teaches contributors to reach for the thing that silences it. That is the failure mode the gate exists to prevent, arrived at from the other direction.

The refresh is at least visible: the baselines are pretty-printed, so "total_findings": 2404 → 2786 shows up in the pull request diff where a reviewer can see it. That is a review-time catch, not a tooling one.

Options ​

  1. Accept it. Refresh on rebase and treat the failure as noise. Free, and normalizes the reflex.
  2. Re-record from main on a schedule. A workflow runs qual-baseline on main and commits the result, so a branch compares against a baseline that is at most a day old. One extra workflow; the drift window shrinks but does not close, and the commits are noise in the log.
  3. Drop the committed baseline. Analyse HEAD and the merge-base in the same job and compare the two reports. Staleness becomes impossible, because both numbers come from the run. Costs four analyses per job instead of two, and needs the merge-base tree checked out — probably a second worktree.

Option 3 removes the whole class of problem and is the one to aim for. Worth measuring how long one rustqual pass takes over the workspace before committing to it: if a pass is seconds, the extra cost is irrelevant and option 3 wins outright.

Introduced with the gate itself in #1190. The same pull request already replaced rustqual's own --compare --fail-on-regression, which scored a regression as the aggregate quality ratio dropping and so missed new findings entirely; .config/rustqual/regressions.jq now does a per-category comparison. Whatever replaces the committed baseline should keep that filter — the per-category check is orthogonal to where the numbers come from.

Comments ​

  • From: jp
  • Date: 2026-10-06T15:03:44Z

Observed: the gate fires on commits the branch does not contain ​

First real CI run of qual-ci on #1190 failed with srp_module_warnings: 207 -> 208 (+1) while the same recipe passed locally with identical numbers on both sides.

Cause is narrower than "baselines go stale", and worse:

.github/workflows/rust.yml:167 checks out with no ref:, so a pull_request run gets refs/pull/N/merge — the head merged into current main. The branch was rebased onto 150ba7868 and its baselines re-recorded there (e7e3b424c). main then took dbafe5485 (#1242, 16:30) and dfae02a91 (#1207, 16:33) the same afternoon. CI measured branch + dfae02a91; the baseline describes branch + 150ba7868. The +1 is a file crossing a threshold in that delta — #1242 adds net +21 lines to crates/jp_cli/src/cmd/query/tool/prompter.rs, which is otherwise unflagged.

Two consequences beyond staleness:

  1. Not reproducible locally. No sequence of local commands shows the failure, because the tree CI measured does not exist in any worktree. The author's only lever is to rebase and re-record — which is the reflex this ticket exists to prevent, now with no diagnostic alternative.
  2. The comparison is invalid in both directions. A committed baseline describes exactly one tree. Compared against a merge tree, it invents regressions and can mask real ones: if the merge happens to delete findings elsewhere, a genuine new finding nets to zero and passes.

So a committed baseline and the merge ref cannot both be right. The measured tree and the baseline have to share a base. That makes option 3 (analyse HEAD and the merge-base in the same job, no committed baseline) the correct target rather than merely the tidiest.

A cheaper interim exists: a per-task checkout_ref through matrix.include, set to github.event.pull_request.head.sha for qual only, leaving test and the other tasks on the merge ref where testing the merge result is the point. Needs one thing verified first — whether actions/checkout treats an empty ref: as unset, which is what the other matrix entries would pass.

  • From: jp
  • Date: 2026-10-06T21:46:15Z

Decided: option 3, with no committed baseline. Findings are compared one by one by (rule, file, symbol) rather than per-category counts, and the comparison lives in a new qual crate (T-0yh5frr). CI compares the merge ref against HEAD^1 in the same job (T-0yh5npc), so the measured tree and the base always match and the interim checkout_ref workaround isn't needed. This ticket closes when T-0yh5npc lands.