RFD D59: Cross-Session Rate-Limit Budgets

• Status: Draft
• Category: Design
• Authors: Jean Mertz git@jeanmertz.com
• Date: 2026-09-04

Summary

Providers meter usage per account, but each jp process retries in isolation, so several sessions sharing one API key overdraw the same bucket and then retry into it together. This RFD records what the provider reports about the remaining budget in a user-local store keyed by credential, and makes a session wait before sending when that budget is nearly spent. The budget is observed, never predicted: JP stores what the provider said, not what JP thinks a request will cost.

Motivation

Issue 1069 asks for per-minute throttling across all sessions against Cerebras. Cerebras enforces its limits per organization, and jp runs one process per conversation, so parallel sessions (RFD 020) on one key contend for a bucket none of them can see.

Today the only defence is the retry loop, which is reactive by construction: the request goes out, comes back 429, and the turn stalls for the retry-after the provider names. With max_retries at its default of 5 and Cerebras answering retry-after: 60, an unlucky session can spend five minutes waiting and still fail.

Why a client-side ledger cannot work

The obvious design is a shared ledger: estimate what a request will cost, decrement a counter, refill it at the limit rate. Two things rule it out, and they have different reach.

Every provider: the meter counts consumers JP cannot see

A rate limit is enforced against an account, not against a process. Another key on the same account, a colleague's machine, a curl in someone's terminal, a script nobody remembers writing. Even a perfect record of what JP itself spent would be missing an unbounded amount of the total, so a ledger built from JP's own arithmetic is wrong by an amount it cannot measure.

This holds however well a provider documents its charging, and it alone is enough to reject the ledger.

Cerebras specifically: the charging rule is not derivable

The second argument is narrower, and is offered as corroboration rather than as the basis for the design. It is what the measurements below actually establish, and it establishes it for one provider.

Cerebras takes its reservation at admission, before the response exists. Six readings with a short prompt gave exactly min(max_completion_tokens, 16384) with no contribution from the prompt at all:

max_completion_tokens	remaining after	reserved
1	499,999	1
512	499,488	512
4,096	495,904	4,096
16,384	483,616	16,384
40,960	483,616	16,384
unset	483,616	16,384

The same request with a ~64,000-token prompt reserved 80,384, which is 64,000 + 16,384. So the prompt contributes at that size and not at 68 tokens. A prompt-caching explanation fit until the same large prompt was sent twice and the second request was charged more than the first, not less.

The rule may be coarse-grained input estimation, or something else. It is not derivable from Cerebras's documented behaviour, and it is not stable enough to encode. Whether any other provider is this opaque is unknown; several document their accounting clearly, and for those the first argument is the only one that applies.

What both point at

Stop computing, start reading. A provider that meters an account has to tell its clients where they stand, and Cerebras returns the answer on every successful response:

x-ratelimit-limit-tokens-minute: 500000
x-ratelimit-remaining-tokens-minute: 419616

Reading it is correct whether or not the charging rule is knowable, and correct whether or not JP is the only client. That is the property worth designing around.

Design

What the user sees

A session that would push the budget past its floor waits instead of sending, using the same shape as conversation-lock contention (acquire_lock): silent for a moment, then a timer line, then an interactive prompt if the wait runs long.

⏳ Waiting for rate-limit headroom (cerebras, tokens/minute) — 12s

? Rate limit nearly exhausted for cerebras (tokens/minute: ~8,200 of 500,000).
> Continue waiting
  Send anyway
  Cancel

Ctrl-C during the wait jumps straight to the prompt rather than aborting the turn, matching lock-wait behaviour. Non-interactive runs wait up to the timeout and then send regardless, because unlike a lock conflict a rate limit resolves itself: sending produces a 429 the existing retry layer already handles, and failing the command outright would be a regression on today's behaviour.

Configuration mirrors the existing pair, behaviour in its own section and presentation under style:

[rate_limit]
enable = true          # set false to restore today's send-and-retry behaviour

[style.rate_limit_wait]
show = true
delay_secs = 1
interval_ms = 100
timeout_secs = 10      # then prompt

The observation

/// What a provider reported about one of its rate-limit buckets, at one instant.
pub struct RateLimitSnapshot {
    pub bucket: BucketId,          // provider-labelled: unit plus scope
    pub limit: Option<u64>,
    pub remaining: u64,
    pub recovery: Recovery,
    pub observed_at: DateTime<Utc>,
}

/// How a bucket returns to full, as the provider describes it.
pub enum Recovery {
    /// Fully replenished at this instant.
    FullAt(DateTime<Utc>),

    /// Refills continuously at this many units per second, capped at `limit`.
    Continuous { per_second: f64 },

    /// Not reported. The reading is only good for the instant it was taken.
    Unknown,
}

Recovery is an enum rather than a rate because providers describe replenishment differently, and the difference matters: Anthropic states the absolute instant a bucket is whole again, Cerebras documents a continuous refill and states nothing per response, and a provider that resets on a fixed boundary does neither. Assuming any one of these on a provider that means another invents headroom that does not exist.

Providers extract zero or more snapshots from a response head:

trait Provider {
    /// Rate-limit buckets this provider reported in a response head.
    fn snapshots_from_headers(&self, _headers: &HeaderMap) -> Vec<RateLimitSnapshot> {
        vec![]
    }
}

The default returns nothing, so a provider that reports nothing simply gets no throttling.

The method is named for its source because a response head is not the only one a provider could have. See Ceilings, readings and other sources below for what else exists and why none of it is built here.

Two providers are known to supply the raw material, and they supply different amounts of it. Cerebras yields six snapshots (tokens and requests, across minute, hour and day) with a limit and a remaining each and no reset of any kind, so its recovery is Continuous derived from limit / window. Anthropic documents anthropic-ratelimit-{requests,tokens,input-tokens,output- tokens}-{limit,remaining,reset} on every Messages API response, where reset is an RFC 3339 instant, so its recovery is FullAt straight from the header and needs no arithmetic at all. OpenAI has in-tree evidence but only half of it: extract_retry_after already parses x-ratelimit-reset-requests and x-ratelimit-reset-tokens for retry timing, and whether it also reports remaining has not been checked.

This keeps providers and buckets orthogonal: a new provider is one adapter, a new bucket kind touches no provider.

Snapshots are taken from the response head, not at stream completion. Headers arrive before the first byte of body, so a stream that fails or is interrupted still yields its observation — and those are exactly the observations taken during the trouble that makes throttling worth having.

Ceilings, readings and other sources

A snapshot bundles two facts with very different half-lives. The ceiling (limit) changes when a plan changes, so a stale one is harmless. The reading (remaining) is stale within seconds. They arrive together in a response head today, which is why one struct carries both, but nothing in the design requires that they always do: limit is an Option precisely so a provider that reports remaining without a ceiling can still participate.

Some providers expose the ceiling separately. Anthropic has a Rate Limits API that lists the configured limits for an organization and its workspaces. It is worth being precise about what that endpoint does and does not offer, because it is easy to read as a way to ask how much room is left:

• It returns {type, value} pairs where value is the configured limit. There is no remaining. Anthropic's own documentation frames it as something to compare against usage data from a separate API, not as a source of usage.
• It requires Admin API credentials: an admin key, an OAuth token with org:admin, or an unscoped account key. The workspace-scoped key JP would use for inference does not work, and the endpoint is unavailable to individual accounts entirely.

So for Anthropic the endpoint would supply a number JP already gets for free from anthropic-ratelimit-*-limit, at the cost of a second privileged credential most users do not have. No active source is built here, and the trait carries no method for one.

The axis is named rather than paved. A provider could plausibly report remaining from a queried endpoint rather than a response header, and if one does, adding snapshots_from_query beside snapshots_from_headers costs nothing that the current shape has spent. Building that plumbing now, for a case no provider presents, would be paying for a second source before there is one.

The store

One file per credential under the user data directory (jp_workspace::user_data_dir), holding the most recent snapshot per (unit, window):

$JP_USER_DATA_DIR/rate-limits/cerebras-a3f19c2e.json

The store is keyed by an opaque credential id that the shell supplies. It never learns how that id was derived, and deliberately so. Today a provider authenticates with a single API key, so the id is a truncated SHA-256 of it: enough to tell two keys apart without storing anything reversible, and key rotation starts a fresh budget for free. RFD 090 replaces that with a chain of named credential profiles, some of them OAuth tokens with no API key to hash, at which point the id becomes the provider and profile name that RFD already established as a credential's identity. The store does not change when that happens; only what fills the id does.

A credential chain means a session can move between credentials mid-turn when one is exhausted, and each has its own meter. Switching credential switches bucket, which falls out of keying on the id rather than on the provider.

The id is never logged. Writes go through the advisory locking in jp_storage::lock and land atomically. Any failure to read, parse, or lock the store degrades to "no budget known", which sends. A throttle that can stall the CLI is worse than no throttle.

The decision

Before a provider request, a session loads the snapshots, projects each forward to now, and waits while any projected remaining sits below a floor.

Projection follows the snapshot's Recovery, which the adapter filled in from what the provider actually said. FullAt interpolates towards the stated instant, Continuous adds per_second capped at limit, and Unknown projects nothing: the reading stands as taken and ages into uselessness rather than into invented headroom.

That the shared logic never assumes a recovery model is the point. Anthropic hands over a reset instant and Cerebras hands over nothing, and a single rate baked into the decision would be wrong for one of them.

The floor cannot be a prediction of what this request will cost, for the reasons in the Motivation. It is instead the largest single drop observed recently: the difference between consecutive snapshots, corrected for refill, is the measured drain of real traffic including consumers JP cannot see. The store keeps the last few drops per bucket and uses their maximum.

With no history the floor is zero and JP sends, so a first run is never worse than today.

Where the pieces live

jp_llm owns the snapshot type and the per-provider extraction. Neither needs to know whose credential produced the response: an adapter reads headers and returns numbers.

The shell owns the credential id, the store, the decision and the wait. That split is not a preference. RFD 090 rules that credential identity never crosses the provider boundary, and a credential-keyed store inside jp_llm would break that rule the moment that RFD lands. Prompting is a terminal concern besides.

The collect-path callers (collect_with_retry, used by title generation, summarization and inquiries) consume quota too and take the same wait, but silently and bounded, since there is nobody watching to answer a prompt.

Drawbacks

A new blocking state in jp query. Today a query either streams or fails. Adding a wait before the request means a new state to interrupt, explain and test, and it is the bulk of the work in this proposal. The accounting is bookkeeping; this is the cost.

Behaviour now depends on state other processes wrote. A session's decision is shaped by a file another jp may be writing concurrently. The degradation rule above bounds the damage, but the coupling is real and did not exist before.

The floor is a heuristic and will be wrong sometimes. Too low and JP still hits 429s, which is today's behaviour. Too high and JP throttles itself harder than the provider would, which is a new failure mode and a worse one, because it is invisible: the user sees a wait with no external cause.

One more thing to keep current. Every provider's header dialect is a hand-maintained mapping against an API that moves, in the same way the model tables are.

Alternatives

Predict the charge and keep a ledger. The original plan, and the reason this RFD exists in its current form. Rejected on the evidence in the Motivation: the charging rule is not derivable, and even a correct one would miss consumption by other clients on the same account.

Derive the budget from recorded usage. A companion design records per-request token usage on the conversation stream. Summing it looks like it would answer the same question, but it measures what JP spent rather than what the account has left, and the gap between them is unbounded. It would also require reading every conversation's stream to answer a question scoped to sixty seconds. The two designs read different parts of the response (body versus headers) at different times, and share a theme rather than a seam.

Serialize provider requests across sessions with a lock. Correct and simple, and it throws away the parallelism RFD 020 exists to provide.

Do nothing beyond the retry layer. Reacting to the 429 is what this design falls back to, and that fallback is sound on its own terms: the wait honours the provider's Retry-After, and PR 1080 spreads concurrent sessions so they do not all resume on the same instant. The case against leaving it there is that reacting costs a full round trip each time, and on a small plan the bucket is exhausted for most of every minute.

Non-Goals

Anticipating consumers JP cannot see. Another machine, another key, a script. Every reading already includes their effect, since the provider meters the account rather than the process — that is the whole reason for reading rather than computing. What JP cannot do is see them coming, so a burst from elsewhere between two observations will still produce a 429.

Organizations sharing one key across machines. The store is user-local by design. Two developers on one key each keep their own view and each believe they have the full budget. The failure mode is graceful: they learn from the 429, which is today's behaviour.

Multiple credentials on one account. Cerebras meters per organization, so two keys on one org share a meter while this design gives each its own budget. The credential is the best identity JP has, not the true one, and nothing a client can observe distinguishes the two cases.

Replacing the retry layer. This is additive. The 429 path stays exactly as it is and remains the backstop.

Recording usage or cost. A separate concern with a separate home.

Risks and Open Questions

Can we see the response headers at all? reqwest_eventsource::EventSource takes ownership of the request and its Event::Open carries no headers, so the successful path may have no access to the very data this design needs — while the error path already has it via InvalidStatusCode. That would be backwards. The likely route is dropping to reqwest plus eventsource-stream (which reqwest-eventsource is itself built on, and which is already a dev-dependency of jp_llm) in the providers that implement the trait. This needs a spike before the rest of the plan is worth scheduling, because it decides whether the change is confined to a new module or reaches into every streaming provider.

How should the floor be tuned? The largest-recent-drop rule is a starting point chosen because it needs no formula, not because it has been validated. It cannot be validated without real multi-session traffic. Worth shipping behind rate_limit.enable and revisiting with data.

Which bucket dominates? Cerebras reports six. A day-window bucket sitting at zero should probably not trigger the same wait as a minute-window one, since the wait would be hours. The design likely needs to ignore buckets whose window exceeds some bound, and that bound is unsettled.

Does a 429 update the store? Cerebras strips the x-ratelimit-* headers from its 429 responses, so the most informative moment yields no observation. The fallback is to record "remaining was zero at this instant" from the status code alone, which is a different shape from a snapshot and may not be worth the special case.

A bucket's identity can shift under its own name. Anthropic's anthropic-ratelimit-tokens-* headers report whichever limit is currently most restrictive, so the same header names describe an organization limit on one response and a workspace limit on the next. A store keyed by header name would silently compare two different buckets and conclude the budget had jumped. The anthropic-workspace-id response header names the scope a request counted against, which is probably the discriminator BucketId needs, and confirming that is part of the Anthropic adapter rather than a blocker for the design.

Readings can be deliberately imprecise. Anthropic rounds every *-remaining to the nearest thousand. A floor computed from differences between rounded numbers inherits that error, which matters most when the remaining is small, which is exactly when the decision is being made. The floor may need to carry the reported precision rather than assume exactness.

Two records about the same credential. RFD 090 gives each credential profile an exhausted_until timestamp, persisted in the user data directory, so a fresh invocation skips a credential whose quota has not reset. That is a coarse form of what this design stores: both say when a credential is usable again, keyed the same way, in the same place. They answer different questions — one is a hard cooldown after a billing quota error, the other a soft headroom estimate from a refilling bucket — and collapsing them would conflate the fatal case with the transient one, which InsufficientQuota and RateLimit are deliberately kept apart elsewhere. But two stores about one credential is a smell, and whoever builds the second of the two should decide whether they share a file. Neither exists yet, so this is not blocking.

Implementation Plan

Phase 1 — Spike: response-head access. Determine whether a streaming provider can read response headers without abandoning reqwest_eventsource. Output is a decision, not necessarily code. Everything else depends on it.

Phase 2 — Snapshot type and Cerebras extraction. RateLimitSnapshot, BucketId, Recovery, the defaulted trait method, and the Cerebras adapter with unit tests against captured header sets. Cerebras fills Recovery::Continuous from limit / window, so the token-bucket arithmetic lands in the adapter rather than in the shared decision. Mergeable alone; nothing consumes it yet.

Phase 3 — The store. Load, atomic write, advisory locking, the credential id, and the degrade-to-send rule on every failure path. Mergeable alone, still unconsumed. Tests cover a corrupt file, a missing directory, and a lock held by another process.

Phase 4 — The decision, without waiting. Project snapshots forward, compute the floor, and log when a request would have waited. Shipping this first gives real data on how often the floor triggers and how wrong it is, before any user sees a pause.

Phase 5 — The wait and the prompt. The blocking state, the timer line, the Ctrl-C path, the non-TTY behaviour, and the silent bounded variant for the collect path. Depends on 4, and is the phase with a user-visible behaviour change.

Phase 6 — A second provider. Anthropic, whose documented headers exercise the parts of the shape Cerebras does not: an explicit reset instant rather than a derived rate, a ceiling and a reading that need not travel together, rounded remainings, and a bucket whose scope can change between responses. If the shape survives that it is worth treating as settled.

References

• Issue 1069 — the report this addresses
• RFD 020 — parallel conversations, the source of the contention
• RFD 045 — the interrupt handler stack the wait plugs into
• RFD 090 — credential profiles and the boundary this design keys against
• acquire_lock — the wait-then-prompt pattern this mirrors
• Cerebras rate limits — dual-bucket model and token-bucket replenishment
• Anthropic rate limits — per-response headers, including the reset instant
• Rate Limits API — Anthropic's admin endpoint for configured ceilings