RFD 125: Configured Model Details per Provider

DiscussionDesign2026-10-27Jean Mertz

Summary

Each LLM provider block gains a models table, keyed by the served model name, that configures a model's context window, output limit, reasoning support, structured output, and prefill support. Configured values override both the provider's model listing and JP's built-in tables on every request to that model.

Motivation

JP shapes each request around a ModelDetails: it clamps a reasoning effort to the levels a model supports, converts a token-count effort to a level using the output limit, decides whether to prefill, and truncates inquiry and title input to the context window. Those details come from two places: the provider's model listing, and tables built into JP for providers whose listing omits those fields.

For a self-hosted server, neither source is enough. vLLM's /v1/models reports a model's id and its served context window (max_model_len), and nothing else. No vLLM endpoint reports the reasoning parser, the output limit, or whether structured output works; the closest, /tokenizer_info, is off by default and vLLM's security guide advises against enabling it. The server swaps models without JP being rebuilt, so a built-in table cannot keep up either. Every model on such a server is unknown to JP, so JP falls back to permissive defaults for any missing field. For reasoning, that means efforts pass through unclamped, and vLLM documents that some chat templates silently fall back to maximum reasoning for an effort they do not recognize.

Cloud providers have the same gap on a smaller scale. A model released after the JP binary gets empty details (Cerebras logs Unknown Cerebras model, using empty details.) until a JP release adds it to the table.

Without configuration support, supplying missing model details requires a code change, even when the user who deployed the model or read its model card already knows its capabilities.

Design

Configuration

Every provider block takes a models table. The key is the model name exactly as the provider serves it, the same name that follows the / in a model id; names with a . or / are quoted, as TOML requires.

[providers.llm.vllm.models."deepseek-v4.1-flash"]
max_output_tokens = 65536
structured_output = true
reasoning = { mode = "leveled", levels = ["low", "high"] }

[providers.llm.vllm.models."Qwen/Qwen3.8-27B-FP8"]
reasoning = { mode = "budgetted", max_tokens = 32768 }

[providers.llm.cerebras.models."zai-glm-5"]
context_window = 131072

An entry has these fields, each optional:

Field	Meaning
context_window	Maximum combined prompt and output tokens.
max_output_tokens	Maximum tokens generated per response.
structured_output	Whether the model honours a JSON schema response format.
prefill	Whether the model continues a trailing assistant message.
reasoning	Reasoning configuration; see below.

reasoning is "unsupported", or a table naming a mode:

• budgetted: reasoning measured in tokens, with optional min_tokens and max_tokens.
• leveled: discrete efforts, listed in levels from xlow, low, medium, high, xhigh, max.
• adaptive: the model schedules its own reasoning; levels may add xhigh and max to the base low, medium, high.

A reasoning table also takes can_disable, default true; set it to false for a model that rejects a request to turn reasoning off.

The fields mirror ModelDetails in jp_llm. subscription, features, display_name, knowledge_cutoff, and deprecated are omitted: the first two are internal to the provider implementation, and the last three do not affect request construction. features in particular holds each provider's private tags (sampling-parameters, between-tools-thinking), so exposing it would make every rename of a tag a config migration. If users later need to configure one of those properties, it can be exposed as a dedicated field.

Every provider block also takes fetch_model_details, default true. When fetch_model_details is false, JP skips fetching model details from the provider before each turn, inquiry, and title query, which saves a round trip on slow connections or on servers without a model-details endpoint:

[providers.llm.vllm]
fetch_model_details = false

If a request is rejected for an unknown model, JP still fetches the model list for the interactive model picker.

Behaviour

Each field on ModelDetails is resolved from the first source that sets it:

1. the model's models config entry;
2. the provider's API response for the model;
3. JP's built-in table for the provider;
4. the provider's fallback default for an unknown model.

An entry for a vLLM model can declare reasoning levels while keeping the max_model_len reported by /v1/models, and a value reported by the provider overrides a stale built-in table entry.

An entry applies to exactly one model: the one whose served name equals its key. There are no patterns, and an entry for a model the provider does not list adds nothing to the listing. It still applies when a conversation requests that model, since a request can name a model the listing omits.

Fetching model details from the provider never fails a query. If the lookup fails or times out after three seconds, JP logs the failure and falls back to the built-in table and defaults for any fields not set in config. This allows a models entry to supply accurate limits for a gateway without a model-listing endpoint, such as Synthetic's Anthropic-compatible endpoint. Pre-request checks outside the HTTP lookup still fail the query, such as a missing credential or OpenAI's check that a ChatGPT subscription supports the requested model.

Entries merge across config layers like providers.llm.aliases: by key, and within an entry by field. A user-global config can set a model's context_window while a workspace config sets its reasoning.

On vLLM, declaring reasoning in models does not enable reasoning by default: when assistant.model.parameters.reasoning is unset, JP omits reasoning fields from the request so the model's chat template and the server's --default-chat-template-kwargs remain in control. The entry is consulted once reasoning is configured: to clamp an effort to the ladder, bound a token budget, or decide how off is sent.

Other providers treat a configured capability the way they treat one from their built-in table. A model configured with reasoning receives that provider's default effort when reasoning is unset, matching how Ollama and Cerebras treat models in their built-in tables.

Where it applies

The overlay is a pure function in jp_llm: given a ModelDetails and a configured entry, return the details with the entry's fields applied.

get_provider already builds every provider from its config block. It wraps the provider it builds in a decorator holding that block's models table and fetch_model_details, and the decorator applies the overlay to whatever model_details and models return. When fetch_model_details is false, the decorator calls a new Provider method instead of model_details that returns details from the built-in table without making a network request; its default implementation returns empty details, so only providers with a built-in table override it. Every caller (the query turn, inquiries, title generation, summarization, the unknown-model picker) sees the same details.

Individual provider implementations do not read models config directly. Each request builder consumes the overlaid ModelDetails fields it already reads, and this RFD does not require every builder to honour every field. Phase 2 updates the vLLM request builder to honour min_tokens/max_tokens and can_disable, which the motivating case depends on.

Generic code reaches the tables through one accessor on LlmProviderConfig, keyed by ProviderId. If JP later supports multiple instances of the same provider (such as two vLLM servers), the models table stays inside its provider block, and only that accessor's lookup key changes.

Limitation: --cfg KEY=VALUE

A dotted --cfg key is split on every ., so it cannot address an entry whose name contains a .: providers.llm.vllm.models.deepseek-v4.1-flash.max_output_tokens=65536 splits the model name in two. A config file (-c models.toml) and the JSON form both work:

jp q --cfg '{"providers":{"llm":{"vllm":{"models":{
  "deepseek-v4.1-flash":{"max_output_tokens":65536}}}}}}'

Drawbacks

A wrong entry breaks requests that would have worked. Declaring prefill = true for a model that rejects prefill turns a few extra tokens into a request error, and the error does not point at the config.

Config wins over JP's own tables, so an entry written to work around an outdated table keeps overriding it after a release corrects the table.

The field names become part of the config surface, and with it of the JSON Schema JP exports. ModelDetails can still change, but each rename of a mirrored field is a config migration.

Alternatives

A single map at providers.llm.models, keyed by provider/name. Places all overrides in one field next to aliases instead of one per provider block. Rejected because the provider/name key format differs from every other key under providers.llm, and the map would need re-keying if multiple instances of the same provider are supported; a table inside the provider block moves with it.

A field on the vLLM block only. Self-hosted servers need it most, but cloud tables lag too, and the field costs one delegating line per provider block.

Infer details from the chat template. /tokenizer_info returns the template when the server enables it. Rejected: it is off by default, vLLM advises against enabling it, and reading capabilities out of a Jinja template is guesswork.

A separate models file, as pi's models.json. JP already layers config files and loads extra ones with -c; a second format is a second loader with no new capability.

Non-Goals

• Adding models to a provider's listing. The listing reflects what the server serves; the unknown-model picker relies on that.
• Extra request body fields, such as a chat template kwarg JP does not send. That is request shaping rather than a fact about the model, and it is applied by each provider rather than in one place. A vLLM operator can set such kwargs server-wide with --default-chat-template-kwargs, but a backend behind a router may have no equivalent; per-model request fields would need their own design.
• Configuring one provider more than once. This RFD keeps the table inside the block so that work does not have to move it.
• Pricing and cost tracking.

Risks and Open Questions

The reasoning shape is the part most likely to be wrong. Each provider reads ReasoningDetails differently (Anthropic's adaptive thinking, Google's levels, Cerebras's efforts), and the config has to express every variant without exposing the Rust enum.

Two gaps are known. The vLLM request builder sends an absolute effort as thinking_token_budget without bounding it by min_tokens and max_tokens, and turns off into reasoning_effort: "none" without checking can_disable; phase 2 fixes both. Other providers may have gaps of their own for fields their tables never set, and a configured value there behaves as a reported one would.

Whether a configured context_window below the served one should also cap requests JP sends, or only inform truncation and compaction, is open. This RFD only changes what JP believes about the model; anything that reads context_window behaves as it would had the provider reported that value.

Implementation Plan

1. Config type. Add ModelDetailsConfig in jp_config, with its partial, delta, fill-defaults, to-partial, and key assignment implementations, a models: MergeableMap<ModelDetailsConfig> field on each of the nine provider blocks (merged with map_with_strategy), and a fetch_model_details flag on each. Add the accessor on LlmProviderConfig and update the config schema snapshots. Changes no behaviour, so it merges on its own.
2. Overlay. Add the pure overlay function in jp_llm, the decorator applied in get_provider, the table-only Provider method used when fetch_model_details is false, and tests per field, including a vLLM request whose effort is clamped to a configured ladder. Update the vLLM request builder so the motivating case works: it sends no reasoning fields when reasoning is unset, even for a model with a configured ladder; it bounds an absolute effort by the configured min_tokens and max_tokens; and for off on a model with can_disable = false it sends the lowest level of a leveled model, and nothing otherwise. Each has a request-body test. Depends on phase 1.
3. Documentation. Update the provider section of the configuration guide with a vLLM example. Depends on phase 2.

References

• vLLM reasoning outputs, including --reasoning-effort-map and --default-chat-template-kwargs.
• vLLM security guide, on /tokenizer_info.
• pi's model configuration, which separates added models from overrides of existing ones.