RFD 133: Decision Models
Summary
A decision model answers typed questions about an input with probabilities instead of generating text. This RFD adds a providers.decision namespace for servers implementing the System One API, question and answer types in jp_conversation, and a decide protocol request for command plugins.
Motivation
A decision model takes a state (text or a JSON value) and a set of named questions, and returns one typed answer per question: the probability that a statement is true, a selection from a fixed set of options, or a position on an ordered rubric. TypeSafe's Jev introduced this model class and its POST /v1/systemone endpoint, which Ollama, AWS's Strands Decider, and open models such as Kev also serve. Responses return in 70 to 500 ms at a fraction of an LLM's input cost.
JP plugins frequently make these classifications: which crate owns an issue, whether a ticket needs action, or whether a staged diff requires a changelog entry. With only LLM providers available, each classification requires a query request with a schema (RFD 072), which runs a full turn with seconds of latency and returns a structured value without a probability. Because no JP provider speaks System One, a plugin that uses a decision model must manage its own credentials and HTTP client.
Design
Terms
- Decision model: A model that answers questions about a state with typed, probabilistic answers.
- Decision: A single request to a decision model, consisting of a state, named questions, and the returned answers.
- State: The input evaluated by every question in a decision.
A decision question is distinct from an inquiry, which is a question a tool asks during a turn and the user or assistant answers (RFD 028).
Questions and answers
Question types follow the System One API:
| Type | Question | Answer |
|---|---|---|
noul | Is this statement true? | Probability of true (0 to 1) |
choice | Which of these options applies? (2 to 255) | Selected option, probability per option, and confidence |
score | Where does this sit on a rubric? (2 to 10) | Expected level, probability per level, and confidence |
Questions and answers are defined as JP types in jp_conversation::event::decision. Each decision source converts its server's wire format into these types so answers have a uniform representation regardless of which server produced them, matching how jp_llm providers convert wire formats into ChatResponse and ToolCallRequest. Their JSON serialization matches the System One schema for the portable subset below, so plugin authors can reference the System One documentation directly.
Where System One servers differ in request validation, JP's types enforce the portable subset accepted by every server:
- State and instructions: Must be a non-blank string, a JSON object, or a JSON array (empty objects and arrays are allowed). Every question requires
instructions(optional in TypeSafe, required by Ollama). - Criterion descriptions: Must be strings for
choice,score, andnoul(choiceoptions also allownullto omit a description). TypeSafe also accepts structured object descriptions, which JP's types reject. - Option counts: 2 to 255 options for
choiceand 2 to 10 levels forscore(TypeSafe accepts a single option).
Server- or model-specific limits (such as maximum questions per request or allowed characters in question names on Ollama Cloud) are not enforced by JP; the server rejects violations and the source returns a provider error.
Configuration
[providers.decision.typesafe]
source = "systemone"
base_url = "https://api.typesafe.ai"
api_key_env = "TYPESAFE_API_KEY"
[providers.decision.local]
source = "systemone"
base_url = "http://localhost:11434"
[decision.model]
id = "typesafe/jev-latest"providers.decisionis a map of user-named provider entries, each specifying asourceand that source's settings. Unlikeproviders.llm, which has one fixed struct field per implementation, a map lets a single source back multiple servers (such as a hosted endpoint and a local server).systemoneis the only supportedsource. It acceptsbase_urland an optionalapi_key_env; omittingapi_key_envsends unauthenticated requests for local servers.sourceis required in the finalized configuration, but an override layer patching an existing provider entry can omit it. Becausesystemoneis the only source, all entries share a single settings struct and partial layers merge without repeatingsource.decision.modelis optional, andproviders.decisionis empty by default. Configurations that do not use decision models require neither section.decision.model.idhas the form<provider>/<model>. Referencing a provider not declared underproviders.decisionfails config resolution with an error listing the declared providers.decisionis a top-level configuration section alongsideassistant, participating in normal config layering and--cfgoverrides.
Decision and LLM providers use separate namespaces and distinct model ID types. Each model ID resolves its provider within its own namespace. If decision.model.id names a model that the configured server cannot serve (such as a chat model), the server rejects it when a request runs.
Command plugins
A plugin sends decide, and the host replies with decided, or with error if the request fails:
{"type": "decide", "id": "7",
"state": {"ticket": "…"},
"questions": {"area": {"type": "choice",
"instructions": "Which crate owns this ticket?",
"criteria": {"cli": null, "llm": null}}},
"cfg": ["decision.model.id=local/kev-4b"]}{"type": "decided", "id": "7", "model": "kev-4b",
"answers": {"area": {"type": "choice", "choice": "cli", "confidence": 0.81,
"probabilities": {"cli": 0.9, "llm": 0.1}}}}questionsandanswersuse thejp_conversation::event::decisiontypes.- Each
deciderequest snapshots the plugin's current configuration when accepted (the startup configuration, or the latest configuration from a successfulread_configreload; failed reloads leave it unchanged). Anycfgoverrides on the request layer over that snapshot alone, matching--cfgon the CLI, so subsequent reloads or concurrent requests do not affect in-flight decisions. - If
decision.model.idremains unset after applyingcfg, the host replies with anerrornamingdecision.model.id. - A
deciderequest does not read or modify any conversation, and the host resolves provider credentials just as it does forquery. - The host executes
deciderequests concurrently and replies as each completes rather than in arrival order, so a plugin classifying a batch of items can keep multiple requests in flight. - Because replies can arrive out of order,
deciderequires anidfield, matchingqueryandinterruptin RFD 072. The host echoesidondecidedanderror; adeciderequest without anidis rejected immediately with anerrorbefore reaching the provider. - In-flight
deciderequests are cancelled without a reply when the plugin exits, closes stdout, or receives a host shutdown. Unlike delegatedqueryturns, which the host awaits after plugin exit because they persist conversation events, adeciderequest writes no state and has no reader once the plugin terminates. - Adding
decideanddecidedincrementsPROTOCOL_VERSION(from 13). A plugin that sendsdecidemust declare the new protocol version inready.
TIP
RFD 135 extends decide with an optional conversation field to record the decision into a conversation stream. Because recording mutates conversation state, the host awaits those requests after plugin exit (matching delegated query turns) instead of cancelling them.
Crates
plugin `decide` ──> decide runner (jp_cli) ──> jp_decision::Provider ──> systemone sourcejp_decision: New crate defining theProvidertrait (decideandmodels) and thesystemonesource. Keeping the System One wire format inside thesystemonemodule isolates changes to the external API.jp_conversation: Defines the decision question and answer types.jp_config: Addsproviders.decision,decision.model, and a decision model ID keyed by user-declared provider name (unlikeModelIdConfig, which is keyed by the closedProviderIdenum).jp_cli: Adds the plugin protocol messages and a runner that resolves configuration, builds the request, and invokes the provider.
jp_llm is unchanged: decision models have no messages, tools, streaming, or reasoning, so they do not fit jp_llm::Provider.
Drawbacks
- Additional provider namespace and model ID type. Introducing
providers.decisionadds a third provider namespace alongsidellmandmcpwith a map structure that differs fromproviders.llm, plus a second model ID type alongsideModelIdConfig. - Coupling to an early specification. The System One API is two months old. Wire-format changes stay isolated inside the
systemonesource, but JP's question and answer types mirror the System One data model.
Alternatives
Add a systemone source under providers.llm. Rejected. Sharing a single namespace would force chat and decision models to share one model ID type and provider lookup, requiring every caller of jp_llm::Provider to distinguish the two at runtime.
Run decisions through an LLM with structured output. Rejected as the primary mechanism. Although this works with every existing LLM provider, each decision incurs full turn latency, and without logprobs any returned probabilities are uncalibrated estimates. An LLM-backed source remains a possible future addition (see Non-Goals), provided its answers indicate that their probabilities are uncalibrated.
Have plugins call decision servers directly. Rejected. Avoiding a protocol addition would force each plugin to bundle its own HTTP client and manage credentials that JP already holds.
Add a model field to the decide plugin request. Rejected. While slightly shorter for plugins that switch models per request, a dedicated field would bypass the configuration pipeline that every other request uses, and cfg already supports per-request model overrides.
Non-Goals
- A CLI subcommand or conversation persistence for decisions. Both are deferred to a separate RFD.
- A
decidetool for the assistant. boxquestions and image states. TypeSafe gates both behind opt-in headers and sends images in anassetsfield, whereas Ollama sends images in a separateimagesfield, and JP has no rendering surface forboxanswers. Thesystemonesource therefore sends text and JSON states only.- Other decision sources, such as OpenAI's Decisions API (which has no published schema) or an LLM-backed fallback source.
- Using decision models inside JP's own internal flows, such as answering tool inquiries (RFD 028) or gating tool calls. Those workflows can build on
jp_decision::Providerin separate RFDs.
Risks and Open Questions
Prompt injection against decision models. A University of Southern California study found that short, natural-looking injected text flipped 61% of correct Jev decisions. Any decision over untrusted input (such as an issue body, web page, or tool output) that gates an action is vulnerable to steering, and the decide protocol documentation notes this risk.
Divergence across System One servers. Implementations of System One differ in request validation, option limits, and confidence calculation. For example, Ollama requires instructions and a non-blank string state, caps cloud requests at 64 questions, and echoes the requested model name where TypeSafe returns the resolved model version. Context limits also vary (Jev allows 64k tokens per request and 32k per question, Ollama Cloud allows 65,536 tokens, and local Ollama models cap input at 64 KiB of text). JP's types enforce the requirements shared by all servers, but JP does not track server-specific context or question limits; oversized requests are rejected by the server and returned as provider errors. JP forwards the server's answers as returned and makes no guarantee that probabilities or confidence scores are comparable across models.
Implementation Plan
- Configuration. Add
providers.decisionwithsystemonesettings,decision.model, the decision model ID type, validation that the referenced provider is declared, and updated schema snapshots. This phase is independently mergeable. - Types and
jp_decision. Add the question and answer types injp_conversation, thejp_decision::Providertrait, and thesystemonesource with recorded request/response fixture tests matchingjp_llm. This phase depends on Phase 1 and is independently mergeable. - Plugin protocol. Add the
deciderunner,decideanddecidedmessages, concurrent execution, requiredidvalidation, and the bumped protocol version, and listdecidealongsidequeryandinterruptin RFD 072's Request IDs section. This phase depends on Phase 2. - Documentation. Add a configuration guide section and glossary entries for Decision and Decision Model (distinguishing decisions from inquiries). This phase depends on Phase 3.
References
- System One API reference
- Introducing System One Models & Jev, TypeSafe
- Ollama's System One API reference
- RFD 028: Structured Inquiry System for Tool Questions
- RFD 072: Command Plugin System