# Add a vLLM provider

- **Status**: Done
- **Kind**: Feature
- **Authors**: jp
- **Date**: 2026-09-16
- **Label**: domain=llm
- **Label**: package=jp_config
- **Label**: package=jp_llm
- **Label**: type=feature

vLLM serves `GET /v1/models` and `POST /v1/chat/completions` behind a Bearer
token, speaking the same Chat Completions dialect that
`jp_llm::provider::openai_compat` already parses for llama.cpp.
A vLLM provider is therefore mostly config plumbing plus a thin provider module
over the shared dialect code.

## Config

- `ProviderId::Vllm` in `jp_config::model::id`, with `as_str() == "vllm"`.
- `providers/llm/vllm.rs` holding `VllmConfig`: `api_key_env` defaulting to
  `VLLM_API_KEY`, and `base_url` defaulting to `http://127.0.0.1:8000`.
  The four trait impls (`AssignKeyValue`, `PartialConfigDelta`, `FillDefaults`,
  `ToPartial`) follow `deepseek.rs`.
- A `vllm` field on `LlmProviderConfig`, wired into each of the four impls in
  `providers/llm.rs`.
- The `jp_config` snapshots for config fields, schema shape, and partial
  defaults all move; review them with `cargo insta`.

## Provider

- Hoist `to_system_messages`, `convert_events`, `convert_tools`, and
  `convert_tool_choice` out of `llamacpp.rs` into `openai_compat.rs` as
  `pub(crate)`, and have llamacpp call them there.
  Behavior-preserving.
- `provider/vllm.rs` with `Vllm { client, base_url }`.
  `TryFrom<&VllmConfig>` reads the key from the environment and sets the Bearer
  header, as `cerebras.rs` does.
- `models()` maps `GET /v1/models` entries to `ModelDetails`, taking
  `context_window` from the reported `max_model_len` and keeping the full id
  (e.g. `Qwen/Qwen3-8B`) as the name.
  `model_details()` returns `ModelDetails::empty()` for an unknown name.
- `create_request()` builds the chat body: model, messages, stream, temperature,
  top_p, max_tokens, tools, tool_choice, `response_format` for a structured
  schema, and `chat_template_kwargs.enable_thinking` from the reasoning setting.
  No `reasoning_format` field — that one is llama.cpp-specific.
  `chat_completion_stream()` posts it and parses the SSE stream with
  `parse_chunk()`.
- `mod vllm`, the `get_provider()` and `build_request_value()` arms in
  `provider.rs`, and the `Vllm` arms in `test.rs` for `base_url` and
  `api_key_env`.
- `vllm_tests.rs` covering `create_request()` for a plain message, a tool call
  round trip, a structured schema, and reasoning off, each against a static
  expected JSON body.

## Docs

Name vLLM in the provider sentence in `docs/features/tools.md`.

## Comments

-----

- **From**: jp
- **Date**: 2026-09-16T02:01:11Z

Filed after the fact: the config, provider, and docs work described above is
implemented and staged.
Deviations from the original plan worth recording:

- The plan also called for adding vLLM to a provider list in
  `docs/configuration.md`.
  That file has no provider list, so there is nothing to add there.
- The plan's dotfiles step (the `nebius` alias and the `[providers.llm.vllm]`
  table in `agentic-shepherd/dotfiles/jp-user-config/user-config.toml`) lives in
  another repository and is out of scope here.