Make provider cassette tests assert semantic outcomes and consumed interactions
- Status: Todo
- Kind: Chore
- Authors: jp
- Date: 2026-08-21
- Label: domain=llm
- Label: package=jp_llm
- Label: type=task
The provider VCR tests snapshot whatever happened but usually have no semantic assertion. Vcr::cassette also discards the MockSet returned by playback_async, so recorded interactions are not checked for exactly one use. Optional tool follow-ups can be silently skipped and leave stale cassette entries without failing.
Current green examples include empty llama.cpp model details, malformed Ollama structured output, invalid tool arguments, and empty Google post-tool responses.
Acceptance criteria:
- Assert every cassette interaction is consumed exactly once during playback.
- Fail on unexpected requests and unused recorded responses.
- Require request-kind invariants: forced tool name, tool argument conformance, structured schema conformance, nonempty or explicitly empty post-tool outcome, requested model ID, and nonempty model lists.
- Remove no-op assertion defaults where the request kind has a meaningful contract.
- Make skipped
ToolCallResponserequests explicit in the test declaration rather than controlled by a boolean that silently returnsNone. - Add a harness regression test with an intentionally unused second cassette interaction.