Evals

A dry run answers one question once: what would this turn do? An eval asks it across a set of saved turns, twice over — once on your current configuration, once on a candidate — and does the arithmetic. It is the answer to “is this new model actually better for my agent”, which a benchmark leaderboard cannot tell you: what is being evaluated is the model inside your harness, with your persona, your skills, and your tools.

The loop

1. Build a test set from real turns — saved from Chat, suggested from history, or imported as JSONL.
2. Run the set against your current config and a candidate. Samples execute on the agent’s live engine under the dry-run execution policy: reads happen, writes are suppressed, nothing is persisted.
3. Read the objective scorecard — rejected and failed tool-call rates, mean rounds, cost, latency, per-task deltas. This half can reject a candidate on its own.
4. Judge the blinded A/B pairs from Claude Code with the /judge-eval skill. This half is what can promote one.
5. Read the verdict with its gate table, and apply the model to the agent if it is an upgrade.

Test sets

A test set is a named collection of test cases. A case is a prompt, a category, optional pinned history, and optional notes.

Pinned history is captured at save time rather than re-read from the source conversation at run time, because the source drifts: clearing a session empties it, retention prunes it, and its latest window is not the window that preceded the saved message. A test case that silently re-scopes itself between runs is not a test case.

There are three fill paths:

• Save as test case in the Chat page’s message menu, optionally pinning the preceding turns.
• Suggest from history on the Evals page (GET /api/v1/eval/suggest) mines past turns for ones worth saving: any rejected or failed tool call, three or more tool rounds, a reply cost in the pool’s top decile, or a command-triggered skill. Candidates come back stratified across the four categories rather than ranked overall, because a set drawn purely by interestingness would be all failures and would represent nothing the agent normally does. Turns already saved as a task are skipped. Nothing is written — accepting a candidate is a separate call.
• Import JSONL — POST /api/v1/eval/task-sets/{name}/import, one case per line, all-or-none so a typo halfway down leaves the set untouched. GET .../export is the other half, so a curated set can be hand-edited or committed to git.

A test set is an appreciating asset: the next candidate model starts here rather than at a blank page.

Runs

A run names a task set, a base agent, and two or more variants. A variant is an overlay on the base agent’s configuration — llm_model, llm_provider, or neither:

{
  "task_set": "regression",
  "base_agent": "pamela",
  "variants": [
    { "name": "incumbent" },
    { "name": "candidate", "llm_model": "moonshotai/kimi-k3" }
  ],
  "k": 3,
  "sample_tasks": 10,
  "cost_cap": 2.0
}

An empty variant runs the agent’s live config. By convention the incumbent is listed first: per-task deltas and the pairing are baselined against the first variant by creation order.

• k is how many samples each (task, variant) pair runs, so the metrics have something to average over. Defaults to [eval] default_k.
• sample_tasks runs a stratified random subset of the set instead of all of it — the cheap first signal. The server draws it, and the drawn ids are pinned on the run, so a task added to the set afterwards cannot retroactively change what the run was measuring. 0, or a value at or above the set size, runs everything.
• cost_cap defaults to [eval] max_cost_per_run.

POST /api/v1/eval/estimate prices a prospective run before it exists, with the same request shape. The per-(task, variant) basis is, in order: history (the task’s source conversation has real telemetry, giving an honest per-exchange average, scaled by the list-price ratio when the variant runs a different model), list price (the advertised per-million-token price against a nominal per-turn budget), or unknown. Nothing is fabricated — a variant that can be priced neither way reports basis: "unknown" with a zero range, and the caller shows the hard cap alone.

Bounds and failure

A run is bounded twice, by spend and by rate, and both are always in force — there is deliberately no uncapped value.

• The cost cap is checked before each dispatch, so a sample already in flight always finishes; the run then ends capped and keeps its partial results. Never a silent truncation.
• [eval] max_concurrent bounds how many samples run at once, process-wide across all runs.
• POST /api/v1/panic cancels active runs along with everything else, and resume deliberately does not revive them: a panic is not a pause.
• A sample that fails takes only itself down and records its error. A run-level failed means a store error, never a provider hiccup.
• Below [eval] completeness_floor the scorecard still reports its numbers but is flagged inconclusive rather than reading a verdict off thin data.

Isolation

Samples run on the agent’s live engine — real persona, real skills, real tools — because a capability-reduced engine would measure a different system than the one in production, and the unit of evaluation is the model in its harness. Safety comes from the execution policy instead: the eval path never reaches the persistence layer, so conversations, telemetry, memory, nudges and the post-turn reviewer are untouched structurally rather than by filtering. Variant overlays clone the router rather than mutating it, so a live turn in flight can never be retargeted.

Eval events are audited like live turns, attributed to a pseudo-agent {agent}#eval:{variant} with source = "eval", and hidden from the dashboard behind the Audit Log’s Previews toggle. See Dry Runs & Previews for the full policy and the audit-marking rules.

Judging

The objective half can reject a candidate but cannot promote one — “cheap and quiet” is not the same as “better”. That call needs a judge.

When a run reaches a terminal state it pairs itself: for each (task, k) it pairs the baseline sample against each candidate’s, assigns the pair a random A/B identity that never leaves the server, and queues two judgment items with the presentation order swapped. Pairing runs for capped and stopped runs too, so a terminal run arrives at the judge queue already populated. A (task, k) whose sample is missing or failed on either side yields no pair, which is why completeness counts pairs separately from samples.

Blinding is total and the payload is built rather than filtered — withheld: variant name, model, provider, serving upstream, cost, latency, token usage, tool-call duration (a consistently slower side is an identity hint), and the sample’s conversation id. Filtering fails open the day a column is added; construction fails closed.

The judge is Claude Code over Denkeeper’s MCP server, working the queue with eval_pending → eval_get_pair → eval_verdict, then reading eval_summary. Give it a key scoped to eval:read,eval:write and nothing more: that key can judge a run but cannot start one, which is the right shape given runs spend real tokens.

denkeeper keys create judge --scopes eval:read,eval:write
claude mcp add --transport http denkeeper https://your-denkeeper/api/v1/mcp \
  --header "Authorization: Bearer $DENKEEPER_JUDGE_KEY"

The rubric lives in the repo at .claude/skills/judge-eval/SKILL.md, where it can be read and edited: four dimensions in priority order — task_success, tool_path, persona_fit, length — with instructions to cite the specific persona or skill clause behind any deduction. Unknown dimension names are rejected rather than stored, because a typo that silently vanishes from the results is worse than a failed call. A verdict records the rubric version the judge worked to, and the summary reports the distinct set it saw, so a queue worked across a rubric edit says so instead of averaging two rubrics into one number.

A pair counts only when both presentation orders carry a verdict. If the two calls name different sides, the judge tracked position rather than quality and the pair records as a tie — so position bias splits the vote instead of deciding it. A half-judged pair is not evidence and stays out of the tally.

The verdict

GET /api/v1/eval/runs/{id}/summary returns the verdict with its work. The rule is conjunctive and asymmetric, evaluated in this order:

The three gates are gate_rejected_rate_pp (rise in the rejected tool-call rate, in percentage points), gate_rounds_pct (rise in mean tool-call rounds) and gate_cost_pct (rise in mean cost per task). They can never promote a candidate, and a judge preference can never override one of them.

Every verdict carries the gate table (value, delta, threshold, unit, pass/fail), a one-line reason, the win-rate against its threshold, and a per-category breakdown, so a candidate that wins overall while regressing on tool-heavy tasks says so out loud rather than hiding inside an average:

downgrade: mean rounds regressed +35.0% against a +20.0% threshold
upgrade: judge win-rate 62% over 45 judged pair(s) meets the 55% threshold, and no objective gate regressed
  ↳ wins overall; regresses on tool_heavy

GET /api/v1/eval/runs/{id}/pairs is the operator’s unblinded view of the same evidence: which variant produced each side, every verdict with its dimensions and notes, and each pair’s resolved outcome. It is REST-only on purpose — the judge’s MCP surface must not be able to look up which variant produced which response.

In the dashboard

The Evals page drives the loop, and everything it does is REST underneath — see the REST API reference for the request and response shapes.

With no test set saved yet, the page opens on an empty state that explains the loop and offers the two ways in: Suggest from history, or an inline JSONL import.

Filling a set from history

Suggest from history opens a panel of past turns worth keeping, each card carrying the prompt, its category, and the reason it was offered — a tool call was rejected or failed, the turn took three or more rounds, it cost in the top ten per cent, or a skill command triggered it. Select the ones you want, choose an existing set or name a new one, and add them in a batch.

Accepting a candidate writes the same shape the Chat page’s “Save as test case” does, pinned history included, captured at that moment rather than re-read at run time. Rejecting a candidate writes nothing at all: it is hidden for the session, so reloading the page offers it again.

Launching a run

The launcher covers the common case — “compare current against a candidate on a test set”. Pick the agent, and its current model is shown beside it; pick a candidate model, with its pricing; pick a test set; then choose one of two presets:

The cost cap is editable and sits next to a live estimate that updates as you change the shape of the run, so it is never a mystery whether you are about to spend ten cents or two dollars. When a run cannot be priced, the estimate says so rather than guessing, and the cap is shown alone.

Watching a run

Each run gets a card with a status chip, a progress bar of turns done against expected, spend against the cap, an ETA while it is active, and a Stop button behind a confirmation dialogue that spells out what stopping keeps.

Progress is polled, and the eval_progress WebSocket frame nudges the page to refresh sooner. The frame is a droppable convenience: GET /api/v1/eval/runs/{id} is the authoritative view, and if progress readings stop arriving the card says it is showing the last reading rather than freezing on a stale number. Leaving the page loses nothing — a run continues in the background, and stopping one keeps the work already finished.

Reading the results

Expanding a finished run’s Results panel lays the evidence out in the three layers the decision rule uses.

The verdict, with its work. One block per candidate: the label, the one-line reason, the divergence note when a candidate wins overall while regressing on a category, the gate table with each gate’s value, delta, threshold, and pass mark, the judgment tally against the win threshold, the rubric versions the counted verdicts were made under, and the per-category breakdown. Never a bare label.

When the objective half is in but the pairs are not all judged, the block says how many are outstanding, states plainly that the verdict rests on the objective checks alone until they are, and gives the next step: the /judge-eval invocation, or a copyable claude -p command, with a reminder to use a key scoped to eval:read and eval:write and nothing else. A run cannot silently dead-end.

An upgrade offers Apply to agent, which switches the base agent’s model through PATCH /api/v1/agents/{name} behind a confirmation naming the agent and both models. A quick check that cleared its gates offers to set up the full eval instead, pre-filled with the same candidate and test set.

The scorecard. Each variant is a column, the objective metrics are rows, and a completeness line underneath reports turns finished, comparisons judged, and whether that clears the floor.

Test case by test case. Every test case is a row with each variant’s cost, rounds, and latency; expanding one puts the responses side by side with their tool traces and the judge’s verdict for that pair, its per-dimension winners, notes, and rubric version.

Configuration

Every [eval] key has a default, so the subsystem needs no configuration to work — see the [eval] reference for the table. GET /api/v1/eval/config returns the same values at runtime, which is how the dashboard shows the policy the server actually judges against instead of hardcoding a copy. Thresholds are TOML plus a reload; there is no runtime writer, and every verdict carries the thresholds it was measured against.

Full request and response shapes are in the REST API reference.