Kensa keeps the regression contract inside pytest. An eval is an ordinary test file: you define cases, run each one through your agent, assert on the resulting trace, and use a judge only for semantic checks.
Directory layout
Evals live in your test tree. Kensa’s generated evidence lives under .kensa/.
You write evals and the kensa_run harness. .kensa/ holds ignored runtime artifacts;
kensa doctor computes readiness live.
Cases
A case is a single input to your agent, built with kensa_case:
Cases take either a literal input or a messages conversation. They are parametrized into a test with @pytest.mark.parametrize("case", [...]). See Cases for the full field reference.
The harness
Kensa never guesses how to call your agent. Setup traces repository control flow to the production
function or class that starts one conversation, cites its source location, and waits for approval
before writing a tests/evals/conftest.py::kensa_run(case) adapter. Otherwise it reports
cannot wire.
Your repository keeps ownership of construction, prompts, tools, routing, state, dependencies,
effects, and cleanup. Kensa owns execution only after fixture resolution:
The fixture creates one production conversation per trial. case.run(kensa_run) reuses it across
turns and returns a CaseResult with canonical messages, output, termination, and a read-only trace.
kensa doctor rejects adapters that do not call the approved production code.
Traces
While a case runs, Kensa collects OpenTelemetry spans into a trace exposed on successful runs as
result.trace. Tool calls and model calls are captured automatically when you wrap them with the
recording helpers (record_tool_call, record_llm_call) or run instrumented SDK code. Imported
evidence is narrower than raw telemetry: Kensa keeps only its allowlisted TraceView fields and
redacts retained values before storage.
result.trace exposes:
Assertions
Assertions answer binary questions about the run. Deterministic assertions are free and fast - plain assert and result.trace.* (tool calls, cost, turns, latency). Because pytest stops at the first failed assertion, ordering them before the judge means obvious regressions never reach an LLM call.
A judge is the semantic escape hatch: judge(result, criteria, ...) sends ordered
messages, normalized output, and termination as structured evidence. Use it only for criteria that
resist deterministic checks. Trace evidence remains explicit: pass trace=result.trace when the
criterion needs it. See Assertions and Judge.
Trials
Agents are non-deterministic, so an eval can run a case more than once:
Each trial is one pytest item with its own trace. Kensa aggregates the trials per case into a single verdict at session end.
fail, flaky, and error fail the pytest session. trials: 1 is a smoke check; trials > 1 is measured evidence.
Eval ideas
When you have trace evidence, your coding agent (the kensa-inspect skill) mines it into reviewable eval ideas - proposed evals recorded as a YAML review queue under .kensa/inspect/. You approve the ones worth keeping by changing status: pending to status: approved, validate the queue with kensa inspect lint, then the kensa-generate skill materializes them as tests/evals/test_<id>.py. Materialized evals are plain pytest files you own and edit.
For integrations, BehaviorCandidate.from_inspect_idea(...) publishes an idea as the strict
kensa.behavior_candidate.v1 contract. Its semantic_fingerprint hashes only the normalized
failure pattern and expected outcome, under the separately versioned
kensa.behavior_fingerprint.v1 algorithm. Equivalent behaviors must therefore use the same
canonical wording; the fingerprint does not infer equivalence between differently worded prose.
Pipeline
Every stage is optional on its own: you can hand-write evals and skip generation, or run plain pytest tests/evals/ and skip the kensa CLI entirely. Last modified on August 17, 2026
