Assertions

Assertions are ordinary pytest assert statements. Deterministic ones are free and fast, so put them before any judge(...) call - pytest stops at the first failure, and a failed assertion never spends judge tokens. case.run(...) returns a compact CaseResult: check visible dialogue in .messages, evaluated data in .output, the stop condition in .termination, and what the agent did in .trace. Pass trace=result.trace to judge(...) only when the criterion needs trajectory evidence.

Trace assertions

Successful runs expose the collected trace as result.trace.

Tool calls

result.trace.tools answers questions about which tools were invoked and what JSON evidence each call recorded:

def test_refund(case, kensa_run):
    result = case.run(kensa_run)

    assert result.trace.tools.include(["lookup_customer"])
    assert result.trace.tools.exclude(["issue_refund", "delete_account"])
    assert result.trace.tools.order(["search_flights", "book_flight"])
    assert result.trace.tools.no_repeats()

    calls = result.trace.tools.matching(
        "lookup_customer",
        arguments={"include": {"charges": True}},
        result={"found": True},
    )
    assert calls
    assert calls[0].sequence == 0
    assert calls[0].status == "ok"
    assert result.trace.tools.called(
        "lookup_customer",
        arguments={"customer_id": "cus_123"},
    )

Member	Returns	Checks
`tools.calls`	`tuple[ToolCallEvidence, ...]`	Immutable normalized calls in trace order
`tools.names`	`list[str]`	Tool names in call order
`tools.matching(name, ...)`	`tuple[ToolCallEvidence, ...]`	Every call matching the supplied argument, result, and status filters
`tools.called(name, ...)`	`bool`	Whether at least one call matches the supplied filters
`tools.include(names)`	`bool`	Every listed tool was called
`tools.exclude(names)`	`bool`	None of the listed tools were called
`tools.order(names)`	`bool`	Listed tools appear in this relative order
`tools.no_repeats()`	`bool`	No tool was called more than once

Argument and result filters are recursive JSON object subsets. Every expected key must exist. Nested objects use the same subset rule. Arrays and scalar values require exact JSON type and value equality, so true does not match 1, and array order and length matter. Observed objects may have extra keys. Omit arguments, result, or status to leave that field unfiltered. Supplying an empty object matches a recorded object, but never a missing payload or a recorded scalar or array. Filters must be mappings; inspect tools.calls directly for exact scalar or array assertions. Missing and explicit JSON null are separate facts:

missing_call = result.trace.tools.calls[0]
null_call = result.trace.tools.calls[1]

assert missing_call.arguments is None
assert missing_call.arguments_recorded is False

assert null_call.arguments is None
assert null_call.arguments_recorded is True

Repeated names produce separate calls with separate zero-based sequence values. matching(...) returns all matching calls. Independent assertions do not reserve or assign a call, so the same observed call can satisfy more than one assertion. Kensa reports captured facts only. A successful tool span can return any application payload, including a field named status; Kensa does not infer whether that payload means the business operation succeeded.

Resource bounds

result.trace also reports cost, turns, and latency for budget assertions:

def test_within_budget(case, kensa_run):
    result = case.run(kensa_run)

    cost_usd = result.trace.cost_usd
    assert cost_usd is not None              # every billable span has a known cost
    assert cost_usd < 0.10                    # under 10 cents
    assert result.trace.llm_turns <= 5         # at most five model calls
    assert result.trace.duration_ms < 30_000   # under 30 seconds

Accessor	Type	Meaning
`result.trace.cost_usd`	`float \| None`	Total cost, or `None` when any billable span is unpriced
`result.trace.known_cost_usd`	`float \| None`	Sum of priced spans, even when total cost is incomplete
`result.trace.llm_turns`	`int`	Number of LLM spans
`result.trace.duration_ms`	`float`	Total trace duration
`result.trace.spans`	`list`	Raw collected spans, for custom checks

result.trace.incomplete is True when the trace could not be fully collected. Guard budget assertions on it if your harness can return early.

Output assertions

case.run(kensa_run) always returns CaseResult. Assert on its typed fields with plain Python.

import json
import re


def test_output_shape(case, kensa_run):
    result = case.run(kensa_run)
    text = json.dumps(result.output)

    # Substring (case-insensitive)
    assert "confirmation number" in text.lower()

    # Regex match
    assert re.search(r"\d{6,}", text)

    # Structured output: assert on fields directly
    assert result.output["status"] == "resolved"

    assert result.messages[-1]["role"] == "assistant"
    assert result.termination.reason == "direct"

    # Safety check
    assert "refund" not in text.lower()

Assertions gate the judge

Order matters. Deterministic assertions first, semantic judge last:

def test_refund_policy(case, kensa_run):
    result = case.run(kensa_run)

    # 1. Free, deterministic - fails fast.
    assert result.trace.tools.include(["lookup_customer"])
    assert result.trace.tools.exclude(["issue_refund"])

    # 2. Only reached if the cheap checks passed.
    verdict = judge(
        result,
        "The response must not promise an unsupported refund.",
        input=case.input,
        trace=result.trace,
    )
    assert verdict.passed, verdict.reasoning

If a tool assertion fails, the test fails immediately and the LLM judge never runs. See Judge for the semantic half.

Getting started

Reference

Workflows

Releases

Trace assertions

Tool calls

Resource bounds

Output assertions

Assertions gate the judge

​Trace assertions

​Tool calls

​Resource bounds

​Output assertions

​Assertions gate the judge

Trace assertions

Tool calls

Resource bounds

Output assertions

Assertions gate the judge