Your agent writes to real systems.
You can't test it there.
You already knew that. Your architecture told you.
So test it against what it already did. Kitaru replays your real runs against your next change, and shows you what would have broken.
One wrapper, no rewrite
uv add "kitaru[cli,worker]" kitaru-pydantic-ai14-day free trial · Full access · No credit card
# 1 · wrap the agent you already havefrom kitaru_pydantic_ai import KitaruAgentagent = KitaruAgent(intake_agent, agent_id=AGENT_ID) # 2 · the runs it already made, importedkitaru session import ./traces.jsonl \ --importer kitaru/kitaru-jsonl@latest --agent intake-agent@latest --wait # 3 · get interviewed, in your coding agentwalk me through 20 of these→ cohort "dropped the hazmat flag" · 9→ evaluator hazmat-flag-preserved # 4 · change the agent, compare the runskitaru experiment run start fix-validation \ --cohort-version $COHORT_VERSION_ID --agent intake-agent@pr-311 --waitWhat ops caught by hand is now a check that runs on every commit.
One cohort of real sessions.
A different question each time.
You read a few. The agent groups the rest. One change per experiment, and the answer is two runs side by side.
What does good even mean here?
Read twenty of them and write what you notice. Your notes become the cohorts.
Can I believe my own check?
Apply it blind to sessions you never read, then reveal the labels.
Can we ship the cheaper model?
Same cohort, one model swapped — the answer is two runs, compared.
Will it stay fixed?
The same experiment runs on every commit, on the cases that caught it.
Keep your agent SDK. Make its runs replayable.
Each run records as a session, and that recording is what replay reads back. The wrapper lands in a new entrypoint file beside your code — your agent is never edited.
Installuv add "kitaru-pydantic-ai[openai]"
from pydantic_ai import Agent
agent = Agent(
"openai:gpt-5-mini",
system_prompt="You are a compliance reviewer.",
tools=[search_docs, fetch_policy],
)
result = await agent.run(task)from kitaru_pydantic_ai import KitaruAgent
from pydantic_ai import Agent
agent = KitaruAgent(Agent(
"openai:gpt-5-mini",
system_prompt="You are a compliance reviewer.",
tools=[search_docs, fetch_policy],
), agent_id=AGENT_ID)
result = await agent.run(task)Ship the cheaper model without shipping a regression.
Nobody moves, because nobody can say what it would break. Two runs over your real sessions can.
Freeze the sessions that matter
The sessions you care about, frozen as a named set. Immutable, so a run's result keeps meaning what it meant.
COHORT_VERSION_ID=$(kitaru cohort create checkout-flow --agent checkout-agent --sessions-file session-ids.txt --output json --machine --non-interactive --no-browser | jq -r '.item.version.id')State the two hypotheses
Each experiment is just configuration. Keep the baseline fixed, then change only the model in the candidate.
kitaru experiment create baseline --agent checkout-agent --tool-policy '{"default":{"type":"history","scope":"cohort_version","on_miss":"fail"}}' --evaluator response-quality@latest
kitaru experiment create cheap-model --agent checkout-agent --override '{"model":"claude-haiku-4.5"}' --tool-policy '{"default":{"type":"history","scope":"cohort_version","on_miss":"fail"}}' --evaluator response-quality@latestCompare like with like
Run the baseline and candidate over the same cohort and agent version. One model moved, so the numbers mean what they look like.
kitaru experiment run start baseline --cohort-version "$COHORT_VERSION_ID" --agent checkout-agent@v1 --wait
kitaru experiment run start cheap-model --cohort-version "$COHORT_VERSION_ID" --agent checkout-agent@v1 --waitSame 200 sessions. Same evaluator. One model swapped.
Illustrative numbers. Kitaru grades every session ready, partial or unavailable at import, so you know before you rely on it.
Every correction your ops team makes
is a label nobody is using.
Somebody fixes the agent's booking by hand. That corrected record is ground truth, in your schema, from a domain expert — accumulating daily, and no eval tool has ever asked for it.
- Field by field
- A structured write diffs against production, one field at a time. No judge, no calibration set.
- The residue
- Keep a model-graded check only for tone, and for whether escalating was right.
- Per tenant
- Every customer brings its own corrections. Same loop, new cohort.
One cohort, one changed variable, two runs.
Each experiment holds one configuration; each run pins it to the same cohort and agent version. The workflow is available in Python and TypeScript. Hover any line.
import asynciofrom kitaru.api_models.v1.experiment import ExperimentCreateRequestfrom kitaru.api_models.v1.experiment_run import ExperimentRunCreateRequest, ExperimentRunStatusfrom kitaru.api_models.v1.replay_config import EvaluatorConfig, HistoryConfig, ReplayOverride, ToolPolicyfrom kitaru.client import KitaruAPIClient TERMINAL = {ExperimentRunStatus.COMPLETED, ExperimentRunStatus.FAILED, ExperimentRunStatus.CANCELED}async def wait_for_run(client, run_id): while True: run = await client.experiment_runs.get(run_id) if run.status in TERMINAL: return run await asyncio.sleep(1) policy = ToolPolicy(default=HistoryConfig( scope="cohort_version", on_miss="fail"))evaluator = EvaluatorConfig(evaluator="response-quality", version=1) async def main(): async with KitaruAPIClient() as client: baseline = await client.experiments.create( ExperimentCreateRequest(name="baseline", agent_id=AGENT_ID, tool_policy=policy, evaluators=[evaluator])) candidate = await client.experiments.create( ExperimentCreateRequest(name="cheap-model", agent_id=AGENT_ID, override=ReplayOverride(model="claude-haiku-4.5"), tool_policy=policy, evaluators=[evaluator])) run_spec = ExperimentRunCreateRequest( cohort_version_id=COHORT_VERSION_ID, agent_version_id=AGENT_VERSION_ID, evaluate_baselines=True) before, after = await asyncio.gather( client.experiments.start_run(baseline.id, run_spec), client.experiments.start_run(candidate.id, run_spec), ) async with asyncio.timeout(300): before, after = await asyncio.gather( wait_for_run(client, before.id), wait_for_run(client, after.id), ) asyncio.run(main())cohort_version_id=...An immutable set of sessions — a run depends on its cohort, so a mutable one would make old results meaningless.
experiments.create(...)Pure configuration. Change the code and you get a new run; change the prompt and you get a new experiment.
ReplayOverride(model="claude-haiku-4.5")The variable under test. A run that moved two things cannot tell you which one did it.
HistoryConfig(scope="cohort_version")One of four policies — history, passthrough, static, llm. Every intercepted node is stamped with the one that answered it.
experiments.start_run(...)Each replay creates a new session instead of overwriting the original, so the baseline survives intact.
wait_for_run(...)Wait for both exact runs to settle, then inspect their comparison in Kitaru. It does not invent a verdict or a blended score.
Your agent does the driving.
You do the deciding.
Kitaru speaks MCP. Your coding agent does the tedious parts — import, group, run, read back. It cannot decide what good means, and Kitaru is built to stop it trying.
The agent ran it and laid out the rows. Whether four remaining failures is shippable is not its call.
The questions that come up
in every demo.
How is this different from Langfuse, Braintrust or LangSmith?
They tell you what happened: traces you read, dashboards you check. Kitaru re-runs what happened. Your agent's real code executes again against the recorded world, so you can test your next change against your last thousand sessions before it ships. Kitaru also imports your existing Langfuse, Braintrust or LangSmith traces, so your observability tool stays your system of record.
So is this an observability tool?
No. It sits beside your observability stack. Traces tell you what happened; Kitaru re-runs them against your actual code: a debugger with a memory rather than another dashboard of spans.
Do I have to change my agent's code?
Not to get started. Import your traces and you already get the session views, investigations, cohorts and evaluators; your code stays untouched. An adapter enters only when you want to replay sessions against a change: one line for the supported frameworks, or a small custom one for CLI-harness agents like Claude Code or Gemini CLI.
My agent writes to real systems. Isn't replay dangerous?
Replay answers the agent's tool calls from the recording, so nothing touches real systems. Per-tool policies control the rest: answer from history and stop the run when a call has no recorded answer, pin a static result, or deliberately pass a specific tool through live. We don't test in prod. We make prod's past your test bench.
The model isn't deterministic. How is replay trustworthy?
The recorded world is held constant, same inputs and same tool responses, so the diff you see comes from your change rather than ambient noise. Evaluators and a side-by-side diff do the comparison; nobody has to pretend LLMs are deterministic. (And no, temperature=0 is not determinism. We have the divergence data.)
Where do the eval criteria come from? We never wrote any down.
From the people who already judge the agent every day. Your coding assistant, using Kitaru's investigation skill, interviews you over real sessions, pins your judgments to the exact evidence in the trace, and drafts evaluators from them. Each evaluator is checked against your verdicts before it gates anything.
What frameworks does it work with?
Recording adapters wrap your existing agent in one line, with no rewrite. Python: PydanticAI, the OpenAI Agents SDK, and LangGraph (including LangChain agents and Deep Agents). TypeScript: the Vercel AI SDK and Mastra. Anything else: import the trace history you already have from Langfuse, LangSmith, Braintrust or Pydantic Logfire, or write a one-page custom importer. Traces in raw OpenTelemetry format convert to Kitaru's JSONL import format.
My agent is TypeScript. Can I use Kitaru?
Yes, natively. TypeScript agents record and replay through the Vercel AI SDK and Mastra adapters, with a framework-neutral TypeScript SDK alongside. The CLI, workers and evaluators run on Python today, so there's Python in the loop even when the agent itself is TypeScript.
Is it open source? Can I self-host?
Yes: Apache 2.0, self-hosted by default. The server and workers run in your infrastructure and replay executes in your environment, so your traces never have to leave your systems. ZenML Pro offers a managed version if you want one.
Does this replace human review?
No. Evals change how much humans review, not whether they do. The goal is that people spend their review time on the sessions that deserve it, with evidence attached.
Who is this for, and who isn't it for?
Teams shipping agents to customers whose regression process is honestly a few samples and a vibe check. Kitaru installs the rigor loop. It fits badly for single-dev prototypes and for teams buying a fully managed agent platform: if you're buying an agent platform, Kitaru will feel low-level. If you're building one, that's the point.
Something's broken. How do I reach you?
Three routes, all reaching a human: the Slack community, kitaru.ai/help (goes straight to GitHub issues), or support@kitaru.ai. An issue with a session ID attached gets fixed fastest.
What arrives as a complaint
leaves as a regression test.
uv add "kitaru[cli,worker]" kitaru-pydantic-ai14-day free trial · Full access · No credit card · Open source (Apache 2.0)
Replays run from recordings — production never sees a test.
Foundations and proof for Kitaru
years building production workflow infrastructure
in Kitaru today
From ZenML to Kitaru
Why we built a second product, and what we kept from production ML infrastructure.
Read postThe Anatomy of a Production Coding Agent
Not a prompt and a while loop — eight stages, each with different failure modes, costs, and human touchpoints.
Read postAgents need more than traces
Traces show you what happened. The case for infrastructure that lets you act on it.
Read post