Better, faster, cheaper agents, tested on production data
Your coding agent investigates real sessions, finds what repeats, and replays the fix, so you know what improved, what regressed, and what it costs before it ships.
Watch the launch video, 90 seconds
Install, then let your coding agent do the investigation.
One command on your laptop. One sentence to Claude Code, Codex, or Cursor. The skill finds what repeats in your real runs, fixes it, and shows where you can save cost without losing quality.
$ curl -fsSL https://kitaru.ai/install | bash✓ Kitaru installed. Skills and MCP wired into Claude Code.$ kitaru login --local✓ Local server at http://localhost:8000 # in your agent's repo> Use kitaru-investigation to investigate this agent. Imported 1,824 sessions from Langfuse. 1,791 ready to replay. Read 20. One pattern repeats: the hazmat flag is dropped when the customer mentions batteries. 90 of 1,824.cohort "dropped-hazmat-flag" · 90 sessions · frozen v1 Three opened in the UI. Real bug?> yes Evaluator drafted. Replaying the 90 against your branch pr-311.v1 90 / 90 failpr-311 4 / 90 fail Suggested improvement: same cohort, cheaper model.gpt-5 4 / 90 fail $0.31 per sessiongpt-5-mini 4 / 90 fail $0.09 per sessionsame quality, a third of the costcurl -fsSL https://kitaru.ai/install | bashThen, in your agent's repo, tell your coding agent:Use kitaru-investigation to investigate this agent.
- 1,824sessions imported
- 20 readnot 1,824
- 90 → 4failing after the fix
- 3×cheaper, same quality
Start where you are
Already have traces? Good. Starting fresh? Also good.
Import what your current tools already collected, or wrap your agent in one line and record new runs. Both roads end at the same thing: sessions you can replay.
Have traces? Import them.
$ kitaru session import langfuse-export.jsonl \
--importer kitaru/langfuse@latest \
--agent support-agent@latest \
--tag imported-baseline \
--media-type application/x-ndjson \
--wait
reading langfuse-export.jsonl … 452 traces
converting to sessions … done412 sessions created · 38 skipped · 2 failed
replay readiness ready 361 · partial 44 · unavailable 7
Have an agent? Wrap it.
from kitaru_pydantic_ai import KitaruAgent
from pydantic_ai import Agent
agent = KitaruAgent(Agent(
"openai:gpt-5-mini",
system_prompt="You are a compliance reviewer.",
tools=[search_docs, fetch_policy],
), agent_id=AGENT_ID)
result = await agent.run(task)Agent running on Claude Code, Codex, or the Gemini CLI? Import its session logs today, no adapter needed. For replay, the adapter-builder skill wraps the harness call the same way.
You already suspect what's wrong.
Now you can check.
Every trace you import becomes a session, a replayable record of one real agent run. Take the ones that went sideways, change the one thing you suspect, and run them again. Two runs, side by side, and you know.
What does good even mean here?
Read twenty of them and write what you notice. Your notes become the cohorts.
Can I believe my own check?
Apply it blind to sessions you never read, then reveal the labels.
Can we ship the cheaper model?
Same cohort, one model swapped — the answer is two runs, compared.
Will it stay fixed?
The same experiment runs on every commit, on the cases that caught it.
Want the cheaper model?
Here's how you find out if it's safe.
Freeze a set of real production sessions. Replay them with the model you have and the model you want. Then see exactly which cases changed, and whether those changes are acceptable.
Pick the production cases that matter
Start with a cohort you found during investigation, or hand-pick your own. Versioning the set means you compare every change against the same cases.
Change one thing
Keep the agent setup fixed and swap the model. Now any difference in the replay comes from the change you are testing.
See what actually changed
Replay the same cohort with both setups and compare cost, latency, evaluator results, and behavior side by side.
Out of 200 real sessions, 196 came back unchanged. Now you only have four differences to review before deciding whether to ship.
Illustrative numbers from a sample run.
Frequently asked
questions
Everything you need to know about Kitaru.
How is this different from Langfuse, Braintrust or LangSmith?
They tell you what happened: traces you read, dashboards you check. Kitaru re-runs what happened. Your agent's real code executes again against the recorded world, so you can test your next change against your last thousand sessions before it ships. Kitaru also imports trace files exported from Langfuse, LangSmith, Braintrust, Logfire or Arize Phoenix, so your observability tool stays your system of record.
So is this an observability tool?
No. It sits beside your observability stack. Traces tell you what happened; Kitaru re-runs them against your actual code: a debugger with a memory rather than another dashboard of spans.
Do I have to change my agent's code?
Not to get started. Import your traces and you already get the session views, investigations, cohorts and evaluators; your code stays untouched. An adapter enters only when you want to replay sessions against a change: one line for the supported frameworks, or a small custom one for CLI-harness agents like Claude Code or Gemini CLI.
We don't have agents, just LLM calls inside a workflow. Is Kitaru for us?
Yes. A one-shot call with a prompt, a model and a structured output, say an address pulled out of a PDF, is a session like any other. Import it from Langfuse or wrap the call, group the ones that matter into a cohort, and replay them against a cheaper model or a new prompt before the change ships. Most teams start exactly here. Tool calls and multi-step agents add replay policies on top; they are not a requirement.
My agent writes to real systems. Isn't replay dangerous?
Replay answers the agent's tool calls from the recording, so nothing touches real systems. Per-tool policies control the rest: answer from history and stop the run when a call has no recorded answer, pin a static result, or deliberately pass a specific tool through live. We don't test in prod. We make prod's past your test bench.
The model isn't deterministic. How is replay trustworthy?
The recorded world is held constant, same inputs and same tool responses, so replay removes every source of variation except the model itself. For the variation that remains, you create multiple experiment runs over the same cohort and compare the distributions, so you can tell a real regression from run-to-run noise instead of judging from a single sample.
Where do the eval criteria come from? We never wrote any down.
From the people who already judge the agent every day. Your coding assistant, using Kitaru's investigation skill, interviews you over real sessions, pins your judgments to the exact evidence in the trace, and drafts evaluators from them. Each evaluator is checked against your verdicts before it gates anything.
What frameworks does it work with?
Recording adapters wrap your existing agent in one line, with no rewrite. Python: PydanticAI, the OpenAI Agents SDK, and LangGraph (including LangChain agents and Deep Agents). TypeScript: the Vercel AI SDK and Mastra. Any framework at all, if the agent already reports to Langfuse, LangSmith, Braintrust, Logfire or Phoenix: the importer-backed adapter wraps the entrypoint and pulls the provider's trace in as the session, at the cost of replay being passthrough only. For a custom harness or a framework we don't support yet, Kitaru ships a skill that walks your coding assistant through generating a new adapter for it.
Can I use traces I already have?
Yes. Importers bring in trace files exported from Langfuse, LangSmith, Braintrust, Pydantic Logfire or Arize Phoenix, or you can write a one-page custom importer for your own store. The Phoenix importer reads JSONL downloaded from the Phoenix UI or JSON retrieved with the Phoenix CLI; it does not connect to the Phoenix API or use Phoenix credentials. Traces in raw OpenTelemetry format convert to Kitaru's JSONL import format.
My agent is TypeScript. Can I use Kitaru?
Yes, natively. TypeScript agents record and replay through the Vercel AI SDK and Mastra adapters, with a framework-neutral TypeScript SDK alongside — and the adapter-generation skill covers custom TypeScript harnesses too. The CLI, workers and evaluators run on Python today, so there's Python in the loop even when the agent itself is TypeScript.
Is it open source? Can I self-host?
Yes: Apache 2.0, self-hosted by default. The server and workers run in your infrastructure and replay executes in your environment, so your traces never have to leave your systems. ZenML Pro offers a managed version if you want one.
Does this replace human review?
No. Evals change how much humans review, not whether they do. The goal is that people spend their review time on the sessions that deserve it, with evidence attached.
Who is this for, and who isn't it for?
Teams shipping agents, or LLM steps inside a product workflow, to customers whose regression process is honestly a few samples and a vibe check. Kitaru installs the rigor loop. It fits badly for single-dev prototypes and for teams buying a fully managed agent platform: if you're buying an agent platform, Kitaru will feel low-level. If you're building one, that's the point.
Something's broken. How do I reach you?
Three routes, all reaching a human: the Slack community, kitaru.ai/help (goes straight to GitHub issues), or support@kitaru.ai.
What arrives as a complaint
leaves as a regression test.
curl -fsSL https://kitaru.ai/install | bash14-day free trial · Full access · No credit card · Open source (Apache 2.0)
Replays run from recordings — production never sees a test.