AI that keeps
getting better

An always-on AI engineering team that watches your agents, finds what’s broken, and proves the fix works.

Start free

Or start open source

curl -fsSL https://zenml.io/install-kitaru | bash
An AI engineer at work on an insurance claims agent. It imports 18,402 traces from Langfuse, LangSmith and Braintrust where they already live. It reads the agent's code on GitHub and finds the prompt and 9 evals. It finds a gap in the evals: decision_format passed 31 appeals that never reached a decision, so it drafts a decision_made check. It tests a prompt fix as an experiment on 214 past appeals against what runs today: appeals decided go from 86% to 100%, agreement with reviewers from 91.2% to 93.8%, and none of 40 good appeals change. Coming next: it fine-tunes an open model on 41,200 of your traces that passes 9 of 9 evals at $0.10 a decision instead of $0.36.
See the claims agent benchmark

How it works

Meet your AI engineering team

One for each job, always on. Ask them anything, or find the answer waiting in the morning. Add your own for anything else.

  1. 01

    Eval engineer

    Reads every agent session next to your evals, finds what they miss, and writes the check.

  2. 02

    Reliability engineer

    Works out why a session or a pipeline run failed, fixes it, and tests the fix against what runs today.

  3. 03

    Cost engineer

    Reviews your agents and pipelines every week, finds the waste, and tests a cheaper model before you switch.

  4. 04

    Fine-tuning engineer

    Coming next

    Trains a cheaper open model on your traces as a ZenML pipeline and holds it to the same evals. You get the weights.

Benchmarks

Measured on real work, published in full

We run real claims, appeals and underwriting cases through open and frontier models, test model swaps, and let our AI engineer improve the agent batch after batch. Every result publishes its method, its noise and its limits.

All benchmarks

  • Improving a Claims Agent

    Who keeps a claims agent improving, batch after batch?

    Gain over the agent left alone, 800 appeals · higher is better
    ZenML AI engineer
    +4.0 points
    Codex
    +0.7 points

    Agent improvement · Model switch · Public data

  • Underwriting Agent

    Which model should an underwriting agent run on?

    Cost per correct decision, models tied at the top · lower is better
    Claude Haiku 5.5
    $0.0038
    GLM 5.3 Flash
    $0.0092
    GLM 5.3
    $0.0737

    Model choice · Prompting · Cost · Public data

  • Appeals Decisions

    Can an agent decide real appeals?

    Accuracy against real outcomes, Qwen3.8-27B (dense) · higher is better
    Health appeals
    76.0%
    Veterans' appeals
    80.2%
    German health suits
    61.0%

    Reliability · Open models · Public data

  • SwapBench

    Is it safe to swap the model behind your agent?

    Right verdicts out of 10 swaps · higher is better
    Next-step agreement
    3 right, 7 wrong
    Full replay, 60 scenarios
    4 right, 0 wrong
    Full replay, 260 scenarios
    5 right, 0 wrong

    Model swap · Customer service · Kitaru replay

How it fits

Plugs into the stack you already run

Nothing to rewrite, no new framework. Your AI engineering team works from the tools you already use, wherever you need it to run.

Your traces

  • Langfuse
  • Braintrust
  • LangSmith
  • Phoenix
  • MLflow
  • Logfire

Your evals

  • LLM judges
  • pytest
  • promptfoo
  • DeepEval
  • Scores in your tracer

Your agent

  • Claude Agent SDK
  • OpenAI Agents SDK
  • PydanticAI
  • LangGraph
  • Vercel AI SDK
  • Mastra

Run it in our cloud, your cloud or air-gapped. Your traces stay where they are, read through keys you can revoke.

  • SOC 2 Type II
  • ISO 27001
Security and trust

Five years running AI in production, from pipelines to agents, at

  • AXA
  • JetBrains
  • ADEO
  • Leroy Merlin
  • Brevo
  • Safran
  • AECOM
  • Airbus Defence & Space
  • Rohlik
  • Knuspr
  • Maven Robotics
  • CrossScreen Media
  • GEMA
  • Homa Games
  • Koble
  • IKEA
  • Sciemo
  • Vodafone
  • Stepstone
  • Neara
  • Rivian
  • Happening XYZ
  • Veridas

What is ZenML Pro?

ZenML ProFrom $39 a month · 14 days free

Your AI engineers, across your agents and your ML

ZenML Pro is our paid platform. It hosts Kitaru and ZenML for your team and adds a team of AI engineers that watch every agent trace and every pipeline run, around the clock. One account, one bill.

  • AI engineers
  • Hosted dashboard
  • Schedules and alerts
  • Unlimited seats
See pricing

Built on two open source products

Orchestrate AI on the infra you choose

Run pipelines and agents in Python on the orchestrator and cloud you already have, with every run, artifact, and model version recorded.

Explore ZenML

Find what your agent’s evals miss

Import the traces you already have, see where your evals pass the wrong thing, turn your verdicts into evaluators, and test any change on real sessions before it ships.

Explore Kitaru

Both are free and Apache 2.0. Use them on their own, on your own machines.

Better AI
starts here

Connect your repo and your tracing keys. Your AI engineers get to work right away. 14 days free, no credit card.

Start free