Menu

Agent data evaluation · open source · Apache-2.0

Know which data actually makes your AI better.

Measure which documents, examples, policies, and conversations improve your agent, and by how much. Rank them by impact and cost, then put each where it belongs.

Run the quickstart, freeView on GitHub ↗
curl -sSfL https://raw.githubusercontent.com/uknoAI/kno/main/install.sh | sh
  • v0.1 · early
  • Single Go binary
  • No infrastructure
  • Works with your existing evals
  • OpenAI-compatible endpoints
  • Anthropic · Bedrock · Vertex
  • exec: your own agent
  • Local fake agent, free
  • Budget guarded
  • Apache-2.0
KNO / VALUERUN 0142
ASSET                  IMPACT    DECISION
new_refund_policy.md    +18%    keep → knowledge_base
example_42.json         +7%    keep → context
example_91.json         +1%    reject
old_refund_policy.md    -9%    reject → harmful

Illustrative numbers; real runs report each delta with a 95% confidence interval.

See the real output, free local agent, no API keys
kno value, actual runreal output
ASSET         DELTA (95% CI, positive = goal dir)  CONTROL             NOTE
brand-guide   +0.0000  [-0.3960, +0.3960]   low -0.5815 (underpowered)
refund-example-17  +0.0000  [-0.3960, +0.3960]   low -0.5815 (underpowered)
refund-policy-v3  +0.0000  [-0.3960, +0.3960]   low -0.5815 (underpowered)

Deltas read zero because the free agent answers every case with what the case expects. The point is the loop: routing, injection, fresh controls, intervals, all working, for $0.

[01 / Problem]

You're measuring your models. Why aren't you measuring your data?

Agent teams route data constantly, into RAG, into context, into the fine-tune, with intuition as the decision method. Five hundred help-center docs, two thousand support conversations, fifty policies: which earn their place? The damage shows up later, a stale policy contradicting the new one, a few-shot example dragging answers down. Kno turns each of those decisions into a measured experiment.

WITHOUT KNO
  • Documents────────►RAG
  • Examples────────►Context
  • Conversations─────►Fine-tuning
  • Policies──────────►???

DECISION METHOD / INTUITION

WITH KNO
  • Data─────►KNO─────►impact
  • ├──►confidence
  • ├──►cost
  • ├──►destination
  • └──►keep / reject

Same data. Same agent. The only difference is who decides — and the number on the table when they do.

[02 / Outcomes]

Know what helps. Know what hurts. Know what it costs.

  1. 01

    Know what helps

    Measure the marginal impact of each candidate asset against your own evals.

  2. 02

    Know what hurts

    Find contradictory and harmful data, and regressions other tools miss, before they reach production.

  3. 03

    Know what it costs

    Compare improvement against inference cost. Ranking is per dollar, not per point.

  4. 04

    Know where it belongs

    Context, knowledge base, or tuning set, every selected asset gets a destination, not just a score.

  5. 05

    Know what's missing

    Find the case types nothing in your data fixes, the report lists them as gaps (GAP / UNKNOWN), so you know what to collect next.

[03 / MECHANISM]

Every step answers one question.

  1. Initshipped

    How is this run set up?

  2. Mineshipped

    Where do the evals come from?

  3. Baselineshipped

    How good is the agent now?

  4. Valueshipped

    Which assets improve it?

  5. Selectshipped

    Which combination should I keep?

  6. Validateshipped

    Does the combination still work on untouched evals?

  7. Exportshipped

    Where should each asset go?

  8. Reportshipped

    What does the whole run say?

Status as of v0.1.5, generated from the repository — the full table is on the roadmap.

What each stage does, one page →

[04 / Use cases]

Answers to questions you already have

  1. 01Which examples deserve space in my context window?
  2. 02Which conversations should I fine-tune on?
  3. 03Which data is actively hurting my agent?
  4. 04Is this policy actually improving my agent?
  5. 05Which documents should go into my RAG system?
  6. 06Which data is redundant?
  7. 07Which knowledge sources actually help my support agent?
[05 / Objections]

Questions engineers actually ask

Does Kno replace my eval framework?
No, it uses them. Kno re-scores your agent with each candidate asset and reports which asset moved the score, by how much, and whether it's worth keeping. Your evals stay the source of truth.
Does Kno send my data somewhere?
Runs are stored locally in SQLite, including agent output, which may be conversation content. Kno itself sends nothing anywhere: there is no telemetry of content, ever. Your cases go only to the provider you point Kno at, and that provider's retention is theirs.
What happens to traces?
They stay on your disk until you delete them, and nothing expires on its own. `kno purge` removes trace content when you no longer need it, keeping the scores and costs so the run stays resumable.
Can it accidentally spend unlimited API money?
No. Every path that can call a provider goes through a budget guard: it estimates the cost, asks you to confirm, and checkpoints spend as it goes. Caps bind before the call, not after the bill, and if the cost can't be estimated, the run is refused up front. `--max-cost-usd` with `--yes` runs without prompts, and an interrupted run resumes without paying twice.
Are the results statistically meaningful?
Kno reports confidence intervals, never naked point estimates, a delta without its interval is not reported at all. Controls measure regression separately, selection decisions are made at Bonferroni-corrected intervals, and small samples are flagged underpowered instead of dressed up as answers. The honest number is the holdout number.
Why not script this myself?
You can, until the edge cases find you. Kno's differences are the boring parts done as mechanisms: the holdout is sealed by the type system, not convention, so selection physically cannot peek. Every keep/reject decision is a Bonferroni-corrected interval, not a glance at a chart. Runs checkpoint per case and resume without paying twice, and a budget cap is estimated and confirmed before the first call. None of that is hard to fake once; it is hard to get right every time. We do not ask you to take that on trust: the hundred-line version is committed and run in CI beside Kno on the same cases, with both outputs printed side by side in why not just script this yourself.
What providers work?
`openai:` for any OpenAI-compatible endpoint (vLLM, Ollama, llama.cpp need no key), `anthropic:` for the Anthropic API, `bedrock:` for Claude on AWS (Converse, SigV4), `vertex:` for Claude on Google Cloud (`:rawPredict`), `exec:` for your own agent behind a shell command, and `fake:` for the local agent that costs nothing. `tuned:` arrives with the stages that need it. Keys come from the environment, never from a flag, `kno doctor` prints what this build supports and contacts nothing.
[06 / Quickstart]

From zero to your first valuation

No API keys, no spend. Five commands.

  1. Install

    Single Go binary, checksum-verified. Homebrew and go install also work, see the install guide.

    curl -sSfL https://raw.githubusercontent.com/uknoAI/kno/main/install.sh | sh

  2. Write some cases

    One scoreable interaction per line. Ten gives Kno room to seal a holdout, with fewer than three cases it refuses to start: a run that can never produce a holdout number is not worth spending.

    for i in $(seq 1 10); do printf '{"id":"refund-%02d","input":"How do I get a refund?","expected":"Refunds are issued within 5 business days."}\n' "$i" done > cases.jsonl

  3. Measure the agent as it is today

    The reference every later number is compared against.

    kno baseline --evals cases.jsonl

  4. Write a pool of candidate assets

    The documents, examples, and policies you are considering adding.

    cat > pool.jsonl <<'EOF' {"id":"refund-policy-v3","content":"Refunds are issued within 5 business days.","kind":"knowledge"} {"id":"refund-example-17","content":"Example: a 30-day-old refund request is declined.","kind":"knowledge"} {"id":"brand-guide","content":"Use sentence case everywhere.","kind":"knowledge"} EOF

  5. Value them

    Each asset is injected into the slices it could affect, re-measured against controls, and reported with an interval. Paste the run id from step 2's Baseline line. The deltas read zero because the fake agent answers every case with what the case expects, the point is proving the loop, not the asset.

    kno value --evals cases.jsonl --pool pool.jsonl --baseline-run-id <run id> refund-example-17 +0.0000 [-0.3960, +0.3960] low -0.5815 (underpowered) refund-policy-v3 +0.0000 [-0.3960, +0.3960] low -0.5815 (underpowered)

Still zero dollars: everything above ran on the local fake agent. When you're ready for real numbers, point Kno at a real provider and cap the run before it starts: kno baseline --evals cases.jsonl --agent openai:gpt-4.1 --max-cost-usd 5.00 --yes Real runs cost roughly two calls per asset per case it is measured on, plus a control arm: the ten-case quickstart above plans 36 calls for three assets, not 30. Kno prints the exact count before the first call, estimates the bill from it, and the cap binds first.

Read the full walkthrough →

[07 / Differentiation]

Your eval framework measures the agent. Kno measures the data feeding it.

One measurement on top of your evals. Same cases, same scores.

CapabilityEval frameworkKno
Model performance✓uses
Prompt performance✓uses
Asset-level impact—✓
Marginal data value—✓
Data cost—✓
Destination selection—✓
Portfolio validation—✓
[08 / Open source]

Kno is designed to be extended

Five surfaces, from bring-your-own-data to the valuation math itself.

  1. Adapters

    Bring another data source. Langfuse, LangSmith, and Braintrust eval adapters, Hugging Face evals and pools, CSV and Markdown pools, and a full Zendesk recipe all work today, and the pattern transfers to any stack that exports data.

  2. Judges

    Add evaluation logic so Kno can score a new kind of case, a rubric judge, a retrieval judge, yours.

  3. Goals

    Add an optimization target so select can rank for it, answer cost alongside accuracy, say.

  4. Providers

    Connect a new agent runtime. exec: already runs your own agent behind a shell command; the wiring pattern is the extension point.

  5. Core

    Improve the valuation engine itself, plan first, review adversarially, then code.

Find good first issuesRead CONTRIBUTING.md

Your next data decision can be measured.

Install Kno and run the quickstart, the whole loop, in minutes, with zero spend. Point it at a real provider with a budget cap, and the numbers start meaning something.

curl -sSfL https://raw.githubusercontent.com/uknoAI/kno/main/install.sh | sh
Run the quickstart