// kav-eval — the Kavron RLHF pipeline
Kav-Eval is the software our interview-vetted experts work inside: a walled-garden grading pipeline where AI screens every row, calibrated humans verify what matters, and the hardest data gets multi-level adjudication — shipped with agreement metrics and error bars, not promises.
// three levels of vetting, one pipeline
// tier 1 — AI vetting
Every submission passes automated gates before a human ever sees it: schema and duplicate checks, AI-content detection on submitted work, reward-model quality scoring, and consistency sampling (the same judgment posed twice — wavering answers are never trusted).
// tier 2 — AI + human
The machine routes, humans decide: every AI-flagged row plus a deterministic statistical sample goes to a domain expert in the walled-garden workbench. Experts are allow-listed, scored against golden sets, and their agreement rates are tracked continuously.
// tier 3 — multi-level adjudication
High-end sets — clinical, legal, structural-engineering RLHF — get multiple independent expert grades. Consensus below threshold escalates to a senior judge; disagreement is itself recorded as signal. Shipped with inter-rater agreement (κ) and published error bars.
// the state machine underneath
raw
maker submits
ai_screen
tier-1 gates
reviewing
expert grade(s)
escalated
judge on low consensus
approved
certified + audited
// rejected and low-consensus rows are signal, not waste — disagreement patterns feed the calibration loop and the next golden set.
// inside the walled garden
One row at a time, DLP-locked, fully audited. Left: the task and the two model responses. Right: winner, confidence, and a Markdown rationale that becomes part of the training signal.
Grading Workbench
Illustrative · synthetic dataWalled garden · one row at a time · expert #DEMO-01
Sample #DEMO-0001
Prompt
Model A
Model B
Your grade
Winner
Confidence / logic score: 9/10
Rationale (Markdown) *
Adjudication queue
consensus < threshold → senior judge
| Task | Domain | Consensus | State | Agreement |
|---|---|---|---|---|
| #DEMO-07 | Civil Eng — Structural | 2 / 3 agree | escalated → judge | κ 0.81* |
| #DEMO-08 | Medical — Clinical RLHF | 3 / 3 agree | approved | κ 0.92* |
| #DEMO-09 | Civil Eng — Structural | 1 / 3 agree | judge reviewing | κ 0.54* |
| #DEMO-10 | Legal — Contract QA | AI flag: style dup | reviewer assigned | — |
Illustrative rows · synthetic data
* example values
// illustrative UI — synthetic sample data, not real tasks
// engineered for provenance
Copy/paste and selection are blocked in the workbench shell; the expert's own rationale field is the only writable surface. Nothing leaves the room.
Checkout leases exactly one task; it must be graded or released before the next. No batch scraping of the queue, complete per-row timing telemetry.
A session is necessary but not sufficient — the DB-backed expert allow-list is authoritative, fail-closed. Every grader maps to a verified interview-vetted profile.
Every state transition, grade, and escalation is an append-only audit row with a server-assigned actor. Anti-cheat detections block payout before export.
The browser never talks to the data plane: all traffic passes the Next.js gateway into the pipeline API, one authenticated seam for rate-limits and policy.
Golden-set rows are interleaved invisibly; per-expert precision drives routing weight. Certification metrics (accuracy ± CI, κ) ship on every dataset card.
// evaluate us the way we evaluate data