AI Reliability Engineering
Turn “good enough to ship” into a release decision you can defend.
30 minutes. Bring the evidence, trace, or deadline.
This page is for
- AI product and platform teams preparing a customer-visible launch
- Engineering managers tired of weekly spreadsheet reviews and surprise regressions
- Security or support owners who need high-risk cases to block release
Start here
The short answer
Direct answer
LLM evaluations before launch should test the workflows that matter to customers, not just whether a model produces fluent text. The minimum useful system connects a versioned golden dataset, failure taxonomy, deterministic checks, calibrated graders, human review, and release actions to the exact model, prompt, retrieval, and tool changes being shipped.
Search intent this page serves: The average score improves while a high-risk customer workflow gets worse.
Failure model
The problems this intervention is designed to expose
- The average score improves while a high-risk customer workflow gets worse.
- A grader rewards fluent answers that are unsupported, unsafe, or operationally incomplete.
- A prompt, retrieval, tool, or model change ships without a reproducible baseline.
- A test set contains only happy paths and misses tenant, language, escalation, or refusal behavior.
Stage-level diagnosis
Where the failure can live
The diagnostic table prevents teams from optimizing the last visible layer while the first broken layer remains unchanged.
| Surface | Failure or risk | Control to inspect | Evidence | Priority |
|---|---|---|---|---|
| Case design | The dataset reflects demos rather than customer workflows and known failure modes. | Build cases from traces, support tickets, incidents, acceptance criteria, and adversarial scenarios; tag risk slices. | Dataset manifest, provenance, slice counts, and review record. | Critical |
| Evaluation | A judge rewards style or keyword overlap instead of correctness, grounding, safety, or completion. | Use deterministic checks first, rubric graders for semantic properties, and human calibration on ambiguous cases. | Rubric, grader agreement, examples, and adjudication notes. | High |
| Release gate | A global average hides a critical regression or teams cannot explain who accepts it. | Set per-slice thresholds, hard blockers, minimum sample rules, and an explicit exception owner. | CI report, threshold config, failed-case artifacts, and approval record. | Critical |
| Trace linkage | A failed case cannot be connected to the prompt, model, retrieval, tool, or release that caused it. | Record versions and trace IDs for every evaluation run and keep the previous baseline available. | Run metadata, artifact links, commit SHA, model ID, and comparison report. | High |
| Post-launch loop | Production failures never become new regression cases or the suite becomes stale. | Sample outcomes, triage failures, redact and add representative cases, and review coverage on a fixed cadence. | Failure intake, dataset PRs, coverage report, and review calendar. | Medium |
Metrics and thresholds
Metrics that change a release or repair decision
Every target is set against a baseline and a business consequence. Exact thresholds are agreed with the product owner before implementation.
| Metric | Why it matters | Baseline | Target | Release action |
|---|---|---|---|---|
| Critical workflow pass rate | A launch can be unsafe even when the aggregate score looks strong. | Current production or pre-change baseline by critical workflow. | No regression on critical slices; target is agreed per workflow. | Block release on any material critical-slice regression without named exception. |
| Grounded claim support | Fluent answers can still invent policy or cite irrelevant evidence. | Human-calibrated sample with supported/unsupported claim labels. | At least the baseline, with a hard floor for high-impact answers. | Block or require human review when unsupported claims exceed the slice floor. |
| Unsafe action rate | A small number of unauthorized actions can outweigh many correct drafts. | Adversarial and permissioned tool cases by action class. | Zero unapproved high-impact actions in the release suite. | Immediate block and incident-style triage for any confirmed unsafe action. |
| Grader-human agreement | A poorly calibrated grader creates false confidence. | Double-scored sample with adjudication for disagreement. | Agreement target set before scale; disagreements remain visible. | Pause automated gate if grader drift or disagreement exceeds tolerance. |
| Case coverage | Unrepresented tenants, languages, tools, and failure modes are invisible risk. | Matrix of workflow, persona, data, language, model, and failure class. | Every launch-critical slice has minimum case count and owner. | Treat missing coverage as an explicit launch risk, not a passing score. |
Production-shaped cases
Scenarios that make the diagnosis reproducible
Golden-set regression after a prompt change
- Precondition
- A new system prompt is expected to improve helpfulness without changing tool policy.
- Exercise
- Run baseline and candidate against normal, ambiguous, refusal, citation, and tool-authorization cases.
- Expected control
- Critical slices remain within threshold; any trade-off is visible with a decision owner.
- Evidence captured
- CI comparison, failed transcripts, prompt version, and approval record.
Grader fooled by fluent unsupported content
- Precondition
- A judge scores answer quality using a rubric and retrieved context.
- Exercise
- Supply an answer that is well-written but contradicts the source or invents a policy exception.
- Expected control
- Citation or claim-support checks and calibrated grading mark it as a failure.
- Evidence captured
- Rubric result, human label, disagreement note, and new regression fixture.
High-risk tool case hidden by averages
- Precondition
- Most evaluation cases are read-only or low impact, with a small action slice.
- Exercise
- Induce a refund, permission, deletion, or account change with an unauthorized target.
- Expected control
- The hard blocker fires regardless of aggregate pass rate or average reward.
- Evidence captured
- Tool trace, policy result, CI gate, and exception handling.
Production failure becomes a test
- Precondition
- A customer report or trace reveals a new failure mode after launch.
- Exercise
- Redact the example, classify its cause, reproduce it, and add it to the dataset with an owner.
- Expected control
- The failure is linked to a test, a fix, and a future release gate.
- Evidence captured
- Dataset change, issue link, rerun output, and coverage update.
Implementation handoff
Artifacts, access, and working sequence
Artifacts
- Versioned golden dataset with case provenance and risk slices
- Failure taxonomy covering correctness, grounding, safety, escalation, latency, and cost
- Deterministic checks, grader rubrics, calibration sample, and human-adjudication record
- CI release report with thresholds, blockers, exceptions, and trace-linked failures
- Coverage dashboard and workflow for promoting production failures into cases
- Launch decision memo with rollback criteria and open risk owners
Access requirements
- Representative traces, support cases, acceptance criteria, or incident examples
- Current and candidate prompt/model/retrieval/tool versions
- A CI or reproducible evaluation environment
- One product owner and one technical owner for threshold decisions
Days 1–2
Failure taxonomy
Turn customer outcomes and known risks into case classes, critical slices, and explicit pass/fail rules.
Days 3–4
Harness and graders
Build the dataset, deterministic checks, rubric graders, calibration sample, and trace metadata.
Days 5–6
CI gate
Compare baseline and candidate, set thresholds, expose failed cases, and implement the release decision.
Day 7
Launch rehearsal
Run an exception, rollback, and production-failure intake drill; hand off the suite and owner schedule.
Rollback and limits
What happens when the threshold fails
A release is held or rolled back when a hard blocker fails, a critical slice regresses beyond its threshold, or the evaluator is not calibrated enough to support a decision. The rollback artifact should identify the last passing model/prompt/retrieval/tool versions and preserve the candidate failures for triage.
Scope limit
A benchmark score is not a guarantee of production quality.
Scope limit
Automated graders require calibration and ongoing drift checks; they should not be the only control for high-impact workflows.
Scope limit
The page describes an engineering evaluation system, not a compliance certification or a particular vendor’s product.
Sources and limits
References used for the operating model
These sources provide the framework or vocabulary. The page adds product-boundary tests, evidence requirements, and implementation decisions so the result can be inspected in a live system.
A reference point for model graders; production teams still need calibrated rubrics, human review, and release thresholds for their own workflows.
Metrics such as faithfulness and context precision are useful ingredients; they do not replace product-specific failure slices or acceptance rules.
FAQ
Questions buyers and engineers ask
How many cases do we need before launch?
There is no universal number. Every launch-critical workflow and failure slice needs enough representative cases to expose meaningful regressions, with minimum sample rules documented before the gate is trusted.
Should we use an LLM-as-a-judge?
Often, but only with a product-specific rubric, deterministic checks, and human calibration. A judge that cannot distinguish fluent unsupported answers from grounded ones should not block or approve a release alone.
What is the difference between an eval and a dashboard?
An eval produces a reproducible decision tied to a version and threshold. A dashboard can show behavior, but it does not create a release gate, owner, exception rule, or rollback action by itself.
Next step
Turn this page into an owned engineering decision.
Bring the questionnaire, trace, failed workflow, or provider deadline. We will help decide whether a focused implementation is the right scope.
30 minutes. No deck. Leave with a clear next step.