RAG Reliability Engineering
Find the RAG stage that failed before rewriting the prompt.
30 minutes. Bring the evidence, trace, or deadline.
This page is for
- AI engineers operating support, knowledge, or enterprise search assistants
- Platform leads who see wrong answers but cannot tell ingestion from retrieval failure
- Product and support owners who need measurable improvement in verified resolution
Start here
The short answer
Direct answer
RAG retrieval failure diagnosis starts by locating the broken stage. If retrieval returns no authorized evidence, prompt changes cannot fix it. A useful diagnosis connects the user question, expected source, retrieved IDs, ranks, assembled context, citations, answer, latency, and cost so the team can repair the smallest failing layer.
Search intent this page serves: The knowledge base contains the answer but ingestion or chunking makes it unretrievable.
Failure model
The problems this intervention is designed to expose
- The knowledge base contains the answer but ingestion or chunking makes it unretrievable.
- Retrieval returns relevant text without respecting tenant or document permissions.
- Reranking or context limits remove the evidence needed for a correct answer.
- The answer or citation layer fails even when the right evidence was retrieved.
- Teams optimize token or latency metrics while customer-visible resolution remains flat.
Stage-level diagnosis
Where the failure can live
The diagnostic table prevents teams from optimizing the last visible layer while the first broken layer remains unchanged.
| Surface | Failure or risk | Control to inspect | Evidence | Priority |
|---|---|---|---|---|
| Ingestion and freshness | The source is absent, stale, duplicated, or transformed into unusable chunks. | Track source version, update time, ownership, chunk lineage, and ingestion error state. | Ingestion manifest, freshness metric, failed-job log, and source-to-chunk sample. | High |
| Retrieval and authorization | The correct source is not returned or unauthorized context is included. | Measure recall by expected source and enforce identity/tenant filters before ranking. | Retrieved IDs, filter decision, authorization trace, and recall sample. | Critical |
| Ranking and context | Relevant evidence is ranked below noise or truncated before the model sees it. | Measure rank position, context precision, token budget, duplicate rate, and context assembly order. | Ranked list, assembled prompt, token accounting, and truncation log. | High |
| Citation and grounding | The response claims more than the context supports or cites an inaccessible/stale source. | Validate claims and citations against retrieved content and user access; escalate uncertainty. | Claim/citation pairs, support labels, validator output, and review sample. | High |
| Workflow outcome | A technically correct answer still fails to resolve the customer’s task or causes an unsafe handoff. | Measure verified resolution, reopen/escalation rate, time-to-useful-response, and cost per completed workflow. | Outcome labels, support events, escalation reason, and cost trace. | Medium |
Metrics and thresholds
Metrics that change a release or repair decision
Every target is set against a baseline and a business consequence. Exact thresholds are agreed with the product owner before implementation.
| Metric | Why it matters | Baseline | Target | Release action |
|---|---|---|---|---|
| Expected-source recall | Shows whether the system can find the source that should answer the question. | Labeled questions with one or more expected sources. | Improve critical slices without reducing access correctness. | Block if critical workflows fall below the agreed floor. |
| Context precision | Noise consumes context and makes the answer less reliable. | Relevant-vs-irrelevant retrieved chunks by rank and query slice. | Increase relevant evidence in the context budget. | Tune chunking/reranking or reduce scope when precision falls. |
| Citation support rate | A citation is useful only if it supports the claim and the user can access it. | Claim/citation pairs reviewed with access and freshness labels. | No unsupported claims in high-impact slices. | Escalate or refuse unsupported answers. |
| Verified resolution rate | The business outcome is a solved task, not a plausible paragraph. | Resolved-without-reopen labels, escalation outcome, and human sample. | Increase without trading away safety or CSAT. | Revisit knowledge, routing, and handoff when resolution stalls. |
| Cost per completed workflow | Retries and oversized context can hide waste behind low per-request cost. | Model, embedding, infrastructure, retry, and workflow completion trace. | Lower cost while holding critical quality floors. | Prioritize caching, routing, context reduction, or model change with eval proof. |
Production-shaped cases
Scenarios that make the diagnosis reproducible
Answer exists but retrieval misses it
- Precondition
- A known support answer exists in the source system and has a stable document ID.
- Exercise
- Ask paraphrased, partial, typo-heavy, and tenant-scoped questions; inspect ingestion, recall, rank, and context.
- Expected control
- The diagnosis identifies the first failing stage rather than prescribing a prompt rewrite.
- Evidence captured
- Source version, chunk IDs, retrieval result, rank, context snapshot, and fix comparison.
Relevant chunk removed by the context budget
- Precondition
- The retriever returns relevant and irrelevant chunks with a fixed token budget.
- Exercise
- Increase noisy results or long documents until the expected evidence is truncated or pushed below the model’s usable context.
- Expected control
- The system reports truncation/precision loss and adjusts ranking, chunk size, or context assembly.
- Evidence captured
- Token accounting, ranked list, assembled prompt, and before/after metric slice.
Citation exists but is not support
- Precondition
- The assistant cites retrieved documents in customer-visible answers.
- Exercise
- Ask a question where the source is related but does not establish the claim or is no longer accessible.
- Expected control
- The answer is corrected, hedged, refused, or escalated; the citation validator flags the mismatch.
- Evidence captured
- Claim/citation pair, source permissions, validator decision, and human adjudication.
Quality improves but resolution falls
- Precondition
- A change raises an offline metric while support outcomes are available.
- Exercise
- Compare verified resolution, reopen, escalation, latency, and cost by customer and issue slice.
- Expected control
- The team avoids shipping an offline improvement that harms the completed workflow.
- Evidence captured
- Offline report, outcome report, slice analysis, and release decision.
Implementation handoff
Artifacts, access, and working sequence
Artifacts
- Stage-level RAG trace with source, chunk, rank, context, answer, and outcome links
- Failure taxonomy and labeled diagnostic dataset
- Metric report for recall, precision, rank, citation support, resolution, latency, and cost
- Before/after repair comparison with quality guardrails
- Evaluation cases and CI checks for the diagnosed failure modes
- Operating dashboard definition and owner for ongoing freshness
Access requirements
- A small export of production-shaped traces with sensitive values redacted
- Source and index metadata or a reproducible sample
- Current retrieval/context configuration and model versions
- Support or product outcome labels for a representative slice
Day 1
Trace the stages
Follow one successful, one wrong, and one unanswered workflow from source to completed outcome.
Days 2–3
Label failures
Create stage labels, expected sources, access checks, rank/context metrics, citation support, and outcome fields.
Days 4–5
Repair the first break
Fix ingestion, authorization, ranking, context, citation, or handoff in the smallest complete intervention.
Day 6
Gate the repair
Rerun production-shaped cases, compare workflow outcomes, and leave the diagnostic dataset with the team.
Rollback and limits
What happens when the threshold fails
Keep the previous retriever, reranker, chunking, context template, and model route available until the critical workflow slices pass. Roll back when verified resolution or authorization correctness regresses beyond the agreed floor, even if a retrieval metric improves.
Scope limit
Offline RAG metrics are directional and must be connected to user access and workflow outcomes.
Scope limit
The diagnosis depends on trace and source metadata; a black-box answer-only sample cannot prove where retrieval failed.
Scope limit
This is not a guarantee that a knowledge base is complete, current, or legally sufficient.
Sources and limits
References used for the operating model
These sources provide the framework or vocabulary. The page adds product-boundary tests, evidence requirements, and implementation decisions so the result can be inspected in a live system.
Metrics such as faithfulness and context precision are useful ingredients; they do not replace product-specific failure slices or acceptance rules.
A reference point for model graders; production teams still need calibrated rubrics, human review, and release thresholds for their own workflows.
FAQ
Questions buyers and engineers ask
Why not start with prompt engineering?
If the expected evidence is missing, unauthorized, stale, or truncated before generation, prompt changes cannot repair the first failure. The diagnosis finds the earliest stage that explains the customer-visible result.
What traces are needed?
A small redacted sample should include the question, identity/tenant, expected source where known, retrieved IDs and ranks, assembled context, answer, citations, model/prompt versions, latency, cost, and outcome.
How do you define a successful RAG repair?
Success is a measurable improvement in the affected workflow slice while authorization, citation support, latency, cost, and escalation safety stay within agreed thresholds.
Next step
Turn this page into an owned engineering decision.
Bring the questionnaire, trace, failed workflow, or provider deadline. We will help decide whether a focused implementation is the right scope.
30 minutes. No deck. Leave with a clear next step.