RAG Rescue: From Demo Retrieval to Measured Reliability

Retrieval-augmented generation fails politely. The endpoint returns a valid response, the model writes fluent prose, and the trace contains no exception. The user still receives the wrong policy, a stale limit, a citation that does not support the sentence, or a confident answer to a question the corpus cannot answer. Normal availability monitoring sees a healthy service while product quality is failing.

Begin by naming the observable failure. Retrieval failure means the required source never entered the candidate set. Ranking failure means the source was found but buried below weaker material. Context failure means the right source was selected but truncated, duplicated, or mixed with conflicting text. Generation failure means the context was sufficient and the answer was still unsupported, incomplete, or badly cited. Product failure means the answer may be technically faithful but does not complete the user task.

Do not start by changing the embedding model or rewriting the prompt. Those are interventions, not diagnoses. Instrument the pipeline so one test case preserves the query, filters, candidate identifiers, scores, reranked order, assembled context, answer, citations, model version, prompt version, latency, and token use. Without that trace, every fix is an anecdote.

The dataset should represent the distribution of work the product must handle, not a collection of easy examples written after the architecture was chosen. Start with resolved support tickets, search failures, user feedback, sales demonstrations that required manual intervention, incident reports, and trace samples from high-value workflows. Remove sensitive content or create faithful synthetic variants while preserving the retrieval difficulty.

Every case needs a stable identifier, question, expected source identifiers, reference answer or rubric, tenant and permission context, slice label, and provenance. Store source identifiers rather than exact chunk identifiers because chunking is one of the variables under test. For unanswerable cases, explicitly record that no source supports an answer. Otherwise the evaluation will reward systems that always say something.

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class RagCase:
    case_id: str
    question: str
    expected_document_ids: tuple[str, ...]
    reference_answer: str
    slice: Literal[
        "exact_fact",
        "multi_document",
        "freshness",
        "permissioned",
        "unanswerable",
    ]
    tenant_id: str
    source: str

cases = [
    RagCase(
        case_id="support-017",
        question="Can an annual plan be paused during parental leave?",
        expected_document_ids=("billing-policy-v7",),
        reference_answer="Yes, after support verifies eligibility.",
        slice="exact_fact",
        tenant_id="tenant-demo",
        source="resolved_support_ticket",
    )
]

Slice the dataset by the way failures happen: exact identifiers, synonyms, acronyms, multi-document synthesis, date-sensitive policy, table lookup, long documents, conflicting versions, permissioned content, multilingual questions, vague questions, and unanswerable questions. Add business-critical slices even when they are rare. A single average score gives frequent easy questions the power to hide an unacceptable regression in a high-consequence workflow.

Start small enough to review every case. Fifty to one hundred carefully inspected cases produce more signal than thousands of weak synthetic questions. Add cases when production reveals a new failure class, not merely to increase the count. Keep a holdout set for larger architecture decisions, and prevent prompt authors from tuning against every hidden case.

Define a labeling protocol before scaling the set

Two reviewers can disagree about the expected source even when both understand the domain. Write rules for canonical sources, acceptable alternatives, partially answerable questions, time-sensitive facts, and multi-document evidence. Ask reviewers to label independently, then reconcile disagreements and record the reason. Track agreement by slice. Low agreement often means the product policy or source hierarchy is unclear, not that the reviewers need to work faster.

Keep labels separate from model output. A reviewer who sees the baseline answer first can unconsciously accept its framing or source. Present the question, account context, and searchable corpus before revealing any generated response. For high-consequence cases, require a domain owner to approve the reference answer and prohibited claims. Store the reviewer, timestamp, source versions, and decision note so later changes can be distinguished from labeling mistakes.

Prevent evaluation leakage

The golden set can leak into prompts, few-shot examples, synthetic training data, or tuning conversations. Classify cases as development, validation, or holdout. Developers may inspect development failures freely. Validation supports release comparison but should not become a prompt example. Holdout cases should be accessible only through the evaluation runner and reviewed after major decisions. When a holdout case becomes a debug case, move it to development and replace it with a newly reviewed example.

Version the dataset as an artifact. A changed label can move the score without any product change, so every report must record the exact dataset revision. Review additions for duplicates and near duplicates, since repeated versions of one support issue can overweight a narrow behavior. Weighting should reflect product consequence and expected traffic deliberately, not whatever examples were easiest to collect.

• Cases come from real product workflows and known failures
• Expected sources use stable document identifiers
• Tenant and permission context are part of each case
• Unanswerable questions have explicit expected behavior
• Critical workflows are scored as separate slices
• Every production incident creates a regression candidate

Run the golden questions through the retrieval system without generation. Measure recall at several candidate limits, reciprocal rank, precision where relevant, latency, and filter correctness. Recall at five answers a direct question: did the retriever place an expected source in the first five candidates? Reciprocal rank reveals whether the right source is consistently near the top. Both are transparent enough to debug from a failed case.

def reciprocal_rank(document_ids: list[str], relevant: set[str]) -> float:
    for rank, document_id in enumerate(document_ids, start=1):
        if document_id in relevant:
            return 1.0 / rank
    return 0.0

def recall_at_k(document_ids: list[str], relevant: set[str], k: int) -> float:
    if not relevant:
        return 1.0
    found = relevant.intersection(document_ids[:k])
    return len(found) / len(relevant)

def evaluate_retrieval(case: RagCase, document_ids: list[str]) -> dict[str, float]:
    relevant = set(case.expected_document_ids)
    return {
        "recall_at_5": recall_at_k(document_ids, relevant, 5),
        "reciprocal_rank": reciprocal_rank(document_ids, relevant),
    }

Then add one stage at a time. Compare raw vector search with lexical search, hybrid fusion, metadata filters, and reranking. Preserve candidate lists between stages. If recall is low before reranking, a better reranker cannot recover absent evidence. If recall is high and answer groundedness is low, replacing the vector store is unlikely to help. The stage boundary tells you where the next experiment belongs.

Inspect score distributions rather than copying a universal threshold. Similarity scores vary by embedding model, normalization, index, query style, and corpus. A fixed threshold can silently remove relevant candidates for short questions or admit noise for broad ones. Calibrate thresholds against labeled cases, and include a no-result path when confidence is insufficient.

A retriever cannot return a document that was never indexed, was parsed incorrectly, or lost the metadata needed for access filtering. Measure source coverage: expected documents discovered, parsed, transformed, embedded, indexed, and queryable. Record a content hash, source version, parser version, indexing time, permission state, and canonical document identifier. Alert on age and stage failures by source.

Inspect representative documents after parsing. PDFs can lose reading order, tables can collapse into meaningless text, headings can detach from sections, code blocks can merge with prose, and repeated navigation can dominate chunks. Use format-aware parsers and preserve structural metadata. A chunk should carry the document title, section path, source URL or identifier, updated time, tenant, visibility, and neighboring chunk references.

Chunking is a retrieval policy, not a preprocessing constant. Start from semantic boundaries such as headings, paragraphs, table rows, or code units. Add modest overlap only when a boundary routinely splits necessary context. Large overlapping windows increase index size and return near duplicates that consume the context budget. Evaluate chunk size by question slice, and keep the source document available for parent expansion after a precise child chunk is retrieved.

Handle versions explicitly

Policies, product documentation, and contracts often exist in multiple versions. Indexing every version without a status field asks the model to resolve conflicts it cannot reliably understand. Record effective dates, superseded relationships, publication state, and audience. Default retrieval to current published material, then expose history only when the question asks for it. Include freshness cases in the golden dataset and verify that a superseded document cannot outrank the active source.

Make deletion observable

Deleting a source must remove or invalidate chunks, embeddings, caches, and derived summaries. Track the canonical document identifier through every derived artifact. Test deletion as a product behavior, including the time until stale results disappear. This is both a reliability requirement and a data-lifecycle control.

Semantic search is strong when users paraphrase concepts, but exact error codes, product identifiers, names, and uncommon acronyms often favor lexical matching. Hybrid retrieval gives both methods a path into the candidate set. Fuse ranks rather than raw scores when the systems use incomparable scales. Preserve tenant and publication filters inside each retrieval call, before fusion.

def hybrid_search(query: str, tenant_id: str, limit: int = 20):
    lexical = keyword_index.search(
        query=query,
        filters={"tenant_id": tenant_id, "status": "published"},
        limit=limit,
    )
    semantic = vector_index.search(
        embedding=embed(query),
        filters={"tenant_id": tenant_id, "status": "published"},
        limit=limit,
    )

    fused = reciprocal_rank_fusion(
        rankings=[lexical, semantic],
        weights=[0.45, 0.55],
        constant=60,
    )
    return fused[:limit]

Query rewriting helps when conversation context or shorthand hides the standalone information need. It also creates a new failure boundary. Log the original and rewritten query, evaluate both, and prohibit rewriting from adding a tenant, product, or fact that the user did not supply. For multi-part questions, decomposition can retrieve evidence for each sub-question, but the final answer must retain which evidence supports which claim.

Metadata filters should be derived from trusted application state, not generated by the model. Tenant, role, account, region, plan, and document visibility are authorization inputs. A model may extract a date range or topic, but trusted code must validate allowed fields and combine them with mandatory security filters. Add negative cases for cross-tenant identifiers, missing filters, and content visible only to an unauthorized role.

Route query classes explicitly

Not every question deserves the same retrieval plan. Exact identifiers can start with lexical search. Broad explanatory questions may benefit from semantic retrieval and parent expansion. Questions asking for a comparison can be decomposed into evidence needs, while account-specific questions must join trusted application state rather than search the document corpus. Build a small deterministic or evaluated classifier for these query classes and preserve its decision in the trace.

Evaluate the router independently. A perfect downstream retriever cannot recover when the query is sent to the wrong corpus or workflow. Label expected route, required filters, and acceptable fallback for relevant cases. Measure route confusion by slice. Keep a conservative fallback that runs a broader, permission-safe search when the classifier lacks evidence, and cap fan-out so uncertainty does not multiply latency and cost without bound.

Test authorization as retrieval quality

A cross-tenant result is both a security failure and a retrieval failure. Add paired cases with the same question and different tenant or role context. The permitted case should retrieve the expected source; the denied case should return no source or an approved public source. Run these against caches and rerankers as well as the vector index. Record the trusted authorization context separately from the natural-language question so a prompt cannot override it.

A reranker should improve the order of a candidate set that already contains relevant evidence. Evaluate reranker lift as the change in reciprocal rank or precision while holding the candidate set constant. Inspect regressions by slice. Cross-encoders may improve nuanced relevance at additional latency; lightweight rules may be enough to boost title matches, current versions, or authoritative sources. Choose against the production latency and quality budget.

Context assembly should maximize evidence coverage, not fill the model window. Group nearby chunks from the same parent when they form one argument. Deduplicate repeated passages. Preserve source labels and stable citation identifiers. Order by the structure that helps the task: relevance, chronology, source authority, or question sub-part. Do not concatenate candidates without observing which material is dropped when the token budget is reached.

Conflicting sources require explicit policy. Prefer the active version for policy questions, an authoritative system of record for account facts, and a visible qualification when evidence genuinely conflicts. The model should not silently average two policies. Include conflict cases and score whether the answer identifies the authoritative source or asks for review.

Measure context precision and context recall separately. Precision asks whether included chunks are relevant. Recall asks whether the context contains all evidence needed for the reference answer. High context precision with low recall can produce concise but incomplete answers. High recall with low precision can distract the model and inflate cost. The right tradeoff depends on the workflow and model, and the golden dataset makes it visible.

Measure position and compression effects

Evidence can be present and still be effectively unavailable when it appears between long distractors. Create controlled cases that move the same supporting passage to early, middle, and late context positions. Compare answer support and citation selection. If position changes the result materially, reduce context, group evidence by question sub-part, or repeat a short evidence index before generation rather than assuming the full window receives equal attention.

Context compression introduces another model-backed stage. Evaluate whether the compressed text preserves required facts, qualifiers, dates, negations, and source identifiers. Never compress access-control or provenance metadata away. Keep the original passage linked in the trace and in the citation path. Use deterministic extraction for tables, identifiers, and policy limits when possible; abstractive summaries are most useful for narrative material where exact wording is not itself the evidence.

Give tables and code structure-aware treatment

Flattened tables often retrieve the right row without its headers, while code chunks can lose the type or function that gives a line meaning. Serialize table rows with header names and preserve table identity. Index code with symbol, file path, language, and surrounding declaration. Add cases for row lookup, column comparison, and symbol references. The correct chunk representation is the one that lets a reviewer reconstruct the source, not the one that looks shortest in the index.

Answer evaluation starts after the retrieval baseline is understood. Use deterministic checks for schema, citation existence, allowed sources, required fields, and exact values. Use rubric-based graders for completeness, relevance, refusal quality, and support, then calibrate them against human review. Never ask one judge prompt for a single universal score. A decomposed result is easier to debug and harder to game.

def citation_support(answer_claims, cited_chunks):
    results = []
    for claim in answer_claims:
        candidate_evidence = [
            chunk for chunk in cited_chunks
            if chunk.document_id in claim.citation_ids
        ]
        results.append({
            "claim": claim.text,
            "has_citation": bool(claim.citation_ids),
            "supported": support_grader(claim.text, candidate_evidence),
        })
    return results

def grounded_answer_rate(results):
    scored = [item for item in results if item["has_citation"]]
    if not scored:
        return 0.0
    return sum(item["supported"] for item in scored) / len(scored)

Split the generated answer into factual claims and require a source for each material claim. Verify that the cited source contains supporting evidence rather than merely sharing the topic. A citation can be valid and still fail to support the sentence. Preserve the supporting passage in the evaluation report so reviewers can inspect failures quickly.

For unanswerable questions, score whether the system refuses or asks for missing information without inventing a fact. For workflow answers, score the required action, constraint, and escalation path. For summaries, measure coverage of critical facts and absence of unsupported claims. Product-specific rubrics outperform generic helpfulness because they encode what success means for the user.

Human review remains necessary for ambiguous cases and judge calibration. Sample disagreements between automated graders and domain experts. Track false passes and false failures, then revise rubrics and examples. The evaluator is another model-backed production system and needs versioning, tests, and change control.

Calibrate graders with an error budget

Build a calibration set with clear passes, clear failures, and disputed boundary cases. Ask domain reviewers for labels before running the grader. Measure false-pass and false-fail rates by rubric and slice. A groundedness grader that misses unsupported policy claims is not acceptable merely because its overall agreement is high. Set a stricter error budget for critical claims and route uncertain results to human review.

Test grader sensitivity to answer length, citation count, writing style, and reference-answer wording. A grader should not reward verbosity or exact phrase overlap when the task allows concise alternatives. Run position swaps in pairwise comparisons and randomize candidate labels. Keep judge rationale for debugging, but compute release decisions from structured rubric fields and deterministic checks rather than parsing persuasive prose from the judge.

Evaluate language and locale as separate slices

Multilingual retrieval can fail in embedding, lexical matching, source availability, generation, or evaluation. Build cases with native questions and reviewed sources rather than translating only the English test set. Preserve locale-specific dates, units, product names, and policy terms. Use reviewers fluent in the target language to calibrate graders. If coverage is limited, make the supported-language boundary visible and route unsupported requests safely instead of relying on accidental model capability.

Run a fast, representative subset on pull requests that change retrieval code, prompts, model routes, parsers, or evaluation logic. Run the full suite on a schedule and before major releases. Cache stable corpus artifacts where safe so evaluation time is spent on changed behavior. Publish a report with baseline, candidate, slice-level deltas, failed case identifiers, traces, latency, and cost.

name: rag-evals
on:
  pull_request:
    paths:
      - "src/rag/**"
      - "evals/rag/**"
      - "prompts/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-eval.txt
      - run: python -m evals.rag.run --dataset evals/rag/golden.jsonl
      - run: python -m evals.rag.check_thresholds --report reports/rag.json

Define thresholds in policy, not inside an opaque dashboard. Critical slices may allow no regression. Broad quality metrics can use a small tolerance to account for evaluator variance. Latency and cost need budgets alongside quality so a change cannot achieve a modest score lift by multiplying candidate count and model calls. Require an explicit, time-bounded approval for exceptions, and record the person and reason.

Protect the evaluation itself. Pin dataset versions, model identifiers, judge prompts, sampling settings, and corpus snapshots. Record random seeds where supported. Mask sensitive production examples and restrict access to traces. A report that cannot be reproduced after a provider update is evidence of a trend, not a release-grade comparison.

Offline evaluation reduces risk but does not reproduce every production query, corpus state, permission edge, or provider response. Release the candidate to internal traffic, shadow traffic, or a small tenant cohort. Compare stage metrics and user outcomes against the baseline. Keep a route-level switch that can restore the previous retriever, prompt, model, and context policy without a code rewrite.

Monitor retrieval no-result rate, candidate overlap, source freshness, reranker distribution, grounded answer rate, escalation, user correction, latency, and cost per completed workflow. Break these down by release and slice. A drop in user escalation can be positive when answers improve, or negative when the interface hides uncertainty; pair product metrics with reviewed examples.

Turn production failures into dataset updates through a defined triage path. Confirm the expected source and answer with a domain owner, classify the failing stage, reproduce the trace, add a regression case, then fix. This converts a complaint into permanent system knowledge. Track recurring failure classes because repeated patches often signal an ingestion or product-design problem that requires a larger change.

End the rescue with ownership. Document the corpus, stage contracts, evaluation commands, thresholds, dashboards, alert owners, and dataset extension workflow. Pair with the engineer who will approve future changes. A repaired pipeline without an operating model will drift back into anecdotal tuning.

Write a quality-incident runbook

Define triggers for widespread unsupported answers, cross-tenant retrieval, stale policy use, citation corruption, source-ingestion gaps, and sudden eval regression. The first response should identify affected route versions, tenants, questions, source revisions, and time window. Operators need switches to disable a source, restore a previous index, bypass a reranker, tighten the answer boundary, or route traffic to human review without redeploying every service.

Preserve failed traces and corpus state under the approved data policy. Reproduce the case against the exact route, classify the first failing stage, and estimate exposure before tuning. Communicate product impact in workflow language: which questions could receive which wrong result, not only that groundedness fell. After repair, add regression cases, backfill affected outputs where appropriate, and review whether monitoring should have detected the issue earlier.

Track dataset and corpus drift together

A stable score can hide a changing product if the golden set remains frozen. Monitor the distribution of production intents, source types, context lengths, languages, no-result queries, and feedback. Compare that distribution with the evaluation set. Add reviewed cases when a slice grows or a new source enters the corpus. Retire cases only when the product behavior is removed, and preserve historical suites for route comparisons that still affect existing customers.

• Define retrieval, ranking, context, answer, and product failures separately
• Build reviewed cases from production-shaped questions
• Trace candidate identifiers and scores through every stage
• Verify ingestion coverage, parsing, versions, and deletion
• Measure recall before adding reranking or generation changes
• Derive tenant and permission filters from trusted state
• Check claim-level support, not citation presence alone
• Gate critical slices, latency, and cost in CI
• Canary with a documented rollback route
• Turn every confirmed production failure into a regression case

The essential discipline is simple: make the failure reproducible, identify the first stage that violates its contract, change that stage, and prove the product outcome improved. That process is slower than guessing for a day and faster than rebuilding the wrong system for a month.