We start by turning vague reports of hallucination into a failure taxonomy that can be measured. A wrong answer may originate in ingestion, parsing, chunk boundaries, metadata, query construction, filtering, lexical or vector search, reranking, context assembly, model instructions, or citation rendering. Treating all of these as an answer-model problem leads to expensive experimentation and hides the component that actually needs attention.
AI Reliability Engineering
Find why your RAG system gives the wrong answer—and prove the repair.
RAG consulting services for production systems with wrong answers or weak citations. Isolate retrieval failures, repair the pipeline, and add CI evals.
Book a 30-minute consultation
30 minutes. No deck. Leave with a clear next step.
System-level interventionProduction-shaped diagnosisInspectable verification
The production trigger
Recognize the failure before naming the service.
Isolate the failing RAG stage and put measured behavior in CI.
The pipeline returns plausible answers, but the wrong chunk, stale document, missing citation, or weak context quietly changes the result. The failure only becomes visible when a customer catches it.
Trigger detail 1We build the golden dataset first, isolate each stage of the pipeline, repair the highest-impact bottleneck, and leave a CI gate that measures future changes.
Trigger detail 2System diagnosis
Find where the failure actually lives.
The visible symptom is rarely the whole problem. The first pass follows it through the production path until the controllable boundary is clear.
Representative questions matter more than a polished demo set. We sample production traces, support conversations, low-confidence cases, empty-result paths, and questions that require joining information across documents. Domain reviewers identify what a sufficient answer must contain and which sources are authoritative. The resulting golden dataset includes straightforward successes, boundary cases, known failures, and negative questions where the correct behavior is to abstain or ask for clarification.
The diagnosis also records operational constraints: index size and refresh frequency, tenant filters, access control, document languages, latency budget, data residency, and the team's ability to maintain the pipeline. A proposed reranker that improves an offline metric but breaks response time or tenant isolation is not a repair. This service fits a production RAG system with accessible traces and reviewers; it is not designed to invent a knowledge product before real user behavior exists.
The intervention
Change the critical path, not the surrounding theatre.
Each workstream targets a different failure boundary. Together they connect product behavior, infrastructure, controls, and verification.
Dataset and acceptance model
Build versioned cases with question intent, authoritative sources, expected facts, acceptable variants, citation requirements, and abstention behavior. Cases are tagged by document type, tenant, recency, language, complexity, and known failure mode. We agree which slices are release-critical so a single aggregate score cannot conceal poor behavior for an important workflow.
Stage-level retrieval analysis
Measure whether the required source entered the index, survived parsing, was split coherently, matched filters, appeared in candidate results, survived reranking, and reached the model context. Recall and ranking metrics are paired with trace inspection. This isolates retrieval quality from answer quality and reveals where configuration, metadata, or source hygiene matters more than a new model.
Targeted pipeline repair
Change only the stages supported by evidence. Work may include parser fixes, parent-child chunking, metadata normalization, hybrid search, query decomposition, reranking, contextual compression, source prioritization, or answer instructions. Each candidate runs against the same dataset and operational constraints, with before-and-after results retained so the team can understand the trade rather than inherit unexplained tuning.
Regression and production feedback
Connect the harness to CI and define how production failures become new cases. Pull requests report changed scores and failed examples, while runtime traces retain enough retrieval context to diagnose future incidents. The workflow includes review rules for updating expected answers when source material changes, preventing a stale golden set from rejecting correct behavior.
Inspectable changes
See what changes in the system.
- Golden dataset built from production questions
- Stage-by-stage retrieval and answer baseline
- Chunking, metadata, query, and reranking diagnosis
- Targeted pipeline changes with measured deltas
- Citation and groundedness checks
- Eval dashboard and failure slices
- CI gate for retrieval regressions
Engineering judgment
The decisions that determine whether the change holds.
Metric and judge selection
Choose deterministic retrieval metrics where ground truth exists, rubric-based grading where language varies, and human review for high-impact ambiguity. We avoid relying on one model judge for every decision. Judges are calibrated against reviewed examples, required to explain failures, and monitored for disagreement so their score is evidence rather than an unquestioned verdict.
Retrieval architecture changes
Decide whether a failure warrants configuration tuning, an additional retrieval stage, a different index, or no architectural change. Complexity has an operating cost. Hybrid search, rerankers, query expansion, and graph techniques are adopted only when they improve relevant slices under latency, access-control, refresh, and maintenance constraints.
Abstention and fallback behavior
Define what the product does when evidence is missing, conflicting, stale, or inaccessible. Options include a qualified answer, clarification request, source-only response, escalation, or refusal. This decision is part of product reliability, not just prompt wording, and is tested alongside successful answers so the system fails in a deliberate way.
Strong fit
- You have RAG in production with real user questions.
- Failures are plausible enough to escape normal monitoring.
Probably not the right fit
- You have not shipped RAG to real users yet.
- You want a demo that looks good instead of a system that measures well.
Scope, timing, and ownership
Know the commercial shape before the call.
A bounded engagement with a service-specific delivery sequence, an agreed price, and artifacts that stay under your control.
Days 1-2
Golden dataset
Turn production questions and failures into measurable retrieval and answer cases.
Days 3-5
Pipeline diagnosis
Measure ingestion, chunking, search, reranking, context assembly, and citations separately.
Days 6-8
Targeted repair
Fix the bottleneck without replacing parts that already work.
Days 9-10
CI and handoff
Add thresholds, failure reports, and a documented process for extending the dataset.
Typical fixed scope
Mid-four to low-five figures
Most engagements fall in this planning range. Your exact fixed price is agreed in writing after we review the production surface, access needs, and success criteria. The range is guidance, not a quote.
Commercial terms
No open-ended consulting meter.
- Fixed scope agreed in writing
- Fixed timeline agreed in writing
- Fixed price agreed before work starts
- Client-owned deliverables from day one
- No hourly meter or surprise overages
Client ownership
The implementation remains yours.
- Your repository
- Your infrastructure
- Your evals and evidence
- Yours from commit one
Book a 30-minute consultation
30 minutes. No deck. Leave with a clear next step.
RAG Rescue
Questions worth settling before the call
Technical, commercial, and handoff questions answered before you book.
Bring the failed questions. Leave with a measured repair.
Bring the failing workflow, current evidence, and the constraints your engineers cannot ignore.
Book a 30-minute consultation
30 minutes. No deck. Leave with a clear next step.