We measure how often your RAG system or AI agent retrieves the wrong passage, makes claims its sources do not support, cites the wrong source or calls the wrong tool, using your own traffic. Below is how to do it yourself in five steps; if you would rather have it done, it is a fixed-price audit from $4,000 in two weeks.
Book the evaluation How to evaluate your RAGUpdated: 21 Nov 2026
A RAG answer can be wrong for several unrelated reasons, and each one has a different fix. Measuring them separately is most of the work.
Does the retriever return the passage that contains the answer, and how much irrelevant text comes with it? We measure this on a gold set of questions with the correct source passages marked. Many answers that look like model hallucinations are retrieval misses.
Is every claim in the answer supported by the retrieved context? An answer can be correct in the real world and still ungrounded, which matters if your users rely on your documents as the source of truth.
When the answer cites a source, does that source actually say what is claimed? Citations that point to the wrong passage look reassuring and are worse than none.
Does the answer cover everything the question needs, and does the system say it does not know when the documents do not contain the answer? Both over-answering and over-refusing are failures.
For agents: wrong tool, wrong or missing arguments, loops, skipped steps, no recovery after a tool error, and final answers that ignore what a tool returned.
The golden set and checks run on every prompt, model or retrieval change, so a fix in one place does not quietly break another.
This works with or without us. Do it by hand first; automate once you trust the labels.
Pull 50 to 100 questions from logs, support tickets or search queries rather than inventing them. For each one, have a person write the expected answer and mark the passage that supports it. Include questions the documents cannot answer, so you can test refusals.
Before looking at any generated text, check recall at k (is the right passage in the top k results?) and precision. If recall is low, no prompt change will fix the answers. Chunking, embeddings, filters and query rewriting come first.
Check groundedness against the retrieved context, correctness against the gold answer, citation accuracy and completeness. Count failures per category instead of using one overall score, so you know what to fix.
Off-the-shelf LLM-as-judge prompts disagree with people more often than teams expect, and they drift when you change the judge model. Have humans label a sample, compute agreement per criterion, and use the judge only where it agrees. Re-check when the judge model changes.
Every failure you find becomes a test case. Run the suite in CI on each change and track the rates over time, plus latency and cost per answer, so you see what an improvement costs.
For checking groundedness automatically we use our open-source package groundcheck, and we publish a comparison of hallucination detectors so you can see how the detectors themselves perform before you rely on one.
Our LLM evaluation and hallucination audit follows the same method on your own traffic. It is for teams with a RAG system or agent in production, or about to ship one, and for startups that need to know how often their feature is wrong before a customer finds out. Our background is PhD-level research on hallucination detection in LLMs, so measurement comes before promises.
Evaluation set from your traffic, measured retrieval, groundedness, citation and agent failure rates, prioritised fixes, the harness handed over and a re-measurement. Two weeks, fixed price.
The final price depends on the number of systems and on how many data sources or agent steps are involved. We confirm it after a 30-minute call, before any work starts. We work remotely with US teams and overlap with US East Coast mornings. See all services.
Our LLM evaluation audit, which covers RAG systems and agents, is a fixed price from $4,000 and takes two weeks. After a 30-minute call about your system we confirm the scope and the final fixed price, so there is no hourly billing.
Start with 50 to 100 real questions with the expected answer and the source passage marked by a person. That is enough to see the big failure modes and to compare two configurations roughly. Grow it to a few hundred as you find new failure types, and always add the failures you see in production.
You can, but not without checking it. An LLM judge has its own biases, such as favouring longer answers, and its scores shift with the judge model and the prompt. Label a sample by hand, measure how often the judge agrees with the humans, and only trust the judge on the criteria where agreement is high. We do this calibration as part of the audit.
We look at the whole run, not only the final answer: whether the right tool was called with the right arguments, whether the agent loops or skips steps, whether it recovers from tool errors and whether the final answer matches what the tools returned. Each of these becomes a test case in a regression suite.
Yes. We work remotely from Europe and overlap with US East Coast mornings for calls and reviews, with written updates for the rest. Teams further west usually agree one or two fixed call slots a week.
A report with measured rates and real examples of each failure, a ranked list of fixes, the evaluation harness and golden dataset to run in your own CI, and a re-measurement after the first fixes. You keep everything and there is no obligation to continue.
Tell us what the system does, which model and stack it runs on, and what you are worried about. We reply within one business day and propose a 30-minute call. We work remotely with US teams and overlap with US East Coast mornings.
hello@protforge.com