We measure how often your LLM feature, RAG system or agent gets things wrong, using your own traffic, and tell you what to fix first. It is for teams with an LLM in production or about to ship one. From $4,000 fixed price, in two weeks, and you keep the evaluation harness.
Book the audit How it worksUpdated 18 November 2026
We start by building an evaluation set from your own traffic or logs, because generic benchmarks say little about your users' questions. Then we run your system against it and count.
The rate of unsupported claims: statements in an answer that the source documents or the tool results do not back up. Each rate comes with real examples, so you can see what a failure looks like.
Whether answers are correct, complete and in the format your product needs, scored against a reference set we agree with you and checked by hand on a sample.
Whether the right passages are retrieved at all, and whether the model uses them. Many hallucinations in RAG systems start as retrieval misses rather than model errors.
Failure modes in multi-step runs: wrong tool, wrong arguments, loops, steps skipped, and answers that ignore what a tool returned.
Time and spend per answer, measured on the same evaluation set, so every fix is judged against what it costs you to run.
A golden dataset and automated checks your team can run in CI, so the next prompt, model or retrieval change is tested before it ships.
Four things, all yours to keep.
The measured rates with examples of each failure type, written so an engineer and a product owner can both act on it.
A list ranked by impact and effort: what to change in prompts, retrieval, guardrails or model choice, and in what order.
The code and the golden dataset, handed over and documented, with checks that run in your CI.
After the fixes, we run the same evaluation again, so you see the before and after on identical data.
For the tooling, we use our own open-source tool Groundcheck (GitHub organisation protforge-ai), which scores an answer against its sources sentence by sentence. See our comparison of hallucination detectors for how we evaluate the detectors themselves.
We build the evaluation set from your traffic, run your system against it and measure hallucination, quality, retrieval, agent failures, latency and cost.
We write the report and the prioritised fixes, hand over the harness and golden dataset, and re-measure once the first fixes are in.
What we need from you: a sample of real traffic (a few hundred requests, redacted if you prefer), access to a staging endpoint or to your logs, and one technical contact who can answer questions about the system.
Fixed price, no hourly billing.
Evaluation set from your traffic, measured rates, prioritised fixes, the harness handed over and a re-measurement. Two weeks.
The final fixed price depends on the number of systems and how many agent steps or data sources are involved. We confirm it after a 30-minute call, before any work starts. See all services.
The audit is a fixed price from $4,000. After a 30-minute call about your system we confirm the scope and the final fixed price, so there are no hourly surprises. The price covers the report, the evaluation harness and the re-measurement after fixes.
Two weeks from the day we have your traffic sample and access. The first week is measurement, the second is the fix plan, the harness handover and the re-measurement.
We need a sample of real inputs, because that is what makes the evaluation set representative. A few hundred logged requests is usually enough, and they can be anonymised or redacted before they reach us. Access to a staging endpoint is enough to run the system; we do not need write access to production.
Any. We work with OpenAI and Anthropic models and with open-weight models, and with systems built on LangChain, LangGraph, LlamaIndex or custom code. We evaluate the behaviour of your system through its inputs and outputs, so the stack underneath matters little.
You keep the report and the harness, and your team can run the checks in CI from then on. If you want help applying the fixes, we can scope that separately, for example as a fractional AI lead one or two days a week. There is no obligation to continue.
Tell us what your system does and which model and stack it runs on. We reply within one business day and propose a 30-minute call.
hello@protforge.comWe take on few clients at a time so each project gets direct attention.