Public AI benchmarks are a useful shortlist filter and a poor selection tool. They narrow twenty models to four, and choosing among the four takes your own data.
We build the test set for your problem, score it against the decision it drives, and hand the suite over.
What Public AI Benchmarks Measure
General capability, against your specific problem. A model that reasons well about graduate physics is not thereby good at reading your claim forms.
Averages, where you are one point. A mean hides the tail, and the tail is where systems fail publicly.
Capability without cost. Two points of accuracy at eleven times the price per call is a poor trade at volume, and no leaderboard prices that for you.
Treat published ai benchmarks as the filter, then measure the four survivors on your own material. That is a week of work and it settles the choice properly.
What We Build Instead
A test set from your cases. Fifty to three hundred real examples with an agreed correct answer for each, written by somebody who knows the work. Assembling this is most of the effort and it cannot be purchased.
A score named for the decision. Where the system routes a ticket, the number is how often it routes wrong and what that costs.
A test suite in your repository. A script that runs the whole set and prints the result, so anyone on your team can check after a prompt edit or a model upgrade.
A cost-per-decision figure beside the score. Measured against your real traffic rather than estimated from a price list, including the picture at ten times volume.
Why Evaluation Comes First
Measuring before building means the number is information rather than unwelcome news about something that already exists.
It tells you whether retrieval or the prompt is at fault, whether a cheaper model would serve, and where the remaining effort should go. Those answers regularly save more than the work costs.
The engineer who builds your test set builds the system it measures, which is why the scoring matches what the system actually does.
What a Test Suite Catches
Silent regression, when a prompt is edited to fix one complaint and three other behaviors move unnoticed. Provider updates, when a model changes underneath a stable version string. Input drift, when the documents change and retrieval quietly starts missing.
All three surface identically once a suite exists: the number moved, on a date you can point at.
What It Costs
One to two weeks for a first evaluation of an existing system, most of it spent building the test set with your people rather than writing code. Shorter as the opening phase of a new build.
The suite is yours and it runs without us. Tell us what the system is supposed to get right and we will tell you how to know whether it does.
Related Services
Retrieval is scored on whether the cited passage actually supports the answer, which is RAG development.
Agents are scored on whether the tool call was the right one, which is AI agents. Classical models scored against a holdout are machine learning.