The Benchmark Builder

Aquin Labs · April 2026

Benchmarking should not be a separate job

Standard benchmark workflows look like this: choose a suite, configure a harness, run the evaluation, wait for results, parse outputs, visualize separately. For a scheduled evaluation of a production checkpoint, that pipeline is reasonable. For an in-session question that comes up while you are mid-inspection — you are looking at a suspicious feature, or the model just gave a bad answer and you want to know how bad — it is a full context switch that typically does not happen. The question gets dropped.

The Benchmark Builder removes the context switch. It is a tool inside each of the three agent chats: model inspection, data inspection, and training monitor. You describe what you want to measure in plain language. The agent writes the prompt suite, runs it against whatever is currently loaded in that session, and returns a scored card directly in the chat thread. The benchmark is grounded in the same session context as the question that prompted it, and it lives next to the conversation that produced it.

The card is interactive. Chart type is switchable between bar, pie, radial, and line. The card expands into a full dialog showing the complete capability breakdown. Results export in four formats. And because the card is in the chat, the reasoning that led to the benchmark — the conversation, the prior findings, the specific question being asked — is preserved alongside the result.

The three contexts

Benchmark Builder is available in all three agent chats. Each one has access to different state at send time, so the same request sent to different chats produces different results. The context a card was generated in is recorded in its metadata and included in all exports.

The most common pattern is model inspection: you are looking at a feature, running a causal trace, and a question about reasoning capability or refusal behavior comes up naturally. Sending the benchmark request in that chat means the card is grounded in the same model and the same session state. You do not need to navigate elsewhere or re-specify the model.

The training monitor context is the least obvious but often the most useful. When a training run completes and the model diff shows a behavioral regression, a benchmark request inside the training monitor benchmarks the specific checkpoint at issue. The score is indexed against the training step and can be tracked in the regression panel alongside behavioral diffs from prior runs. This connects the capability measure directly to the training run that produced it.

How a benchmark is built

When you describe a benchmark, the agent identifies capability dimensions in the request and generates a prompt suite for each one. The suite size is determined by the task complexity: factual recall probes with unambiguous ground truth get fewer prompts; multi-step reasoning tasks with partial-credit scoring get more. You can override both in your request: asking for a quick benchmark produces a smaller suite with faster turnaround, specifying a prompt count gives you exactly that many.

Prompts are generated fresh on every run. Two requests for the same benchmark produce equivalent but not identical prompt sets. This prevents scores from being artifacts of a specific phrasing and means repeated runs give a distribution rather than a single point. For direct A/B comparison, export both runs as JSON and compare the capability rank order, which is more stable across prompt variation than the absolute scores.

There is no fixed set of capability dimensions. You describe what you care about and the agent maps it to dimensions it can score. Common requests resolve to well-defined dimensions like those below. Less common requests — dataset coverage of a specific regulatory category, model consistency on a particular topic area — produce custom dimensions that the agent defines before scoring. The definition is shown in the card summary alongside the score.

Scoring methods

Each capability dimension gets a score between 0 and 100. The scoring method depends on the task type. The agent selects the appropriate method automatically and notes it in the card summary. Understanding which method was used matters when you compare scores across different benchmarks: a 67% on chain-of-thought math with partial-credit scoring is not directly comparable to a 67% on factual recall with next-token probability.

The six capability dimensions below are the most commonly requested. They are not a fixed taxonomy. They emerge from the distribution of requests, and any of them can be refined in your request: "factuality on medical claims only", "reasoning on spatial problems", "refusal on political topics". The more specific the dimension, the more targeted the prompt suite and the more interpretable the score.

The inline card

When the benchmark finishes, the agent appends a card to its message in the chat thread. The card shows the benchmark title, context label, prompt count, and a scored bar for each capability. It is sized to sit naturally in the conversation without dominating it: compact enough to scroll past, detailed enough to read at a glance.

The card is immediately interactive. The chart type switcher in the top right changes the visualization without a round trip. The expand button opens a full-screen dialog with the complete capability breakdown, a larger chart, and the scoring method and prompt count for each dimension. Export buttons live in the card footer and are available without opening the dialog.

The overall score in the footer is the mean across all capability dimensions, weighted equally by default. If some capabilities matter more than others for your use case, specify the weighting in your request and the agent will note it in the card summary. The overall score will reflect the custom weighting.

Chart types

Four chart types are available on every card. They render the same underlying scores, so switching between them is purely a display choice. The active chart type is remembered per card and carries over when you open the expand dialog.

Line chart is a special case: it is designed for comparison across multiple benchmark runs rather than a single result. If you run the same benchmark twice at different training steps, opening both cards and switching to line view shows the score trajectory across the two runs. This works best when the capability dimensions are the same across runs, which they will be if you use the same request text.

Export formats

All four export formats are available from the card footer in both the inline view and the expanded dialog. Every export includes the benchmark metadata: title, context, model or dataset identifier, prompt count per capability, scoring method per capability, and timestamp.

JSON is the most complete format for downstream use. The per-prompt scores let you identify exactly which prompts failed within a capability, which is often more informative than the aggregate score. A reasoning score of 67% means 8 of 12 multi-step prompts were answered correctly. The JSON output tells you which 4 failed.

Reading the results

Scores produced by the Benchmark Builder are relative to the prompt suite generated for your specific request. They are not directly comparable to published benchmark scores unless you explicitly request a standardized benchmark by name. A reasoning score of 82% means 82% of the generated multi-step reasoning prompts were answered correctly by the model in that session. It is not a claim about the model's score on a specific external leaderboard.

The most reliable use of these scores is comparison within a session. Run the same benchmark request against two models, two checkpoints, or two dataset variants. Because prompts are regenerated each time, absolute scores carry run-to-run noise at the level of a few percentage points. What is stable is the rank order: if model A scores higher than model B on reasoning across three separate runs, that ordering is reliable even if the absolute scores shift slightly between runs.

A low score on a capability is not a verdict. It is a starting point. The agent chat is still open after the card is returned. The natural follow-up is to drill into the failing capability: ask which prompts failed, ask why the model is getting them wrong, or request a more targeted benchmark focused on the specific sub-type that is failing. A reasoning score of 67% might be driven entirely by spatial reasoning problems while arithmetic scores perfectly. A follow-up benchmark on spatial reasoning alone tells you that in one additional request.

When you see a capability gap this clearly, the investigation path is direct. Low math, high code, in a model that has seen substantial code training, suggests the math deficit is likely in the reasoning chain rather than the arithmetic operations themselves. A follow-up benchmark on chain-of-thought math specifically, with the agent instructed to log the full reasoning trace for each prompt, will confirm or rule out that hypothesis in the same session. The Benchmark Builder does not close that loop automatically. It gives you the signal; the conversation continues from there.