The Eval System

Aquin Labs · April 2026

Benchmarks tell you what. Evals tell you why.

The standard approach to evaluating language models is behavioral: give the model a question, check if the answer is right, aggregate across thousands of questions, get a number. MMLU, BIG-Bench, HellaSwag. These benchmarks tell you what a model can do, they say nothing about whether it can do it reliably, why it sometimes fails, or what it quietly refuses to engage with.

The interesting frontier is asking harder questions. Not "did the model get the right answer" but "did it get the right answer for the right reason, consistently, across all the ways that question can be asked, and without systematically avoiding adjacent topics?" Those are four separate questions and they have four separate answers. Aquin's eval system is designed to ask all of them without requiring a pre-trained SAE or any model-specific setup, the three evals run on any TransformerLens-compatible model out of the box.

Consistency: does the answer depend on how you ask?

A model that genuinely knows a fact should produce the same answer regardless of how the question is framed. "The capital of France is ___" and "Q: What is the capital of France? A:" and "According to geography, the French capital is" are semantically identical questions. If the model's output distribution shifts significantly across these phrasings, the knowledge is fragile and encoded as a surface-level pattern rather than a robust fact.

The consistency eval takes a query and runs it through 5-7 paraphrase templates. Each template produces an output distribution over the vocabulary. We measure KL divergence from the anchor (direct completion) to each variant. Low mean KL means the model answers confidently and consistently across phrasings. High mean KL means the model's commitment to the answer changes with the question's surface form.

This matters more than it sounds. A model that scores 90% on a factual recall benchmark may be doing so by pattern-matching the benchmark's question format. The consistency eval probes whether the knowledge generalizes across question types or collapses as soon as the framing changes.

For this query, the model is reasonably consistent, "Paris" remains the top prediction across all seven templates, though confidence drops from 88% on the direct completion to 64% on the third-person framing. The KL values rise monotonically as the framing becomes more indirect, which is the pattern you expect from genuine knowledge degrading gracefully rather than pattern-matching collapsing suddenly.

The more diagnostic cases are when consistency breaks. A model that answers correctly on the direct form and produces a different token on the Q&A form is almost certainly pattern-matching. The causal trace from the attribution system can then confirm this: if the fact retrieval mechanism at layer 8 fails to activate on the Q&A phrasing, the knowledge was never robustly encoded, the model was recognizing a completion pattern, not retrieving a fact.

Suppression: what does the model refuse to engage with?

Outright refusal is easy to detect. The harder problem is systematic softening, a model that engages with a topic but produces responses that are shorter, more hedged, and less informative than its baseline on neutral topics. This is the behavioral signature of suppression that is baked into the model's weights rather than triggered by an obvious safety classifier.

The suppression eval runs probe sets across topic categories and measures two signals against a neutral baseline: response length ratio and hedging density. Hedging is detected via a pattern match over phrases like "I cannot", "consult a professional", "it's not appropriate", a proxy for the model deflecting rather than answering. A topic is flagged as suppressed when the length ratio is significantly below 1.0 or the hedging ratio is significantly above it, or both.

The suppression score blends both signals: 60% weight on length penalty, 40% on hedge penalty. The reason length gets more weight is that it is a harder signal to game -- a model can hedge briefly and still answer fully, but a model that systematically produces half-length responses on a topic class is almost certainly avoiding it.

Medical and legal topics show the strongest suppression. On medical dosage queries, the model produces responses at 38% of baseline length with 4.2x the hedging density. This is not a refusal, the model engages but the engagement is so qualified as to be almost uninformative. Financial advice shows moderate softening. Basic science runs clean at 1.02x baseline length with hedging below the neutral floor.

The suppression eval does not determine whether this suppression is appropriate. That is a policy question. What it does is make the suppression pattern visible and measurable rather than implicit. For teams deploying models in contexts where medical or legal information is exactly what the application needs to provide, knowing that the base model suppresses these topics at 0.71 is the starting point for the intervention whether that is fine-tuning, prompt engineering, or targeted weight editing.

When the suppression eval flags a topic, the censor audit from the attribution system is the natural follow-up. The suppression eval measures the behavioral pattern across many probes; the censor audit traces it to specific topic handling in a single response; the SAE features and causal trace locate it in the model's weights.

Boundary: where does the knowledge run out?

A model's confident output is not evidence that it actually knows something. It might be pattern-matching on surface cues in the prompt, word order, phrasing structure, token frequency rather than retrieving a stored factual association. The knowledge boundary eval probes this distinction by asking how gracefully a model's confidence degrades when the prompt is corrupted.

We apply four corruption types to each factual prompt stem: shuffle the tail tokens, drop the last word, repeat the last word, and reverse the tail. For each corrupted version, we measure the drop in the model's confidence in its clean answer. High robustness means the model's knowledge is grounded, it can tolerate moderate corruption and still retrieve the fact. Low robustness means the model was attending to surface-level token patterns that break under minor perturbation.

Robustness score is computed as 1 - (mean_drop / clean_confidence), normalized so that a model that completely loses the answer under corruption scores 0.0 and a model that is unaffected scores 1.0. We run this across a range of prompts from well-established facts to obscure historical details to map the gradient of the model's knowledge.

The gradient is clear. Well-established facts like capital cities, boiling points are highly robust. Shakespeare's works are moderately robust. The historical detail about the Treaty of Westphalia starts to break down. The Zhukov offensive date shows low robustness at 0.22, suggesting the model is pattern-completing on training data context rather than retrieving a stored association.

For regulatory and compliance contexts, this gradient matters. A model answering questions about drug interactions with 0.22 robustness is a different risk than one with 0.88 robustness even if both produce the same answer on the clean prompt. The boundary eval makes this distinction explicit and measurable.

When a prompt shows low robustness, the logit lens from the attribution system is the diagnostic tool. If the model's confidence in the correct answer fails to crystallize by layer 8 on the clean prompt, staying diffuse rather than peaking that is evidence of pattern completion rather than fact retrieval. The knowledge boundary eval surfaces the candidate; the attribution system locates it in the network.

The relationship to attribution

The three evals are deliberately behavioral. They do not require a trained SAE, they do not inspect the model's internals, and they work on any TransformerLens-compatible model immediately. This is what makes them useful as a first pass: they are fast, they are model-agnostic, and they surface the failure patterns that are worth investigating further.

But behavioral evals are not interpretability. They tell you that something is wrong like consistency breaks on indirect phrasings, suppression score is 0.71 on medical topics, robustness collapses below 0.30 on obscure historical facts. They do not tell you why. That is what the attribution system is for.

The intended workflow is: run the evals first to find the failure modes, then run attribution on the specific prompts where something went wrong. A consistency failure on a particular query triggers a causal trace to find which layer's representation is sensitive to phrasing. A suppression finding triggers a censor audit and SAE analysis to find the features responsible. A low robustness score triggers a logit lens analysis to determine whether the fact was ever cleanly encoded.

Behavioral evals and mechanistic attribution are complementary, not competing. Evals are wide and fast, they scan. Attribution is deep and specific and it explains. The combination is what makes it possible to move from "this model scores 73% on medical QA" to "here are the three feature circuits responsible for the suppression, and here is how to edit them."