Benchmarks

Aquin Labs · April 2026

Part I: Feature Benchmarks

Llama 3.2 1B Instruct · SAE layer 8 · 16,384 features

What makes a feature real?

We trained a sparse autoencoder on layer 8 of Llama 3.2 1B Instruct with 16,384 features. Once the SAE was trained, the immediate problem was working out which of those features were worth trusting. A feature vector is just a direction in a high-dimensional space. Before using one for anything downstream, several distinct things need to be true about it. Its label should predict where it fires. It should fire for one coherent concept rather than many unrelated ones. And when it fires, the model should actually use it.

These are not the same question. A feature can be well-labeled but polysemantic. It can be monosemantic but ignored downstream. It can causally matter to predictions but carry a label that misses the point entirely. We evaluate these dimensions independently with three benchmarks, each surfaced as a live panel inside the tool when you select a feature.

The three feature benchmarks

InterpScore: does the label predict firing?

For each feature in our SAE, we generate two sentence sets using an isolated model: sentences where the feature should fire based on its label, and sentences where it should not. We run each sentence through Llama 3.2 1B Instruct, extract the feature's maximum activation across all token positions at layer 8, and compute Cohen's d between the two activation distributions.

Cohen's d is the difference in means divided by the pooled standard deviation. We clip to [0, 1] for display. A high InterpScore means the label is doing real predictive work. We run 10 positive and 10 negative sentences per feature by default, each as its own forward pass through the full model and SAE.

The panel surfaces the highest-activating examples from each set alongside the score, so you can read what the feature is actually responding to rather than trusting the label alone. This turned out to matter: several features in our SAE had accurate labels that nonetheless undersold the breadth of what the feature was firing on.

A low InterpScore does not always mean the label is wrong. It can mean the label is too abstract to generate useful contrasting sentences, or that the feature fires broadly enough that any reasonable label undershoots it. That is where FeaturePurityScore comes in.

FeaturePurityScore: is it about one thing?

InterpScore evaluates the label. FeaturePurityScore evaluates the feature itself, independently of its label. We take the positive examples from the InterpScore run, the sentences where the feature actually fired above threshold, and embed them. We compute mean pairwise cosine similarity of the resulting embedding matrix, keeping only the upper triangle to exclude self-similarity. Cosine similarity lives in [-1, 1]; we remap to [0, 1] for display.

High purity means the activating contexts cluster tightly in embedding space: the feature is monosemantic. Low purity means the contexts are scattered across concepts. Running purity across our 16,384-feature SAE surfaced a meaningful tail of polysemantic features, concentrated in features trained near the sparsity penalty boundary, consistent with what the mechanistic interpretability literature predicts.

A low purity score is an early warning before running weight edits or causal ablations: if a feature is polysemantic, targeting it is likely to produce unexpected side effects in circuits unrelated to the intended subject.

Model Utilization Index: does the model use it?

A feature can score well on both InterpScore and FeaturePurityScore and still be functionally irrelevant. We found this in practice: some of the most cleanly labeled, monosemantic features in our SAE produced near-zero KL divergence when ablated. The model was computing them but not routing through them downstream.

MUI measures this directly. For each token position where the feature fires above a small threshold, we zero out that feature's contribution to the residual stream and run the ablated forward pass. We compute KL divergence between the baseline and ablated output distributions at that position, then take the mean across all firing positions, normalized by the baseline Shannon entropy of the output distribution.

High MUI features that are causally load-bearing but poorly labeled became the first targets for manual relabeling in our review pass. Low MUI features, even well-labeled ones, are deprioritized: they fire but carry little causal weight in any given inference.

Reading the feature scores together

Running all three benchmarks across our feature set, the most common pattern we encountered was the fourth row below: high purity and high MUI with low InterpScore. The feature is coherent and causally important, but the auto-generated label missed the concept. A second prompt pass with inspected activating examples was usually enough to resolve it.

The fifth row, low across the board, represents dead features. These showed up more often than expected in the high-sparsity-penalty region of our training sweep. The panel surfaces them as candidates for filtering before any downstream analysis.

Part II: Edit Benchmarks

Pythia 2.8B · ROME-style rank-one updates · 13 benchmarks

An edit that passes validation is not necessarily a good edit

The weight editing pipeline's validation loop checks three things before committing an edit: does it hold across paraphrases, does the model's behavioral output stay stable, and does the residual stream's representational structure stay intact. Those checks are necessary; an edit that fails any of them is rolled back immediately. But they are not sufficient.

Running edits on Pythia 2.8B, we found that edits could pass all three validation checks and still be shallow, poorly generalized, or fragile when subsequent edits were applied. We built thirteen benchmarks to surface these failure modes. They run after a committed edit and characterize its quality across independent dimensions.

The thirteen benchmarks

All benchmarks use dynamic triple generation: given the subject, relation, and new target of the edit, a generator produces test cases appropriate for each benchmark's purpose. This means the suite adapts to the specific content of each edit rather than relying on a fixed test set.

Dynamic triple generation

Each edit touches a different subject-relation-target triple, so static test sets are not useful here. For each benchmark, the pipeline generates 6 to 8 test triples appropriate to that benchmark's purpose. Each triple specifies a prompt, a target, and whether the model should (is_positive=true) or should not (is_positive=false) produce that target.

The sequential benchmarks

Three benchmarks apply additional edits to the model beyond the one being evaluated. These emerged from a pattern we observed early in testing: edits that looked clean in isolation started degrading when the model received subsequent edits at nearby layers. We built these benchmarks specifically to quantify that interference.

SequentialCollapseThreshold applies several synthetic edits sequentially and measures cumulative KL divergence from behavioral baselines after each one. The collapse threshold is KL greater than 0.3. The benchmark reports both the score (fraction of edits without collapse) and the specific edit index where collapse occurred.

SequentialEditRetention checks whether the original edit specifically survives as subsequent edits are applied on unrelated subjects. An edit stored in a direction shared with other facts in weight space shows declining probability for the original target as subsequent edits overwrite it.

BatchEditConsistency tests MEMIT-style concurrent editing: the original edit and several synthetic edits applied in a single session, then all probed simultaneously. This catches mutual interference that doesn't show up in sequential evaluation. All synthetic edits are rolled back after the benchmark.

Long-form generation quality

Most weight editing benchmarks probe next-token probability on cloze-style completions. We found this left a gap: edits that scored well on direct probes occasionally failed to show up in free-form generation about the same subject. LEME (Long-form Edit Metric) was built to catch this.

The model is prompted to generate a short paragraph about the subject. That paragraph is evaluated on three axes: factual consistency with the edit, fluency and coherence, and absence of contradictions. The final LEME score is the mean of the three. An edit that succeeds on cloze probes but fails LEME has changed what the model says in response to a direct question, but not how it writes about the subject when given latitude.

LocalitySensitivityScore complements this by measuring KL divergence on cross-domain facts after the edit, using exp(-mean_kl) as the score. Unlike RippleBench which tests same-subject facts, this catches whether the edit disturbed facts in entirely different domains that should be unrelated to the edit target.

Portability and indirect recovery

PortabilitySubMetrics breaks edit transfer into three sub-axes. Template variation tests whether the edit holds across different sentence structures: active, passive, question, fill-in-the-blank. Alias variation tests whether the edit holds when the subject is referred to by a nickname, abbreviation, or description. Compositional variation tests whether the edit holds when the fact is embedded as a sub-clause of a longer sentence. The final score is the mean across all three.

IndirectFactRecovery tests multi-hop inference. Given that the edit says "The CEO of Acme is Alice", can the model answer "Who leads Acme?" or "What company does Alice run?" These require inferring from the edit rather than directly recalling it. The pass threshold is intentionally low (0.35) because these are genuinely hard probes.

PreservationVsMemorization (PM objective) directly measures the gap between surface memorization and genuine generalization. It runs two probe sets: nearly-identical surface probes and strongly paraphrased probes. The PM score is the ratio of paraphrase performance to surface performance. A ratio near 1.0 means the edit generalized; a ratio near 0.0 means only the surface form was memorized.

zsRE mirrors the evaluation protocol from the ROME and MEMIT papers. Probes are in "Q: [question] A:" format and the model must produce the edited target as the most probable next token, making results directly comparable to published work on factual editing in transformers.

Reading the edit scores together

EditBench, EditGeneralization, and RippleBench form the core quality triangle. The most common failure pattern we encountered was high EditBench with low EditGeneralization: surface memorization. The rank-one update wrote the new target strongly enough for the exact training prompt, but the key vector didn't generalize to other surface forms. The PM objective benchmark gives a direct score for this; a low PM ratio confirms the diagnosis.

When LEME fails while EditBench passes, the edit is holding in cloze probes but not showing up in generation. When IndirectFactRecovery fails while EditGeneralization passes, the model can paraphrase the edited fact but cannot use it as a premise for further inference. These patterns point to different root causes and inform which follow-up experiments to run next.

Running the benchmarks

Benchmarks run from the Benchmarks tab in the Weight Editor panel, available after a successful edit. The panel shows the subject, relation, and target of the most recent successful edit and a progress bar as results stream in one by one.

The full suite runs end-to-end in roughly 3 to 6 minutes depending on model response latency. Each benchmark card is expandable: clicking it reveals the per-probe breakdown. PortabilitySubMetrics groups its details by sub-axis (template, alias, compositional) so you can see exactly where portability fails.

The benchmarks are informative rather than gatekeeping. Failing a benchmark does not roll back the edit; that is the validation loop's job. The benchmarks run after a committed edit and give you a characterization of its quality. An edit that passes validation but fails EditGeneralization is still live in the model's weights. The benchmark result tells you how much to trust it.