TransformerLens Analysis

The primary metric is per-token head engagement: aggregate attention head activation normalized by token count. This controls for the length difference between variants — MESN™ uses ~14% fewer tokens than prose on average.

The effect is stable-to-increasing across scale. At 124M parameters, GPT-2 already shows +23.9% advantage. At 1.4B, Pythia amplifies this to +26.3%. The multi-specialization signal is robust at minimal scale and doesn't require large models to manifest.

Phase 1 uniquely tests three input variants — prose, MESN™, and Python-style pseudocode. This three-way comparison was dropped in Phase 2 for cleaner statistical testing, but it produced a critical early insight:

• Code activates code_syntactic heads strongly but semantic_conceptual heads weakly
• Prose activates semantic_conceptual heads strongly but code_syntactic heads weakly
• MESN™ activates both simultaneously — code_syntactic AND semantic_conceptual heads

This was the first empirical evidence for the multi-specialization hypothesis: MESN™ uniquely bridges the specialization gap between “language” heads and “code” heads, engaging both in the same expression.

Beyond raw magnitude, the early experiments measured the shape of the activation distribution — how engagement is spread across heads:

MESN™ doesn't just activate heads more strongly — it activates them moreevenly. Prose creates a few highly-active heads and many dormant ones. MESN™ creates a broader, more uniform engagement pattern. This observation directly motivated the cross-family diversity hypothesis in the 43-model study.

Per-family activation analysis across the 4 models with surviving data reveals two distinct clusters — not a smooth gradient:

• Cluster 1 (7B–20B): Mistral, Pythia-12B, GPT-NeoX-20B — consistent ~27–33% advantage across all 8 families
• Cluster 2 (32B): Qwen 2.5 32B — dramatically higher ~46–50% advantage, roughly 1.6x the other models

The jump is not gradual. It's a step-change that may reflect Qwen's larger scale, its GQA architecture, its 152K vocabulary (3x larger than the other models), or training data composition. 32 of 32 family-direction checks are positive.

Head specialization concentrates in the final 25% of network depth across all architectures. Early layers show near-zero specialization scores; late layers carry the highest-scoring specialized heads.

Qwen 2.5 32B shows the most extreme late-layer concentration: its early layers (0.007) are nearly inert while late layers (0.035) carry 5x the specialization. All of Qwen's top-10 most specialized heads fall in the final 10% of the network (layers 57–63 of 64).

Not all semantic categories benefit equally. The advantage follows a consistent hierarchy across all 4 models:

Bidirectional, metaphor, and transformation categories — where MESN™ encodes mutual influence, analogy, and state change — show the largest advantages (49–60%). Causal, complex, and definition categories show more modest gains. Only one negative entry exists in the entire 32-cell matrix: Mistral 7B on causal at −3.5%.

TransformerLens provides zero-friction DLA — z @ W_O @ W_U is computable directly from built-in weight matrices. A single run_with_cache() call captures all intermediate activations. This made it ideal for iterative proof-of-concept work.

At the Mistral-7B scale, TransformerLens hit limitations: ~30% memory overhead from internal format conversion, GQA head expansion that changes activation patterns, and no support for newer architectures. The migration to nnsight addressed all of these while preserving the core DLA methodology. Cross-validation on Mistral-7B confirmed both tools produce the same directional results.