attention-sink-structural

---
name: attention-sink-structural
description: Mechanistic explanation of attention sink phenomenon. Variance discrepancy amplified by FFN super neurons. Head-wise RMSNorm fixes it. Based on arXiv 2605.06611.
category: transformer-attention
---

# Attention Sink: Structural Origin and Control

## Overview
Attention sinks (initial tokens monopolizing attention scores) emerge from a causal chain: value aggregation → variance discrepancy → FFN super neuron amplification → dimension disparity → sink formation.

## Causal Chain
1. **Value aggregation** in self-attention induces systematic variance discrepancy
2. **FFN super neurons** (channel-sparse down-projections) amplify this discrepancy
3. **Dimension disparity** of first-token representation emerges
4. **Attention sinks form** as structural anchors

## Key Interventions
- **Isolate aggregation**: Attention mask modifications replicate sinks at arbitrary positions
- **Amplify variance**: Targeted variance amplification creates sinks on demand
- **Head-wise RMSNorm**: Normalizes attention head outputs independently, restores statistical parity, accelerates convergence

## Key Findings
- Sinks are structural, not accidental
- Channel-sparse FFN down-projections are the amplification mechanism
- Can be created at arbitrary positions through controlled interventions
- Head-wise RMSNorm significantly accelerates pre-training

## Implementation Steps
1. Monitor per-token variance in value aggregation outputs
2. Identify FFN super neurons (channel-sparse high-activation units)
3. Apply head-wise RMSNorm to normalize attention head outputs
4. Verify statistical parity across token positions

## Applicable Use Cases
- Pre-training new LLM architectures
- Debugging attention sink-related instability
- Improving convergence in long-context models

## Triggers / Keywords
attention sink, variance discrepancy, super neurons, head-wise RMSNorm, transformer debugging, pre-training stability