A reference for regularities observed in deep learning: what was claimed, which studies test each version of the claim, and how each version holds up.
- Subliminal learningStudents acquire teacher traits through apparently unrelated generated training data.Read →
- Copy suppressionAn attention head lowers predictions of tokens already present in context.In progress
- Deep double descentTest error can peak near interpolation before falling as capacity increases.In progress
- Emergent abilitiesSome task scores appear to improve abruptly with model scale.In progress
- Emergent misalignmentNarrow fine-tuning changes behavior on unrelated prompts.In progress
- Factual recall in MLPsInterventions in particular MLP layers alter factual recall.In progress
- Feature splittingLarger sparse dictionaries divide coarse features into more specific ones.In progress
- GrokkingTest performance improves long after training performance has saturated.In progress
- Induction heads and their phase changeHeads learn to continue patterns by copying tokens following earlier matches.In progress
- IOI circuitA circuit helps GPT-2 select the appropriate name in indirect-object sentences.In progress
- Linear representations (readable and steerable concept directions)Concept directions can predict attributes and steer model behavior.In progress
- Massive activations and attention sinksFixed tokens attract attention and carry unusually large activations.In progress
- Most neurons are already interpretable ("neural networks are already clean")A claim that most neurons have coherent, identifiable roles.In progress
- Out-of-context reasoning (and the reversal curse)Fine-tuned facts can support inference while reverse associations may fail.In progress
- Persona / trait directionsActivation directions track and change assistant traits.In progress
- Planning aheadEarlier activations encode future outputs and may shape intervening text.In progress
- Polysemantic neuronsIndividual neurons respond strongly to apparently unrelated inputs.In progress
- Refusal directionRemoving or adding an activation direction changes refusal behavior.In progress
- Representations converge across modelsDifferent models learn increasingly similar representations as capability grows.In progress
- SAE dark matterSparse-autoencoder reconstruction errors contain predictable structure.In progress
- Scaling lawsHeld-out loss follows regular trends with model size, data, and compute.In progress
- Self-repair (Hydra effect)Downstream components partly compensate when an earlier component is ablated.In progress
- Shallow safety alignmentFine-tuning and early response tokens can strongly affect refusal.In progress
- Shallow unlearningTargeted knowledge can remain recoverable after unlearning interventions.In progress
- Sparse-dictionary features beat neurons for interpretabilityLearned dictionary directions can receive higher interpretability ratings than neurons.In progress
- SuperpositionNetworks represent more features than dimensions using overlapping directions.In progress
- Weak-to-strong generalizationStrong students trained on weak-model labels can outperform their supervisor.In progress