← All projects
★ Featured · Jul 2026 – present · Current
Mechanistic Interpretability of LLMs
Independent research — LLM reliability evaluation
An LLM-reliability evaluation built on mechanistic interpretability: can a model's internal activations tell us when its answers are wrong? The testbed is a 640-item benchmark of SEC-verified corporate facts, probed against Llama-3.1-8B's residual stream.
Key Contributions
- Built a 640-item benchmark of corporate facts verified against SEC filings, split by whether each fact was stable, superseded, or recently established
- Trained per-layer linear probes on Llama-3.1-8B residual-stream activations to predict answer correctness — 0.74 accuracy under entity-disjoint splits, with shuffled-label and random-direction controls at chance
- Localized factual recall via activation patching across 180 entity-swap pairs, attributing 68% of logit-difference recovery to mid-stack MLPs at the entity token
- Identified a calibration failure: the internal error signal misses 57% of errors on superseded facts, where the model most often emits the outdated value
MLInterpretabilityResearchPyTorchTransformerLens