The black box.

Known recipe, unreadable result.

How to train a model is well understood: architecture, data, an optimization rule. What comes out is not a program a person can read. These systems are grown more than they are built — the growing conditions are set by hand, the grown thing is not. There is no line of code that says if X, then Y — only billions of learned numbers whose joint behavior produces the answers.

Looking inside these systems, what we see are vast matrices of billions of numbers. These are somehow computing important cognitive tasks, but exactly how they do so isn’t obvious. Dario Amodei, The Urgency of Interpretability, 2025

This is the black box problem: full control over the growing conditions, weak insight into the grown thing. Not a bug someone will patch next quarter — a structural property of how these systems come to exist.

The builders say it themselves.

This is not an outsider’s critique. Across rival labs, the admission is the same.

People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. Dario Amodei, CEO of Anthropic, 2025

The inner workings of language models are often a mystery, even to the researchers who train them. Google DeepMind, Gemma Scope announcement, 2024

At OpenAI, many of GPT-4’s abilities came as a surprise to the people who made it. Boaz Barak, a Harvard computer scientist then working on OpenAI’s superalignment team, says many in the field compare their position to “physics at the beginning of the 20th century”: plenty of experimental results, no theory that explains them. Mikhail Belkin, who studies why deep learning works at UC San Diego, is blunter — today’s training recipes are “more alchemy than chemistry.” MIT Technology Review compressed the state of the field into a headline: “Nobody knows how AI works.”

Reading the inside, slowly.

The field is not shrugging. Mechanistic interpretability tries to open the box directly: Anthropic’s Mapping the Mind of a Large Language Model extracted tens of millions of human-readable concepts from a production model, and follow-up work on circuits traces individual steps of a model’s reasoning. The broader discipline of explainable AI has been surveying approaches for years (Adadi & Berrada, 2018). And there is a sharp counterposition: Cynthia Rudin argues that for high-stakes decisions the honest move is not to explain black boxes at all, but to use models that are interpretable by construction.

Real progress, openly contested methods, and a gap that is still wide. That is the current state.

A posture, not a complaint.

Working daily with systems nobody can fully read calls for a particular posture: observe before theorizing, claim only what the evidence carries, keep the uncertainty visible instead of performing past it. This lab keeps that reminder where it cannot be overlooked — in its name.

Maintained questions.

• Does interpretability mature before the systems outgrow it? Amodei frames it as a race; the outcome is not decided.
• Where is the line between understanding a system in principle and being able to read it in detail — and which of the two do high-stakes uses actually require?

2026-07 — First version: problem, admissions from Anthropic, Google DeepMind, and OpenAI-adjacent research, interpretability and its counterposition.