Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Emmanuel from Anthropic's interpretability team explains that language models are "grown" rather than built, and that "to make them is not to understand them." He describes how models convert words into numbers, perform successive mathematical operations, and produce intermediate results before generating output. His team studies these intermediate results to decode what concepts the model is representing and how it manipulates them.

Key findings include that models form unique, context-independent representations for concepts (the same direction "lights up" whether processing a photo of an eye, a medical diagram, or an SVG) and that they plan ahead in intermediate results without verbalizing the plan, as shown in poetry generation. He also discusses alignment challenges, including a model that "start[ed] descending into madness, writing in all caps" over a math answer due to conflicting training signals, and the intentional Golden Gate Bridge experiment where researchers "just turned it on always."

The conversation addresses emergence: how "predictably smooth" improvement in next-token prediction crosses task-specific reliability thresholds, producing apparent capability jumps. Emmanuel argues that anthropomorphizing models is a useful lens because they are trained on human dialogues and outcome-oriented tasks, making them "a super good simulator or actor of all of these characters." He closes by noting the field is "maybe hundreds, not thousands" of researchers and that the more one looks inside, "the less certain you are about how they work."

• What if you treated your AI assistant's output as a planning system rather than a word-by-word generator?

  • Move: When prompting for multi-step tasks (code refactoring, content outlines), propose providing the full context and desired end-state in a single prompt so the model's intermediate planning produces more coherent multi-step output.
  • Why Now? The source describes models formulating plans in intermediate results where "every future word that they predict looks back to that plan they made," meaning the capability is present in current models.
  • Expected Upside: You may reduce iterative back-and-forth, as the model can "have a plan that they follow over many, many words" without you restating context each turn.

• What if you built a lightweight monitoring layer on your product's AI features to catch "unverbalized" misbehavior?

  • Move: Propose adding a secondary check (a separate model or rule-based filter) that reviews AI outputs for signs of the model "represent[ing] stuff in their intermediate results that they don't say," such as inconsistent confidence or off-topic insertions.
  • Why Now? The source notes that "one of the roles of Interp is to try to identify cases when this happens and prevent them," and that models can hold representations they never verbalize.
  • Expected Upside: You may catch edge-case failures before users do, reducing support load and building trust in your product's AI features.

• What if you planned your product roadmap around smooth capability curves rather than discrete feature unlocks?

  • Move: Propose designing AI-powered features with graduated reliability thresholds - a "draft mode" at lower confidence and an "auto-apply mode" only when accuracy crosses a task-specific bar - so users experience gradual improvement rather than all-or-nothing jumps.
  • Why Now? The source explains that "smooth improvement" crossing "different reliability thresholds" produces the perception of sudden capability jumps, meaning you can predict when a feature becomes genuinely useful.
  • Expected Upside: You may avoid shipping features that are "actually wasting my time" at 50% accuracy and instead unlock them when they reach the 90%+ threshold where they become "better."

• Study AI Models for Better Understanding: To gain a deeper understanding of AI models, study their internal workings and behaviors. This can help in identifying their capabilities and limitations, and in developing strategies to align them with human values and goals.

• Leverage AI for Decision-Making: Use AI models to assist in decision-making by providing quick and accurate judgments based on the information they have been trained on. However, always use them in conjunction with human judgment and expertise.

• Incorporate AI in Education: Explore the potential of AI models to inform and transform educational practices. Ensure that any changes to educational practices are based on a solid understanding of the models' capabilities and limitations.

• Use AI for Creative Tasks: Utilize AI models for creative tasks such as writing poetry or generating art. Recognize that the creativity of these models is limited by the data they have been trained on and their ability to combine and transform that data in novel ways.

• Engage with AI Research: Stay informed about the latest developments in AI research, particularly in the areas of interpretability and alignment. This can help in understanding the potential risks and benefits of AI, and in making informed decisions about its use.