sequenced.ai
Articles/Models & infrastructure/Blueprint//7 min read

How Goodfire looks inside a model that learned the wrong lesson

Goodfire investigates how neural networks arrive at their outputs. Its work offers model builders a way to examine a troubling result before changing the training run.

By Sequenced deskAI-assisted, source-led · how we work
Visit Goodfire website ↗
ActivationsInternal evidenceSignals recorded while a model processes an example.
ProbesMonitoring methodSmall learned readouts look for a selected signal.
LLMsProduct areaThe lab works on training and serving behavior.
Life sciencesSecond practiceResearchers examine what predictive models learn.
Goodfire mark
Goodfiregoodfire.com · independent research

Represent this company? Verify your work email to access its workspace, or send the desk a factual correction.

A model can earn full marks on a test for the wrong reason. Imagine an assistant that routes every invoice bearing one supplier’s letterhead to the correct department. Did it read the line items, or did it learn the appearance of the page? The scores can look identical until that supplier changes its stationery. Goodfire works on interpretability, the study of what goes on inside a neural network while it produces an answer. It gives teams that build models another way to investigate this kind of puzzle.

In brief
  1. 01The company A public benefit corporation working on tools and research for understanding and monitoring neural networks.
  2. 02The buyer A team with a specific behavioral question, access to the relevant model and a decision it can make from the findings.
  3. 03The opening move Bring contrasting examples, the training or model versions involved and a clear definition of a convincing result.

01 / ProductWhat a probe adds to a table of correct and incorrect answers

Goodfire’s company description traces the lab to researchers with interpretability experience at OpenAI and Google DeepMind. Its language-model offering covers debugging behavior, finding evidence of reward hacking during training, and monitoring behavior during inference. Those activities have different targets. A training researcher can revise a reward or remove a misleading set of examples. A person operating a deployed service may instead need an alert that identifies a response for inspection.

Internal activations are values produced as a model processes input. A probe is a learned readout that tries to pick out a particular signal from those values. Imagine building one to distinguish examples where the invoice assistant relies heavily on page appearance. The probe would supply another observation beside the predicted department. A correlation would be a lead to investigate. To learn whether the apparent shortcut actually affects routing, the team would still need examples that separate the visual clue from the invoice’s meaning.

Goodfire also describes work in life sciences, where a researcher might ask whether a predictive model has detected a meaningful feature or an accident of how the dataset was assembled. The two product areas share a question about evidence inside a model. Their validation requirements differ. In the invoice example, a finance team can provide the correct department. In a scientific collaboration, an interpretation may yield a hypothesis requiring independent experimental or observational evidence.

02 / AudienceStart where an internal explanation would change a real decision

A model-building team with two plausible explanations for a regression has a reason to consider Goodfire. For instance, a drop in routing quality could come from the model relying on layout, or from a change in what the organization calls the departments. These call for different fixes. If a simple check of the label dictionary settles the matter, an expensive internal investigation would add little. An unexplained gap after those checks is a better starting point.

Teams that consume a third-party API face a practical constraint. They may be able to collect answer texts, timing and user feedback while having no access to the values inside the provider’s model. Ask whether Goodfire can work with exactly that deployment and which party must authorize access. An organization that owns its training run can often act on a finding in ways an application developer cannot, from revising a training set to comparing successive checkpoints.

03 / WorkflowA proposed investigation into the invoice assistant

Assemble pairs of invoices that disassociate visual appearance from the correct destination. In one pair, keep the line items and department fixed and change the template. In another, keep the template fixed while changing the items sufficiently to require a different department. Mark uncertain labels for an accountant to adjudicate. Keep the evaluation examples away from the material used to choose a probe. Otherwise the team could accidentally measure its ability to recognize a handful of prepared examples.

Compare model behavior on both pairs before examining an internal signal. If the predicted destination moves with the template, record exactly when that happens and which model version produced it. Preserve examples where the model gets the right answer despite the changed form. These examples help distinguish a harmless awareness of the template from a dependence that damages the business task. A model can recognize a logo and still make a sound routing decision.

Goodfire’s research on shaping representations examines cases in which a model has learned a shortcut and the researchers try to influence what it learns internally. For the proposed invoice study, ask the lab to explain what its chosen measurement can distinguish, how it performs on held-out examples and what evidence would overturn the interpretation. If the signal follows the template yet behavior remains reliable, the team has found a feature worth understanding. It has yet to justify altering the model.

If the evidence does justify a change, reserve one unseen supplier and a second document layout for a final check. Compare the revised system with a simpler intervention such as diversifying templates in training. Record the errors both approaches fix and any correct invoices they now misroute. This makes the decision practical for a finance operations lead. They can see the work saved, the exceptions introduced and which categories need ongoing review.

04 / PricingThe commercial entry point is a model-specific conversation

Reader needPublic routeDetail to establish
Investigate a model regressionDescribe the model and objective to the teamAccess needed, investigation length and deliverables
Monitor a training or serving workflowDiscuss the language-model offeringDeployment environment, review burden and commercial terms
Examine a published resultRead the lab’s researchApplicability to the model and data you control

Engagement routes checked on 19 November 2026. Goodfire’s inquiry form and its language-model offering.

The inquiry form requests a work email and asks what the team wants to achieve. Goodfire says it follows up if there is a fit. The inquiry and product pages reviewed here give no public seat, token or project price. The practical comparison is a written scope for one question. Specify whether the desired deliverable is an explanation, a reusable detector, an integration into a training pipeline or some combination. Each requires different work from the lab and the customer.

Older demonstrations can give a misleading impression of access. Paint with Ember explicitly says its interactive demo has been retired. A reader pricing an internal model study should use the current partnership route. Establish where weights and examples may be processed, which intermediate data can be retained and whether the organization can rerun an analysis after a model update. These details determine whether the first engagement buys a finding about one checkpoint or an ongoing operating capability.

05 / DistinctionsIts research links monitoring to decisions made during training

In a study of reward hacking, Goodfire researchers report finding internal signals associated with models satisfying a grading mechanism in unintended ways. They compared activation-based observations with other forms of monitoring on selected models and environments. The interesting operational question is what a team can do with a warning. A training owner might inspect the grading rule, repair an ambiguous task or pause a run. No published example establishes a universal detection rate for an unrelated model or grading system.

Two adjacent tools help locate that distinction. Arize covers evaluation and observability, useful when a team needs to discover where behavior changed across application traffic. Comet addresses experiment tracking, useful when reconstructing the datasets and runs behind a reported change. Goodfire’s distinctive question is what evidence inside a model can discriminate between competing explanations. A careful investigation can draw on all three kinds of records.

06 / QuestionsWill the explanation survive the next model version?

Plan for a probe to become less informative as a model changes. The representation it reads can shift even when a user asking the same question sees a familiar answer. Ask which checks would detect that shift and what would require fresh labeled examples. For a live monitor, the cost of reviewing false alarms also matters. Ten flags in a thousand requests are manageable only if investigators can tell which ones merit action in the time available.

Require the final report to identify an observation that would weaken the proposed explanation. For the invoice example, that could be a new template on which the signal remains strong while routing follows the line items correctly. This is a testable disagreement between two accounts of the same behavior. It gives the next person maintaining the model more to work with than an appealing visualization of an internal feature.

07 / DecisionMatch the engagement to the evidence you can act on

01

You own the training process

Bring a reproducible failure and a checkpoint the lab can examine. Describe the retraining or monitoring decision that depends on the finding.

Scope a single falsifiable question.
02

You operate a model endpoint

Specify the available internal access, acceptable alert volume and review owner. Validate a proposed monitor against traffic that resembles deployment.

Set the operating threshold from real review capacity.
03

You only see application responses

Collect contrasting outputs and check ordinary evaluation and data issues first. Establish what model access a deeper investigation would require.

Resolve the access question before commissioning the study.
What should we explore next?

A business worth understanding.

Suggest your business or one you find interesting. Tell us what you want to understand about its product, positioning, design or workflows.

Suggestions are free. Selection and publication stay with the desk.

Sources

Continue reading

All in this category