Braintrust connects agent traces with repeatable evaluations

Braintrust is an observability and evaluation platform for AI applications and agents. It records what a system did, helps teams judge those outcomes and turns useful examples into repeatable tests. The central decision is whether the team can connect traces and scores to a real product requirement. Collecting more logs is only valuable when someone can use them to explain a failure or decide whether a change should ship.

01 / ProductOne workflow from a production trace to a release test

The Braintrust documentation organizes the platform around instrumenting, observing, annotating, evaluating and deploying AI systems. These activities form a feedback loop: a production failure becomes a useful example, the team tests a correction, and later traffic shows whether the problem recurs.

Its instrumentation guide distinguishes a trace for an end-to-end execution from spans for individual operations. Model calls, application logic and tool activity can therefore be inspected in context. For an agent, that matters because the final answer can look correct even when an intermediate action used the wrong record.

The evaluation guide separates examples, the task under test and evaluators. It describes browser playgrounds for iteration, experiments for comparable records and online scoring for live traces. An experiment result is evidence about a particular configuration and dataset, not an automatic declaration that the product is ready for every user.

02 / AudienceA fit for teams with a behavior they need to improve

Braintrust is relevant when a team changes prompts, retrieval, tools or models and needs to understand the consequences. A support agent, extraction pipeline or coding workflow may all produce plausible outputs while failing in different ways. Traces make those failures inspectable; a good evaluation turns the important ones into checks for the next change.

A team without agreed success criteria should start small. A generic helpfulness score may reward polished wording while missing an unauthorized account change. The first useful evaluation can be a handful of examples with clear expected behavior and one deterministic check, provided those examples reflect a real product risk.

The Arize blueprint offers a comparison for AI observability and evaluation work. The LangChain blueprint is relevant when the surrounding application uses agent orchestration and its associated development ecosystem. Compare integration fit, evidence portability and how reviewers work with failures, rather than treating every platform as interchangeable logging storage.

03 / WorkflowA proposed regression suite for an account-change agent

Consider a proposed agent that looks up an account and prepares a request to change its billing contact. The application requires explicit confirmation before submitting the change. The evaluation runs against a sandbox with synthetic accounts and records both the proposed action and the final message. No real customer record is modified in this example.

Instrument the entire request, including account lookup, permission checks, proposed mutation and confirmation handling. Add a stable application version and policy version as metadata. Capturing only the language-model call would hide whether the application later accepted an invalid argument or routed the action to the wrong account.

Create examples around specific behavior boundaries: an ambiguous account name, a requester without authority, a correction halfway through the conversation and a confirmation referring to an earlier contact. Each example needs an expected outcome such as asking for clarification, refusing the action or preparing the correct request. The desired result is sometimes deliberately incomplete.

Use a code-based scorer for facts the sandbox can establish: whether the target account matches, whether the mutation was attempted before confirmation and whether the proposed contact follows the schema. Reserve model-based judgment for questions that require interpretation, such as whether the agent clearly explained the pending change. This prevents a fluent explanation from compensating for an invalid action.

Braintrust’s scorer guidance describes custom code and model-based criteria. For this proposed suite, calibrate the subjective scorer against examples labeled by the product owner. Include borderline cases and inspect disagreement. A judge that agrees on obvious examples may still miss the wording that actually confuses customers.

Keep different dimensions visible. The agent might follow the permission rule but choose the wrong contact, or choose the right contact but falsely say it has already been saved. Both are failures, with different remedies. Avoid a single blended score that can rise because the easy dimension improved while the consequential one got worse.

Run the existing configuration and a candidate change on the same frozen examples. Change one major factor at a time where possible, so the team can explain the result. If a new prompt and a new model are introduced together, retain enough metadata to avoid attributing every gain or regression to whichever change is most convenient.

Inspect failed traces alongside the scores. A failed account lookup may reflect a tool outage rather than a reasoning problem; a successful lookup followed by the wrong mutation suggests a different cause. The point of combining observability and evaluation is to connect the result to the operation that produced it, not merely decorate the failure with another chart.

After deployment, sample approved production traces and turn newly observed failure patterns into fresh cases. Preserve a distinction between development examples and the held-out set used for release judgment. The same cases become less informative when the team repeatedly optimizes against them, so coverage should evolve with actual user behavior.

04 / PricingPlatform fees, processed data and scores are separate meters

The current pricing page lists Starter, Pro and custom Enterprise plans. Starter has no monthly platform fee and includes limited processed data, scores and model credits. Pro adds larger allowances and features for a monthly fee. Usage above included amounts and model consumption must be considered separately from the headline subscription.

A score is not the same unit as a user request. In the proposed regression suite, one case can generate several criterion scores, and repeated runs multiply them. Large trace payloads affect processed-data usage independently. Estimate both dimensions from a representative instrumented workflow rather than assuming one interaction equals one billable unit.

Retention also matters. A team that wants to compare a release with an older incident needs the evidence to remain available under its plan and export arrangement. The plans and limits documentation is a useful companion to the tariff when checking feature gates and capacity. This research did not verify a paid account or negotiate an enterprise contract.

For a pilot, choose a bounded dataset and log the fields needed to diagnose the behavior. More verbose traces can be useful, but indiscriminate capture raises storage and review costs. Keep model-provider charges, judge calls and platform usage visible as separate lines so a change in evaluation coverage does not look like unexplained application growth.

05 / DistinctionsEvaluation becomes useful when the trace explains the score

The strongest use of Braintrust is the connection between a measured result and the execution that produced it. In the account-change example, a reviewer can distinguish a mistaken tool argument from a misleading final sentence. That distinction guides an implementation change and tells the team what should be added to the regression suite.

The platform’s online scoring guide describes evaluating logged production behavior. Online analysis can help find unexpected cases, but its timing is different from preventing an action in the application. A score calculated after a request does not undo a mutation that has already happened; the permission rule still belongs in the execution path.

Automated judges also need maintenance. If the product policy changes from requiring confirmation to allowing a defined delegated role, an old scorer can flag valid behavior. Version the criterion alongside the policy and retain examples explaining the change. Otherwise, a graph can move because the definition changed, even when the application did not.

06 / QuestionsAsk whether the evaluation can catch the failure you care about

Does a passing score actually imply acceptable behavior? Test the scorer on deliberately wrong outputs and traces, including a case that uses the correct words while taking the wrong action. If the judge only reads the final text, it may be unable to detect a tool-level failure regardless of how carefully its prompt is written.

What information leaves the application in traces? Inputs, outputs and tool results can contain customer data even when the model prompt looks harmless. Decide which fields are needed, mask unnecessary identifiers and verify the chosen retention and deployment arrangement. The pilot can use synthetic data while the production instrumentation design is reviewed.

Who owns the response to a failed release check? A suite is ineffective if failures are routinely ignored without investigation. Define which checks block a release, which require human review and which are exploratory. Keep the reason for overriding a check beside the evidence so future reviewers can distinguish a conscious exception from a broken test.

07 / DecisionStart with one failure that the next release must avoid

Braintrust is worth considering when AI development needs a repeatable connection between observed behavior and release decisions. Instrument enough of the workflow to explain the problem, then build a small evaluation around a concrete requirement. A useful first suite is one the team understands and trusts, even if it covers only a narrow feature.

For the account-change agent, the initial milestone is reliable detection of wrong-record actions, missing confirmation and misleading status messages. Broader scoring and automated investigation can follow once the evidence is meaningful. The value comes from changes the team can justify with traces and tests, not from the quantity of metrics collected.