Galileo connects agent traces, evaluations and runtime controls

Galileo helps teams inspect and improve AI applications through observability, evaluation and runtime controls. It records the steps behind an answer, scores selected behaviour and supports comparisons across prompts, models or application versions. The important question is whether those measurements identify the failures that matter to users. Collecting more traces is useful only when the team can convert them into better tests, clearer release decisions or targeted controls.

01 / ProductA measurement layer around the agent you already build

The Galileo overview describes integrations with model providers and agent frameworks, alongside Python and TypeScript SDKs and an API. Galileo supplies the measurement and control layer; the application still defines its business task, tools and acceptable outcomes. This distinction matters when a polished dashboard makes an immature application appear more dependable than its underlying process.

The observability concepts distinguish sessions, traces and spans. Sessions group related interactions, traces represent a request or workflow, and spans capture individual operations such as retrieval or a tool call. These units help explain why a final answer failed without treating the answer itself as the only evidence.

The company also promotes Luna-2 as an evaluation option. The metrics overview explains built-in metrics, model-based judging and code-based custom metrics. Many metrics require an LLM integration or Luna-2 configuration. A ready-made evaluator is therefore a component to validate against your intended rubric, not an independent authority on whether an application did its job.

02 / AudienceTeams replacing anecdotal agent testing with repeatable evidence

Galileo is a strong candidate when developers repeatedly hear that an agent is unreliable but cannot separate retrieval errors, poor tool selection and bad final wording. An application with several steps benefits from seeing the path taken, not merely the user-visible text. Product specialists can help specify what a correct result would require, while engineers connect those requirements to recorded operations.

The less suitable expectation is that an observability subscription will automatically create a reliable evaluation dataset. Someone still needs to define representative inputs, determine acceptable outcomes and investigate disagreements. A team with no owner for those decisions can accumulate scored traces while continuing to ship changes based on a few persuasive examples.

The Arize AI blueprint provides another approach to traces, datasets and experiments, including Phoenix. The Braintrust blueprint is a useful comparison for teams organizing evaluation around development and release. Compare the work required to turn a real failure into a durable test in each system, rather than selecting from the largest count of dashboard metrics.

03 / WorkflowA proposed test of an agent that misroutes support requests

Imagine a support agent that searches the knowledge base, classifies a request and chooses whether to respond or hand it to a person. Some users receive a confident answer to an issue the agent should escalate. This proposed workflow examines the whole decision. It is not a hands-on Galileo test, and it does not imply that a vendor customer achieved a particular improvement.

Instrument the decision boundaries: original question, retrieved material, classifier output, selected tool and final response. Attach application and prompt versions. Keep enough context to explain a mistake while excluding information the evaluation team does not need. If a tool operation is not instrumented, absence from a trace must not be interpreted as proof that it never happened.

Construct a small but varied reference set. Include straightforward questions, requests that genuinely require escalation, ambiguous cases and attempted misuse. Ask subject specialists to state the required action and the reason, not merely write an ideal-sounding answer. For this agent, an eloquent response can still be wrong if it fails to escalate an unsupported account operation.

Use the experiments guide to connect a dataset, execution method and metrics. The documented pattern can compare prompts, models or custom application code. Begin with a baseline and change one decision-making factor. If the retrieval collection and escalation prompt both change, an improvement becomes harder to attribute and harder to reproduce later.

Divide checks by what can actually establish the requirement. Code can verify that a required tool was called or that a record identifier belongs to the approved input. A model judge can assess whether an explanation acknowledged an uncertainty, but should be calibrated against expert decisions. Inspect disagreement cases before using the score as a release condition; apparently sensible feedback can reward a response that violates the business rule.

Read traces for failed examples and compare them with successful examples of the same request type. If retrieval never returns the escalation policy, adjusting the final answer's tone will not address the underlying cause. If the right policy is present but ignored, the next experiment can focus on the decision instruction or tool boundary. This is why a sequence of meaningful spans is more useful than a single aggregate score.

For selected runtime checks, follow the current Agent Control documentation. It describes a central layer evaluating model and tool inputs and outputs. The current documentation navigation labels Protect deprecated, so a new implementation should confirm the Agent Control setup and supported integration rather than copying an older Protect tutorial. Preserve the control version alongside the application version.

Evaluate intervention as well as detection. A control that blocks a request needs a defined response: retry safely, ask for clarification or escalate. Test ordinary requests that resemble problematic ones so that false positives become visible. In this example, a customer describing an error message might quote text that appears suspicious but is essential to solving the issue.

Finally, retain a holdout dataset and review new production failures. When the same examples are repeatedly tuned until they pass, the score measures familiarity with that set. A useful program adds newly observed cases while preserving an independent check on generalization. Assign an owner who decides when a rubric changes and whether historical experiment results remain comparable.

04 / PricingTrace allowances and evaluation calls are different costs

The Galileo pricing page, consulted 23 September 2026, displays Free at $0 with 5,000 traces per month. Pro is shown at $100 per month billed yearly, with 50,000 traces per month; its price scales with trace volume. Enterprise has custom terms, private deployment options and real-time guardrails. The annual-equivalent Pro figure should not be described as a cancellable monthly subscription.

Estimate usage from the application's actual trace boundaries. A conversation can contain several turns and therefore several traces, while each trace can contain many spans. Use a representative sample before forecasting annual expenditure. Also verify retention and rate limits appropriate to the intended logging pattern; an allowance alone does not tell you how a sudden burst will behave.

Budget evaluation-model execution separately where an external provider is involved. Unlimited custom evaluations in plan copy does not establish that every provider call is prepaid. The metrics guide requires configured integrations for model-based metrics, making the chosen evaluation route part of the commercial design. Confirm Luna-2 charges, external provider billing and the exact runtime-control entitlement with the account owner.

05 / DistinctionsExperiments can connect application behaviour to a shared rubric

Galileo's practical distinction is the connection between interaction structure and an evaluation workflow. A support specialist can explain why a case should have escalated; an engineer can locate the wrong decision in a trace; an experiment can test a revised policy on the same cases. That shared record is more valuable than a quality score whose meaning changes between teams.

The company's current platform presentation spans observation, evaluation and protection. Our assessment is that this combination is useful when teams want to learn from production and enforce selected boundaries using related evidence. It does not eliminate the difference between an offline experiment and a runtime intervention. Each needs its own latency, failure-handling and human-review expectations.

06 / QuestionsCalibrate the judge and confirm the supported control path

The central uncertainty is whether an evaluator agrees with the people responsible for the workflow on difficult cases. Measure disagreement separately for ordinary requests, ambiguous requests and consequential failures. A good average can conceal systematic errors on the exact category that prompted the evaluation program. Preserve examples that challenge the rubric, including cases where experts reasonably disagree.

There is also a product-transition question. Public marketing still uses Protect language, while current documentation marks it deprecated and provides Agent Control instructions. Confirm migration and entitlement for the intended account. This source review has not benchmarked Luna-2, verified private deployment controls or tested runtime blocking. Treat latency and effectiveness claims as vendor claims until measured on the chosen configuration.

07 / DecisionStart with the behaviour you need to change

Galileo is worth evaluating when a team can name an agent failure and commit to maintaining the evidence around it. Define a rubric, record the relevant execution steps and compare a targeted change before broadening instrumentation. Select a plan after measuring real trace volume and evaluator usage. The outcome to seek is a clearer decision about what to ship and what to block, supported by inspectable cases.