LangWatch tests the conversation behind an AI agent

LangWatch gives teams a way to inspect an AI agent’s decisions and test how it behaves over a conversation. Its platform brings together execution traces, evaluations, prompt management and simulated users. The useful question is whether a team can turn an observed failure into a repeatable release check. A convincing answer depends on the scenarios and judgement behind the check, as well as the software that runs it.

01 / ProductConnect the execution record to the next change

The documentation introduction describes several connected areas: LLM operations, coding-agent visibility, an AI gateway and governance. Within LLM operations, traces record what happened, evaluations assess results, prompt management versions instructions, and agent testing explores conversations. These parts can be adopted separately. A team need not route every model call through a new gateway simply to investigate one unreliable assistant.

The observability guide describes instrumentation that records calls and interactions in traces, with cost and performance visibility. A trace is valuable because the final answer alone can conceal a failed retrieval or an unnecessary tool call. The record gives an engineer a sequence to investigate. It does not establish that every action in that sequence was correct, authorised or useful.

The broader platform overview also presents voice testing, red teaming and prompt versioning. Those are different jobs within an agent lifecycle. Testing a conversation, observing live traffic and enforcing a request-time rule should have separate success criteria. Combining them in one interface can reduce handovers, but a shared dashboard should not blur the distinction between finding a problem and preventing it.

02 / AudienceUseful when a single answer is too small a test

Consider an appointment assistant that must gather missing details, offer a valid slot and hand off exceptional cases. A test of its first response could pass while the full conversation still fails. LangWatch is particularly relevant when the important behaviour spans several turns, when tools have side effects, or when users regularly correct information they supplied earlier.

A platform team can also use it to give several product teams a consistent vocabulary for failures. One group may care about unsupported claims; another may care about tool sequencing. Keep those requirements explicit instead of forcing both into a single organisation-wide quality score. The decision-maker should be able to see whether a release improved the intended task or merely changed the average.

For a team centred on dataset experiments, Braintrust is a relevant comparison. Where domain specialists primarily maintain prompt definitions and release labels, PromptLayer offers another useful frame. The comparison should follow the team’s daily work: debugging execution, reviewing prompts or rehearsing conversation outcomes. The number of features on a landing page is a weaker guide.

03 / WorkflowProposed workflow: rehearse an appointment conversation

Start with a proposed assistant that only drafts an appointment request for an employee to confirm. Prepare situations covering a complete request, an unavailable time, a correction of the customer’s name and an instruction to cancel. Use fictional records and a test calendar. This keeps the first investigation focused on whether the assistant follows the intended sequence without creating real bookings.

The agent-testing guide describes a simulated user, the agent under test and a judge that checks stated criteria against the conversation and traces. Scenarios can be maintained in the platform or written with the Scenario SDK. Define observable conditions, such as asking for a missing date and calling an availability tool before proposing a slot, rather than a vague instruction to be helpful.

Record the failed conversation and inspect the associated trace. If the availability tool returned an error, changing the greeting is unlikely to help. If the model ignored a clear tool result, a prompt or model change may be appropriate. Preserve the original failure as a regression case, then add a different phrasing of the same situation so that the test does not reward memorising one transcript.

The evaluation overview separates dataset experiments, online evaluations and request-time guardrails. Use a small dataset to check the slot-selection component, then use conversation tests for the whole interaction. A guardrail must be integrated into application control flow to block a response. A score written to a dashboard after the answer has been sent cannot prevent that answer from reaching a user.

04 / PricingBudget for activity inside the agent

The current pricing page lists a free Developer tier and Growth at €29 per core seat per month. Growth includes 200,000 events, with additional events at €5 per 100,000; extended storage is separately metered. Enterprise is negotiated. These are platform charges, so model inference used by the application or testing should be considered separately when estimating a total operating budget.

The billable-events reference is more precise than counting conversations. Ingested spans, evaluation results, experiment activity and simulation activity can each contribute. Reading dashboards is not metered as an event, while Instant Evals uses a separate token-based meter. A conversation that performs retrieval, calls tools and runs several checks can therefore generate multiple billable records.

Estimate a normal release cycle as well as normal production traffic. A team that reruns many scenarios during development may spend differently from one that mostly observes stable production. Retaining verbose traces also has a storage consequence. Choose a retention period that lets investigators revisit an incident, then distinguish that useful history from data kept indefinitely without a defined purpose.

05 / DistinctionsTests can examine actions as well as answers

The useful distinction in conversation testing is that success can include what the agent did internally. A pleasant final message is insufficient if the assistant skipped a required lookup. Conversely, a cautious refusal may be correct even when a simple satisfaction grader dislikes it. An execution-aware review makes these differences visible and gives developers more specific feedback than a general thumbs-up score.

Integrated experiments and traces can also shorten the path from an incident to a test. That is most useful when the team records enough context to reproduce the failure: prompt version, model choice, relevant tool responses and the intended policy. Without that context, a saved transcript can become an anecdote. The practical value comes from preserving the conditions that explain the result.

LangWatch’s pricing advertises self-hosting and enterprise deployment options. For an organisation with existing telemetry requirements, that can make deployment location part of the evaluation. Self-hosting still needs capacity planning, upgrades and operational ownership. Compare the work required to operate the platform with the controls gained; neither a hosted subscription nor an open repository makes that trade-off disappear.

06 / QuestionsA simulated user and judge both need scrutiny

A simulated customer may miss the speech patterns, ambiguity or interruptions that occur in the real service. Build scenarios from observed task failures and have domain staff review the generated interactions. Include cases where the correct outcome is escalation or no action. Otherwise a suite can encourage completion at any cost, which is often the wrong target for agents that act on business systems.

A model judge can also disagree with a human reviewer or vary between runs. Keep its rubric and configuration with the test evidence. When a release fails only on a borderline judgement, inspect the underlying conversation before rewriting the agent. Repeated disagreement may reveal a vague requirement rather than a model defect. Deterministic checks are useful for exact conditions such as whether a required tool was called.

Traces can contain customer messages, retrieved records and tool arguments. Decide which fields to mask or omit before sending them to an observability system. Confirm the retention and access arrangements for the selected deployment. This article has not inspected a customer’s configuration, so the availability of governance features should not be read as proof that a particular installation protects its data correctly.

07 / DecisionChoose one behaviour that the team can explain

LangWatch merits a pilot where an agent’s multi-step behaviour is difficult to assess through individual examples. Start with one workflow and a modest set of scenarios whose expected outcomes are understood by the people responsible for the service. The most useful result is a release decision that can be explained from a trace and reproduced in a test, including the cases where the system should stop.