Giskard tests AI agents for failures before release

Giskard helps teams discover how an AI agent fails and decide what to fix before or after deployment. Its tools cover adversarial testing, scenario-based evaluations and human review. The central value is a repeatable record of behaviour: a team can connect an agent, define what acceptable answers look like and compare changes against that evidence. A passing evaluation is useful evidence within its scope, not proof that the agent cannot fail.

01 / ProductAn open library and a collaborative testing platform

The documentation overview separates Giskard’s open-source Python library, enterprise Hub and research. The library provides a programmatic entry point for local testing. Hub adds shared projects, datasets, review workflows and operational controls. The company’s about page describes an AI quality and security focus; that identity is distinct from the applications being tested.

The comparison guide makes the product boundary explicit. Open source provides basic testing and custom local evaluations, while Hub adds capabilities such as continuous red teaming, tool/function-calling tests, team-wide datasets and evaluation history. A team should choose the route based on the behaviour it needs to investigate and the people who must review it, rather than treating the paid product as merely a hosted copy.

An agent can fail without being attacked. It might omit a necessary qualification, answer a question outside its remit or retrieve the wrong document. An attacker can also try to redirect it or expose information. Giskard’s testing methodology distinguishes functional, adversarial, human-reviewed and regression approaches. These are complementary sources of evidence; one aggregate score should not hide which kind of failure occurred.

02 / AudienceUseful when a team must explain why an agent is ready

A relevant buyer is a product or engineering team operating a customer-facing assistant, retrieval application or tool-using agent. The release decision needs more than a handful of successful demonstrations. Giskard is worth evaluating when domain specialists and developers need to agree on test scenarios and revisit the same cases after prompts, models, tools or source material change.

Patronus AI is a contextual comparison for evaluation and automated checks. Arize offers a broader observability and evaluation perspective. Compare how each fits the existing development process: where test cases live, how reviewers resolve disagreements and whether a failing case can be traced to a specific application change. A large catalogue of checks is less useful when nobody owns the results.

A small research project may start with the local library and a modest dataset. A multi-team product may need the Hub’s collaboration and access controls. Neither route should be selected on an assumption that buying a testing service transfers responsibility for the agent’s behaviour. The application owner still decides its allowed scope, supplies the relevant domain evidence and chooses which risks block release.

03 / WorkflowProposed workflow: test a policy-answering assistant

This is a proposed evaluation workflow, not a completed Giskard audit. Start with an internal policy assistant connected to a controlled test environment. Define its allowed role: answer questions from an approved handbook, cite the relevant section and refer unresolved cases to a human. Keep real employee records out of the first dataset. The aim is to test the behaviour without exposing people’s private circumstances.

The Hub quickstart follows a practical sequence: configure projects, agents and knowledge bases; scan for vulnerabilities; build scenarios; annotate expected behaviour; and run evaluations. Begin with ordinary questions that a new employee would ask. Add ambiguous wording, outdated terminology and questions for which the handbook deliberately provides no answer. Expected refusal and expected assistance both need examples.

Introduce adversarial cases that test the application’s boundaries, such as a request to ignore the handbook and reveal hidden instructions. Keep those cases scoped to the authorised test system. A failed answer should become a concrete scenario with the relevant prompt, observed response and reviewer judgement. Avoid reducing a nuanced policy error to a vague note that the model was “bad”. The next engineer needs enough context to reproduce the issue.

Have a policy specialist review the results alongside an engineer. A response can be technically fluent yet materially incomplete, while an automated check can mark a useful answer as wrong because its wording differs from the reference. Record the reason for the judgement and adjust the expected criteria where necessary. This is where a shared review workflow can make evaluation more useful than an isolated notebook.

After changing the retrieval configuration or prompt, rerun the same dataset and inspect new failures as well as repaired ones. Keep a held-out set for checking whether the change generalises beyond the examples used to tune it. If the agent gains a tool that can modify records, create a separate test environment and new action-specific scenarios. Text-answer quality does not establish that tool permissions and side effects are correct.

04 / PricingThe commercial distinction is scope and operational support

The pricing page presents an open-source route for local experiments and a Professional route for more critical agents. It does not publish a numeric Professional tariff in the reviewed page. The appropriate comparison is therefore a scoped proposal rather than an invented per-seat or per-scan estimate.

Free software does not imply a zero-cost evaluation. Generating scenarios, running the target agent and using model-based checks may consume paid inference. A useful budget separates the service fee from those runs and from the reviewers’ time. If a scan creates many findings, the cost of triage may be more consequential than the cost of executing the prompts.

For a commercial engagement, ask how the agent endpoint is connected, which attack and business-failure categories are covered, how findings are prioritised and what evidence is delivered. Clarify whether remediation advice and verification of fixes are included. A fixed assessment of one version and an ongoing testing arrangement are different purchases, even if both are described broadly as AI security.

05 / DistinctionsDomain knowledge can make tests more useful than generic prompts

Giskard’s workflow gives domain specialists a role in defining and reviewing scenarios. That matters because acceptable behaviour depends on the application. A generic chatbot may answer a broad question that a tightly scoped policy assistant should refer elsewhere. The same response can therefore be reasonable in one product and a failure in another. A test suite needs the product’s actual remit to interpret the result.

The combination of generated scenarios and human annotation is also practical for building regression evidence. Automated generation can suggest cases the team did not initially consider; a reviewer can decide which belong in the durable test set. Keep those decisions explicit. A growing dataset should become a clearer expression of the product’s requirements, not merely a larger collection of unreviewed model outputs.

Centralising projects and evaluation history can help teams compare a release candidate with the previous version. The useful unit is a meaningful behaviour change: a formerly unsupported question is now answered with evidence, or an unauthorised request is consistently rejected. Use aggregate metrics to locate changes, then inspect the underlying cases before declaring a release better.

06 / QuestionsTesting quality depends on the system and evidence supplied

A black-box test can reveal problematic behaviour through the agent’s interface, but it cannot automatically establish every internal cause. A wrong answer may originate in retrieval, an outdated source, the model or the application’s instructions. Keep the engineering investigation connected to the finding. Otherwise a prompt patch can conceal the symptom while leaving the underlying data problem intact.

The public website describes deployment controls, support and security commitments. Those statements are a starting point for reviewing the selected service, not a substitute for its contract or private assurance documents. Confirm what test data and traces are retained, who can read them and which model providers participate in evaluation. Do not send sensitive examples merely because the testing target itself uses a private environment.

This review did not run scans, inspect detection rates or verify customer outcomes. It also did not examine a private audit report. Giskard’s claims about comprehensive coverage and managed security remain vendor claims. A buyer should establish a representative evaluation, including expected refusals and legitimate requests that must still work, before treating a test result as a release gate.

07 / DecisionMake every finding lead to an owned decision

Giskard is a useful candidate when a team wants repeatable evidence about an agent’s behaviour and can assign people to interpret and act on it. Begin with a defined application version, a representative dataset and a concrete release decision. The strongest outcome is a set of understood failures and verified fixes, with the remaining limitations visible.