Evaluations and Guardrails overview
What the feature does
With Evaluations and Guardrails, you test the security of your AI agents and protect them at runtime.
An AI agent reads data, calls tools, and takes actions for a user. This creates risks that a chat model alone does not have. Text from an outside source, such as a customer message or a document, can contain instructions that the agent follows. An agent that can read sensitive data and also send email can be made to send that data to the wrong person.
Relyance finds these risks for each agent, tests them in a sandbox, reports the results, and applies controls in production.
Evaluations and Guardrails feature is in Private Beta and is only available to select customers.
How it works
| Stage | What Relyance does | Result |
|---|---|---|
| Discover | Finds your agents in the AI inventory, in your traces, and in your source code | A list of agents with their models, tools, and data stores |
| Map | Identifies each agent's tools, data stores, sensitive data, and untrusted inputs, and the routes between them | The agent's attack surface and attack paths |
| Evaluate | Generates tests for each attack path and runs them in a sandbox, with and without Runtime Guardrail | Scores for attack success and task completion |
| Protect | Applies Runtime Guardrail to the tool calls and data flows where tests found a problem | Controls in production, with their effect measured in advance |
| Report | Produces executive, technical, and compliance reports | Evidence of adversarial testing for customers and auditors |
Key points
- Tests are specific to each agent. Relyance generates tests from the agent's own tools, data, and instructions. It does not run a fixed, generic test list.
- Security and usefulness are measured together. Each attack test is paired with a normal task, so you can see whether a control stops attacks and whether the agent still does its job.
- Evaluations do not touch production. Tests run in an isolated sandbox with simulated tools and synthetic data. They do not use production data or credentials.
- Controls are tested before you enable them. Relyance replays each mitigation option against the same tests and shows the result.
Where to find it
Go to Enforce → Evaluations and Guardrails. The page lists your onboarded agents and their latest scores. Select Add an agent to onboard an agent, or select an agent to open it.
| Tab | Content |
|---|---|
| Overview | Scores, benchmarks, results by attack category, and model comparison |
| Attack surface | The agent blueprint: components, sensitive data, untrusted inputs, and attack paths |
| Evals | Datasets, experiments, and results for each test case |
| Guardrail | Findings, the mitigation selected for each finding, and approvals |
| Playground | Run scenarios against the agent with guardrails applied |
Related guides
| Guide | Content |
|---|---|
| Onboard an agent | The five onboarding steps |
| How evaluations work | Attack surface, test cases, the sandbox, and scores |
| Runtime Guardrail | Inspection points, mitigation options, and approvals |
| FAQ | Common questions |