# FAQ: Evaluations and Guardrails

## General

### What is Evaluations and Guardrails?

Evaluations and Guardrails is a Relyance feature that tests the security of AI agents and protects them at runtime. It maps each agent, runs attack tests in a sandbox, reports scores, and applies controls in production. See the [overview](/docs/evaluations-and-guardrails/overview/).

### What is an eval?

An eval (evaluation) is a test of agent behavior. Each test case pairs an attack with a normal task and runs several times. The output is a set of scores. See [How evaluations work](/docs/evaluations-and-guardrails/how-evaluations-work/).

### How is this different from a model benchmark?

A benchmark measures a model in a general environment. Relyance measures your agent: the model together with your instructions, tools, and data. Two agents that use the same model can have different risks, because the risk depends on the tools and data that each agent can reach. Relyance shows public benchmark results next to your results for reference.

### How is this different from a penetration test?

A penetration test is a manual test at one point in time. Relyance runs the same tests again after each change and on each model, and records every run. You can use both.

## Safety and data

### Do evaluations attack my production agent?

No. Evaluations run in an isolated sandbox. Tools are simulated, so an evaluation does not send, write, or delete anything in a real system.

### Do evaluations use my production data?

No. The sandbox uses synthetic data with the same structure as your data. The tests use metadata that Relyance already has, such as which columns contain sensitive data, to make the tests realistic.

### Do I need to change my agent?

No. Relyance reads the agent's existing configuration or code. An evaluation does not require changes to your code or prompts.

## Results

### What do the three scores mean?

- **Successful attacks**: how often an attack achieved its goal.
- **Tasks completed**: how often the agent completed the user's legitimate task.
- **Privilege escalations**: how often the agent took an action beyond the requester's permissions.

Relyance shows each score with and without Runtime Guardrail, with a Pass, Warn, or Fail result.

### Why does each test run more than once?

An agent can respond differently to the same input. Repeated runs give a rate instead of a single result.

### Why is each attack paired with a normal task?

A control that stops attacks can also stop legitimate work. The pairing shows both effects, so you can select a mitigation that keeps the agent useful.

### What if an agent cannot be tested?

Some agents have no attack path. For example, no untrusted input reaches a sensitive tool or sensitive data. In this case, Relyance reports that the agent is not testable and gives the reason. If information about the agent is missing, Relyance lists what is missing.

### Can I add my own tests?

Yes. Describe a rule in plain language, and Relyance generates the test cases and scoring. You can also promote sessions from your production traces into tests.

## Protection

### What does Runtime Guardrail do?

It inspects the request, each tool call, each tool result, and the reply. It applies the mitigation that you selected for each finding: monitor, sanitize, redact, require approval, or block. See [Runtime Guardrail](/docs/evaluations-and-guardrails/runtime-guardrail/).

### Does a guardrail stop legitimate work?

Relyance replays each mitigation in the sandbox before you enable it, so you can see its effect on attacks and on normal tasks. Sanitize and redact allow the run to complete. Block stops the call.

### Can a person approve actions?

Yes. Set a mitigation to **Require approval**. Runtime Guardrail holds the call until an approver responds in the app, in Slack, or through a webhook.

## Compliance

### Does this feature help with AI regulations and frameworks?

Each evaluation produces a compliance evidence report. The report relates the testing to robustness and security requirements in the EU AI Act, ISO/IEC 42001, and the NIST AI RMF, and shows coverage of the OWASP Top 10 for LLM applications. Map the report to your own controls before an audit.

### Who can run evaluations?

Viewing results and starting runs are separate permissions. Ask your Relyance administrator for access.
