Docs
relyance.ai ↗
Browse docs

Onboard an agent

Updated November 17, 2026View as Markdown ↗

Overview

Onboarding adds an agent to Evaluations and Guardrails and runs its first evaluation. It has five steps and takes about 15 minutes. Most of that time is the test run, which does not need your attention.

To start, go to Enforce → Evaluations and Guardrails and select Add an agent.

Step What happens What you do
1. Agent You select the agent to evaluate Select an agent
2. Attack surface Relyance maps the agent's tools, data stores, and attack paths Review the map and answer questions
3. Evals Relyance generates tests for each attack path Review the test plan and cost limit
4. Run Tests run in the sandbox Nothing. The run continues if you leave the page
5. Report Findings and reports are available Send findings to your tracker and download reports

Step 1: Agent

Select the source of the agent:

  • AI inventory. Agents that Relyance discovered through your integrations, such as agent platforms and cloud AI services.
  • Repository. Agents that Relyance discovered in your source code. Relyance reads the agent's configuration and the code of its tools.
  • Traces. Connect a trace source or upload trace files. Relyance reads the tool calls, model calls, and retrieval steps from each trace, and converts recorded sessions into tests.

Step 2: Attack surface

Relyance shows the agent as a blueprint with these parts:

  • Components. The trigger that starts the agent, the model, each tool, and each data store.
  • Classified data. Sensitive data that the agent can read. This uses the data classifications that Relyance already has for your data stores.
  • Untrusted input. Places where the agent reads text that an outside party wrote, such as a customer message, a free-text field, a web page, or a document.
  • Attack paths. Routes from an untrusted input to a sensitive action or to sensitive data. For example: a free-text field that the agent reads, and a tool that sends email.

Relyance cannot read some facts from the agent's configuration, so it asks you to confirm them. For example, it can ask whether a reply is allowed to quote a specific field. Relyance uses your answers to decide what counts as correct behavior in the tests.

Tip

An answer can lead to a follow-up question. Relyance asks follow-up questions before it generates tests. If you do not answer a question, Relyance treats that fact as unknown.

Step 3: Evals

Relyance generates a test plan from the confirmed attack surface. The plan shows:

  • Coverage. The number of test cases for each attack path and attack category. Every attack path has test cases.
  • Datasets. The Relyance Security Index for this agent, and any custom evals that you add.
  • Models. The agent's production model, and any other models that you want to compare.
  • Cost. The number of runs, the maximum possible cost, and the spend cap for each model.

Step 4: Run

Relyance runs the tests in an isolated sandbox. The agent's tools are simulated and the data is synthetic. The run does not call production systems or read production data.

Each test runs several times, with and without Runtime Guardrail. Repeated runs are necessary because an agent can respond differently to the same input.

You can watch the progress or leave the page. The run continues.

Step 5: Report

When the run is complete, Relyance shows:

  • Findings, sorted by severity. Each finding includes the evidence and a recommended mitigation. You can send findings to Jira or GitHub.
  • Reports that you can download: an executive summary, a technical report, OWASP LLM Top 10 coverage, compliance evidence, and the run data as a CSV file.

The agent now appears on the Evaluations and Guardrails page with its scores. Open the Guardrail tab to select mitigations, or the Evals tab to see each test case.

Update or remove an agent

When discovery finds a newer version of an onboarded agent, for example after a change to its prompt, tools, or model, Relyance shows that the agent changed. You can then map and evaluate the new version. Relyance keeps earlier evaluations unchanged, so you can compare the results.

To remove an agent, open the agent's menu and select the remove option.