# Amazon S3 Via Direct Connection

### Overview

Relyance provides a direct connection for your AWS S3 buckets without the use of a DLP tool like Macie or Glue.

### Supported formats:

| csv | json | parquet | tsv.zst |
| --- | --- | --- | --- |
| csv.gz | json.gz | pdf | txt |
| csv.zst | json.zst | pptx | xls |
| diff | log | rtf | xlsb |
| docx | md | text | xlsm |
| emp | msg | tsv | xlsx |
| feather | ods | tsv.gz | yaml |

Additionally, our intelligent s3 scan supports partitioned datasets (csv, json, parquet).

Parquet files allow Relyance to return the header and schema without requiring the full file download.

### Details

In order to integrate Relyance AI with AWS S3 using direct connection, you will need:

1. Your **AWS** **Account ID**
2. **Role Name** (Name of the role you create in AWS to access S3)
3. **External ID** (External ID you specify in AWS to secure the connection)
4. **Bucket Name/s**
5. **Prefixes** (partitioned data prefixes (e.g. year=2022), leave empty if no partitioned data)

Role assumption in AWS allows entities (like IAM users, services, or even external identities) to take on an IAM role, granting them temporary permissions to perform specific actions on specific resources without having to attach permissions directly to their identity.

By assuming a role, one gets temporary security credentials to act with the permissions of the role, ensuring a flexible and secure way to delegate and segregate duties without distributing long-lived credentials.

There are two necessary preconditions that must be met for a principal (the Relyance AI System, in this case) to assume a role.

1. The entity needs to have permission to use the `sts:AssumeRole` action for the specific role they want to assume.
2. The role must have a trust policy that allows the principal to use it.

### Recommended setup

- **Pick the connection type by bucket size.** Use **S3 Inventory** when the buckets already publish [S3 Inventory reports](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html): Relyance reads the report instead of listing objects, which is faster and cheaper on buckets holding millions of objects. Use **Direct Connection** for everything else.
- **One connection per account and region.** The role, external ID and region belong to a single connection, so a second account or region needs a second connection.
- **Scope with Buckets and Prefixes.** Left empty, a scan covers every bucket the role can reach. Naming the buckets — and prefixes for partitioned data — keeps scans quick and predictable.
- **Grant `kms:Decrypt` if any bucket uses SSE-KMS.** This is the most common reason a connection looks healthy but classifies nothing. See below.

### Create the AWS side with Terraform

The role, its trust policy and its permissions are available as a Terraform
module, which is the least error-prone way to set this up — the policy it writes
is generated from the permissions Relyance actually uses, so it cannot drift from
this page:

```terraform
module "relyance_aws" {
  source  = "assets.relyance.ai/relyance/integration/aws"
  version = "~> 1.0"

  relyance_principal_arns = ["arn:aws:iam::580082088342:user/tenant-prod-access"]
  external_id             = var.relyance_external_id

  integrations = {
    aws_s3 = {
      resource_arns = ["arn:aws:s3:::acme-data-lake"]
      # Only for SSE-KMS buckets. Omit it and no KMS statement is created.
      kms_key_arns = ["arn:aws:kms:us-east-1:111122223333:key/abcd-1234"]
    }
  }
}
```

Set `all_resources = true` instead of `resource_arns` to cover every bucket in
the account — the module requires one or the other, so broad access is never the
silent default. The connection itself can then be created with the
[Relyance provider](https://registry.terraform.io/providers/Relyance/relyance/latest),
as shown at the end of this page.

If you would rather click through the console, the manual equivalent follows.

### Permissions Relyance needs

Two groups, and they fail differently — which is why a scan can look successful and still be missing data.

**Required.** Without these there is no scan, or no content to classify:

| Permission | Used for |
| --- | --- |
| `sts:AssumeRole` | assuming the role you create below, paired with the external ID |
| `s3:ListAllMyBuckets` | discovering which buckets exist |
| `s3:ListBucket` | enumerating objects within a bucket |
| `s3:GetObject` | reading object content to classify it |
| `s3:GetBucketLocation` | addressing the bucket in the right region |
| `kms:Decrypt` | **objects encrypted with SSE-KMS.** S3 decrypts using *your* caller identity, so without this every object read fails and none of that bucket's contents are classified |

**Metadata, and silently optional.** Relyance reads bucket configuration to enrich the asset inventory. A denial here is skipped rather than surfaced: the bucket still appears, the corresponding field is simply blank.

| Permission | What is blank without it |
| --- | --- |
| `s3:GetEncryptionConfiguration` | whether the bucket is encrypted |
| `s3:GetBucketTagging` | bucket tags |
| `s3:GetBucketAcl`, `s3:GetBucketPolicy`, `s3:GetBucketPublicAccessBlock` | who can reach the bucket |
| `s3:GetBucketVersioning`, `s3:GetLifecycleConfiguration` | retention and versioning posture |
| `s3:GetInventoryConfiguration` | inventory reports available on the bucket |
| `s3:GetBucketObjectLockConfiguration` | object-lock (WORM) status |
| `s3:GetBucketLogging`, `s3:GetReplicationConfiguration`, `s3:GetBucketCORS`, `s3:GetBucketWebsite` | logging, replication and exposure details |

### Configuring AWS

### **Create Custom policy**

### **Option 1:**

1. Navigate to the IAM console and create a new policy.
2. Choose **S3** as the policy Service.
3. For **Actions allowed**, choose the following actions:
    1. ListBucket
    2. GetObject
    3. ListAllMyBuckets **If you are configuring an "Asset Discovery" scan (to view via the Relyance Asset Intelligence and DSPM module), you will need the additional actions listed below.**
    4. GetBucketLocation
    5. GetBucketAcl
    6. GetEncryptionConfiguration
    7. GetBucketObjectLockConfiguration
4. Under **Resources**:
    1. Choose **All** for more permissive access (not recommended)
    2. Choose **Specific** to add ARNs of the buckets and objects.
    3. To find out the ARN for your buckets and objects, navigate to **S3**> **Buckets**> **Select Bucket**> **Properties** and copy the ARN for the top level bucket:
    4. You can give access to objects within the bucket by adding * to the object resource name.
5. Name the policy (e.g. RelyanceS3Policy) - and click **Create Policy**.

![Screenshot](https://assets.relyanceuat.xyz/images/docs/25817128014221/25817461653517.png)

![Screenshot](https://assets.relyanceuat.xyz/images/docs/25817128014221/25817461669005.png)

![Screenshot](https://assets.relyanceuat.xyz/images/docs/25817128014221/25817408730125.png)

![Screenshot](https://assets.relyanceuat.xyz/images/docs/25817128014221/25817461708429.png)

![image10.png](https://assets.relyanceuat.xyz/images/docs/25817128014221/25819330494349.png)

**IAM Role:**

1. Select **Roles** on the left-side navigation menu and click **Create Role**.

![image7.png](https://assets.relyanceuat.xyz/images/docs/25817128014221/25819330474893.png)

2. From the options choose **AWS Account**.
3. Enter **580082088342** for the account ID (This is Relyance’s Account ID) and choose a value for the **External ID**.  The external ID is a secret that you will use and enter in the Relyance platform for additional security. While it could be anything, we recommend using a secret generator such as [https://www.uuidgenerator.net/version4](https://www.uuidgenerator.net/version4) to generate.
4. Grant permissions for: Add the previously created customer policy (e.g. RelyanceS3Policy)
5. Provide a meaningful Role name (e.g. RelyanceS3ConnectionRole)
6. Create the role.
7. Once the role is created, click on the **Trust Relationships** tab and update the trust policy to the following:
```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::580082088342:user/tenant-prod-access"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<< external ID >>"
        }
      }
    }
  ]
}
```
Ensure you replace '<< external ID >>' with the external ID you created previously.

![image3.png](https://assets.relyanceuat.xyz/images/docs/25817128014221/25819330508429.png)

### **Option 2: Use Json Directly**

Here’s a ready-to-use policy template. Replace `<bucket-name>` with the actual name of your S3 bucket.

#### How to Use:

![Screenshot](https://assets.relyanceuat.xyz/images/docs/25817128014221/37132803264525.png)
1. Go to the **IAM Console**→ **Policies**.
2. Click **Create policy**.
3. Choose the **JSON** tab.
4. Paste the above JSON (with bucket name updated).
5. Click **Next** and complete the policy creation.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketObjectLockConfiguration",
        "s3:GetObject",
        "s3:ListAllMyBuckets",
        "s3:GetEncryptionConfiguration",
        "s3:GetInventoryConfiguration",
        "s3:ListBucket",
        "s3:GetBucketAcl",
        "s3:GetBucketLocation",
        "s3:GetBucketTagging"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<bucket-name>/*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Sid": "DecryptKmsEncryptedObjects",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": [
        "arn:aws:kms:<region>:<account-id>:key/<key-id>"
      ]
    }
  ]
}
```

Drop the `DecryptKmsEncryptedObjects` statement only if none of the buckets in scope
use SSE-KMS. List the key ARNs the buckets are encrypted with — one entry per key.

#### In the Relyance AI application:

1. Login to your Relyance account.
2. Navigate to the **Settings** Menu in the bottom left-hand side.
3. Select **Integrations**.
4. Click on the **Vendor** filter.
5. Find the **S3** integration card and click it to open its connections.
6. Under **Authentication Method**, choose **Direct Connection**.
8. Select **Asset Discovery** if you want to see the results in your Asset Intelligence Module.
9. Select the **Enable Data Inspection** checkbox if you wish Relyance also to inspect the data. **Note**: The Min Likelihood sets the analysis sensitivity of the data inspection. Lower likelihoods (unlikely) provide more coverage but may generate false positives, while higher sensitivity (very likely) provides more accuracy but may have less coverage.
10. Paste these **settings** into their respective fields:
    1. **Account ID**: Your AWS Account ID
    2. **Role Name**: Name of the role you created in AWS
    3. **External ID**: The **`<secretkey>`** you specified in the trusted relationship External ID
    4. **Region**: Specify your AWS region (the region where the bucket is located)
    5. **Sampling Percent**: Choose a number between 1-100 to specify the percentage of the dataset that you want to sample
    6. To specify Bucket Names and Prefixes, use the JSON editor. You can specify the **Buckets** and **Prefixes** in *json* format such as:
```json
[
  {
    "bucket": "customer-checkout-data",
    "prefixes": [
      "year=2023/",
      "year=2024/"
    ]
  },
  {
    "bucket": "customer-payment-data",
    "prefixes": [
      "source=mobile/"
    ]
  },
  {
    "bucket": "market-research-partnerships"
  }
]
```
If there are no prefixes, leave the value empty. E.g.
"prefixes": [""]

![The Authentication step, with Direct Connection and S3 Inventory offered as Authentication Method cards](https://assets.relyanceuat.xyz/images/docs/ui/aws_s3/06-wizard-authentication.png)

11. Click **Authenticate**.
12. The connection appears in the list for this integration, with its scan status:

![The connections list for Amazon S3](https://assets.relyanceuat.xyz/images/docs/ui/aws_s3/02-connections.png)

13. Congratulations, you are now connected to your AWS S3 bucket.

### Verify the connection is really working

A connection can reach **Connected** and still classify nothing, because object reads
fail per object rather than at connect time. After the first scan completes:

1. Check the buckets you named appear as assets. If none do, the role cannot list —
   revisit `s3:ListAllMyBuckets` and the trust policy's external ID.
2. Open a bucket you know holds classifiable data and confirm data types were found.
   Assets present but **no** data types on an SSE-KMS bucket is the `kms:Decrypt`
   signature.
3. Check the encryption and tag fields are populated. Blank fields on a bucket you
   know is encrypted or tagged means a metadata permission is missing — the scan will
   not have reported an error for it.

### Common pitfalls

- **SSE-KMS buckets without `kms:Decrypt`.** Every object read fails, the bucket still
  appears in the inventory, and nothing in the UI says why. This is the single most
  frequent cause of an S3 connection that looks fine and finds nothing.
- **Metadata denials are invisible.** A missing `s3:GetEncryptionConfiguration`,
  `s3:GetBucketTagging` or similar is skipped silently, so an incomplete policy shows
  up as an asset with missing fields, not as a failure.
- **External ID mismatch.** The external ID in the role's trust policy must match the
  one entered in Relyance exactly. A mismatch fails at `sts:AssumeRole`, so the
  connection never reaches Connected.
- **Wrong region.** The region on the connection must be the bucket's region.
- **A bucket policy that denies.** An explicit `Deny` in the bucket policy (or an SCP)
  overrides the role's `Allow`, so the role's permissions alone are not sufficient.
- **Empty Buckets and Prefixes.** Left empty this scans every reachable bucket, which
  is rarely what you want on a broad role.

Open a connection to check what Relyance is authorised for and what the last scan
actually covered — the **Scopes and Permissions** and **Last Scan Status** panels
are where a missing permission shows up:

![A connection's detail view, showing scopes, endpoints and per-resource scan status](https://assets.relyanceuat.xyz/images/docs/ui/aws_s3/08-connection-detail.png)

<!-- terraform-examples:begin (generated from the integration catalog; do not hand-edit) -->

## Manage this integration with Terraform

Connections for this integration can be managed as code with the [Relyance Terraform provider](https://registry.terraform.io/providers/Relyance/relyance/latest). Non-secret fields go in `auth.params`; secret fields go in `auth.secrets_wo`, which is write-only — never stored in Terraform state. Rotate secrets by bumping `auth.secrets_wo_version`.

### AWS IAM role — Direct Connection

```hcl
resource "relyance_integration_connection" "aws_s3_0" {
  vendor = "aws_s3"
  name   = "<your connection name>"

  auth = {
    method = "iam-role-direct"
    params = {
      account_id = "<account_id>"
      role_name = "<role_name>"
      region = "<region>"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      external_id = var.aws_s3_external_id
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}
```

### AWS IAM role — S3 Inventory

```hcl
resource "relyance_integration_connection" "aws_s3_1" {
  vendor = "aws_s3"
  name   = "<your connection name>"

  auth = {
    method = "iam-role-s3-inventory"
    params = {
      account_id = "<account_id>"
      role_name = "<role_name>"
      region = "<region>"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      external_id = var.aws_s3_external_id
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}
```

<!-- terraform-examples:end -->