Amazon S3 Via Direct Connection

Overview

Relyance provides a direct connection for your AWS S3 buckets without the use of a DLP tool like Macie or Glue.

Supported formats:

Additionally, our intelligent s3 scan supports partitioned datasets (csv, json, parquet).

Parquet files allow Relyance to return the header and schema without requiring the full file download.

Details

In order to integrate Relyance AI with AWS S3 using direct connection, you will need:

1. Your AWS Account ID
2. Role Name (Name of the role you create in AWS to access S3)
3. External ID (External ID you specify in AWS to secure the connection)
4. Bucket Name/s
5. Prefixes (partitioned data prefixes (e.g. year=2022), leave empty if no partitioned data)

Role assumption in AWS allows entities (like IAM users, services, or even external identities) to take on an IAM role, granting them temporary permissions to perform specific actions on specific resources without having to attach permissions directly to their identity.

By assuming a role, one gets temporary security credentials to act with the permissions of the role, ensuring a flexible and secure way to delegate and segregate duties without distributing long-lived credentials.

There are two necessary preconditions that must be met for a principal (the Relyance AI System, in this case) to assume a role.

1. The entity needs to have permission to use the sts:AssumeRole action for the specific role they want to assume.
2. The role must have a trust policy that allows the principal to use it.

Recommended setup

• Pick the connection type by bucket size. Use S3 Inventory when the buckets already publish S3 Inventory reports: Relyance reads the report instead of listing objects, which is faster and cheaper on buckets holding millions of objects. Use Direct Connection for everything else.
• One connection per account and region. The role, external ID and region belong to a single connection, so a second account or region needs a second connection.
• Scope with Buckets and Prefixes. Left empty, a scan covers every bucket the role can reach. Naming the buckets — and prefixes for partitioned data — keeps scans quick and predictable.
• Grant kms:Decrypt if any bucket uses SSE-KMS. This is the most common reason a connection looks healthy but classifies nothing. See below.

Create the AWS side with Terraform

The role, its trust policy and its permissions are available as a Terraform module, which is the least error-prone way to set this up — the policy it writes is generated from the permissions Relyance actually uses, so it cannot drift from this page:

module "relyance_aws" {
  source  = "assets.relyance.ai/relyance/integration/aws"
  version = "~> 1.0"

  relyance_principal_arns = ["arn:aws:iam::580082088342:user/tenant-prod-access"]
  external_id             = var.relyance_external_id

  integrations = {
    aws_s3 = {
      resource_arns = ["arn:aws:s3:::acme-data-lake"]
      # Only for SSE-KMS buckets. Omit it and no KMS statement is created.
      kms_key_arns = ["arn:aws:kms:us-east-1:111122223333:key/abcd-1234"]
    }
  }
}

Set all_resources = true instead of resource_arns to cover every bucket in the account — the module requires one or the other, so broad access is never the silent default. The connection itself can then be created with the Relyance provider, as shown at the end of this page.

If you would rather click through the console, the manual equivalent follows.

Permissions Relyance needs

Two groups, and they fail differently — which is why a scan can look successful and still be missing data.

Required. Without these there is no scan, or no content to classify:

Metadata, and silently optional. Relyance reads bucket configuration to enrich the asset inventory. A denial here is skipped rather than surfaced: the bucket still appears, the corresponding field is simply blank.

Configuring AWS

Create Custom policy

Option 1:

1. Navigate to the IAM console and create a new policy.
2. Choose S3 as the policy Service.
3. For Actions allowed, choose the following actions:
   1. ListBucket
   2. GetObject
   3. ListAllMyBuckets If you are configuring an "Asset Discovery" scan (to view via the Relyance Asset Intelligence and DSPM module), you will need the additional actions listed below.
   4. GetBucketLocation
   5. GetBucketAcl
   6. GetEncryptionConfiguration
   7. GetBucketObjectLockConfiguration
4. Under Resources:
   1. Choose All for more permissive access (not recommended)
   2. Choose Specific to add ARNs of the buckets and objects.
   3. To find out the ARN for your buckets and objects, navigate to S3> Buckets> Select Bucket> Properties and copy the ARN for the top level bucket:
   4. You can give access to objects within the bucket by adding * to the object resource name.
5. Name the policy (e.g. RelyanceS3Policy) - and click Create Policy.

IAM Role:

1. Select Roles on the left-side navigation menu and click Create Role.

2. From the options choose AWS Account.
3. Enter 580082088342 for the account ID (This is Relyance’s Account ID) and choose a value for the External ID. The external ID is a secret that you will use and enter in the Relyance platform for additional security. While it could be anything, we recommend using a secret generator such as https://www.uuidgenerator.net/version4 to generate.
4. Grant permissions for: Add the previously created customer policy (e.g. RelyanceS3Policy)
5. Provide a meaningful Role name (e.g. RelyanceS3ConnectionRole)
6. Create the role.
7. Once the role is created, click on the Trust Relationships tab and update the trust policy to the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::580082088342:user/tenant-prod-access"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<< external ID >>"
        }
      }
    }
  ]
}

Ensure you replace '<< external ID >>' with the external ID you created previously.

Option 2: Use Json Directly

Here’s a ready-to-use policy template. Replace <bucket-name> with the actual name of your S3 bucket.

How to Use:

1. Go to the IAM Console→ Policies.
2. Click Create policy.
3. Choose the JSON tab.
4. Paste the above JSON (with bucket name updated).
5. Click Next and complete the policy creation.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketObjectLockConfiguration",
        "s3:GetObject",
        "s3:ListAllMyBuckets",
        "s3:GetEncryptionConfiguration",
        "s3:GetInventoryConfiguration",
        "s3:ListBucket",
        "s3:GetBucketAcl",
        "s3:GetBucketLocation",
        "s3:GetBucketTagging"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<bucket-name>/*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Sid": "DecryptKmsEncryptedObjects",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": [
        "arn:aws:kms:<region>:<account-id>:key/<key-id>"
      ]
    }
  ]
}

Drop the DecryptKmsEncryptedObjects statement only if none of the buckets in scope use SSE-KMS. List the key ARNs the buckets are encrypted with — one entry per key.

In the Relyance AI application:

1. Login to your Relyance account.
2. Navigate to the Settings Menu in the bottom left-hand side.
3. Select Integrations.
4. Click on the Vendor filter.
5. Find the S3 integration card and click it to open its connections.
6. Under Authentication Method, choose Direct Connection.
7. Select Asset Discovery if you want to see the results in your Asset Intelligence Module.
8. Select the Enable Data Inspection checkbox if you wish Relyance also to inspect the data. Note: The Min Likelihood sets the analysis sensitivity of the data inspection. Lower likelihoods (unlikely) provide more coverage but may generate false positives, while higher sensitivity (very likely) provides more accuracy but may have less coverage.
9. Paste these settings into their respective fields:
   1. Account ID: Your AWS Account ID
   2. Role Name: Name of the role you created in AWS
   3. External ID: The <secretkey> you specified in the trusted relationship External ID
   4. Region: Specify your AWS region (the region where the bucket is located)
   5. Sampling Percent: Choose a number between 1-100 to specify the percentage of the dataset that you want to sample
   6. To specify Bucket Names and Prefixes, use the JSON editor. You can specify the Buckets and Prefixes in json format such as:

[
  {
    "bucket": "customer-checkout-data",
    "prefixes": [
      "year=2023/",
      "year=2024/"
    ]
  },
  {
    "bucket": "customer-payment-data",
    "prefixes": [
      "source=mobile/"
    ]
  },
  {
    "bucket": "market-research-partnerships"
  }
]

If there are no prefixes, leave the value empty. E.g. "prefixes": [""]

11. Click Authenticate.
12. The connection appears in the list for this integration, with its scan status:

13. Congratulations, you are now connected to your AWS S3 bucket.

Verify the connection is really working

A connection can reach Connected and still classify nothing, because object reads fail per object rather than at connect time. After the first scan completes:

1. Check the buckets you named appear as assets. If none do, the role cannot list — revisit s3:ListAllMyBuckets and the trust policy's external ID.
2. Open a bucket you know holds classifiable data and confirm data types were found. Assets present but no data types on an SSE-KMS bucket is the kms:Decrypt signature.
3. Check the encryption and tag fields are populated. Blank fields on a bucket you know is encrypted or tagged means a metadata permission is missing — the scan will not have reported an error for it.

Common pitfalls

• SSE-KMS buckets without kms:Decrypt. Every object read fails, the bucket still appears in the inventory, and nothing in the UI says why. This is the single most frequent cause of an S3 connection that looks fine and finds nothing.
• Metadata denials are invisible. A missing s3:GetEncryptionConfiguration, s3:GetBucketTagging or similar is skipped silently, so an incomplete policy shows up as an asset with missing fields, not as a failure.
• External ID mismatch. The external ID in the role's trust policy must match the one entered in Relyance exactly. A mismatch fails at sts:AssumeRole, so the connection never reaches Connected.
• Wrong region. The region on the connection must be the bucket's region.
• A bucket policy that denies. An explicit Deny in the bucket policy (or an SCP) overrides the role's Allow, so the role's permissions alone are not sufficient.
• Empty Buckets and Prefixes. Left empty this scans every reachable bucket, which is rarely what you want on a broad role.

Open a connection to check what Relyance is authorised for and what the last scan actually covered — the Scopes and Permissions and Last Scan Status panels are where a missing permission shows up:

Manage this integration with Terraform

Connections for this integration can be managed as code with the Relyance Terraform provider. Non-secret fields go in auth.params; secret fields go in auth.secrets_wo, which is write-only — never stored in Terraform state. Rotate secrets by bumping auth.secrets_wo_version.

AWS IAM role — Direct Connection

resource "relyance_integration_connection" "aws_s3_0" {
  vendor = "aws_s3"
  name   = "<your connection name>"

  auth = {
    method = "iam-role-direct"
    params = {
      account_id = "<account_id>"
      role_name = "<role_name>"
      region = "<region>"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      external_id = var.aws_s3_external_id
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}

AWS IAM role — S3 Inventory

resource "relyance_integration_connection" "aws_s3_1" {
  vendor = "aws_s3"
  name   = "<your connection name>"

  auth = {
    method = "iam-role-s3-inventory"
    params = {
      account_id = "<account_id>"
      role_name = "<role_name>"
      region = "<region>"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      external_id = var.aws_s3_external_id
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}