# Amazon S3 + Glue Integration Guide Contents 1. [**Overview**](#h_01GHM6TD30Y2149PW7QEXP4QHN) 2. [**Integration Instructions**](#h_01GHM6TQCWX9WYXA275ZHFQX1G) 1. [Creating the Glue database (managed)](#h_01GHKTRQK21M887H07EAJBP9YE) 2. [Creating the Glue crawler (managed)](#h_01GHKYPHVDM6ZKYA6903TSBND0) [Create the IAM role for Relyance](#h_01GHKYPZCJ19TH0CR69AX99ETH) 3. [Configuring Relyance](#h_01GHM3ECJ7V2DNXCKMVWFHBR4G) 4. [Running the Integration Job](#h_01GDX4NYSSHXSGA6JZQQAV3EKB) 5. [Apply additional Security](#h_01GHVEDFGRMPP1DWE0QKE56YR1) 3. [**Background**](#h_01GHM6TXGF9EZG3VXP9QR0PDQN) 1. [What is Amazon Glue?](#h_01GHM6Z07HHAZ250CMVRAV7BK0) 2. [How does the Relyance integration work?](#h_01GHM6ZAMWPAN1T7965MFEJJRV) 3. [Pricing guide](#h_01GDX4V2WGWFMF0HS6QX86QEVY) ## Overview The Relyance AI integration with Amazon Glue is different from most of our other Vendor integrations. For most of our Vendor & Data Store Integrations, Relyance queries specific API endpoints of third-party SaaS vendor applications to detect and report privacy datatypes being stored in those apps. For the Amazon Glue integration: - Relyance uses Amazon Glue, Amazon's serverless data integration service that makes it easy for analytics users to discover, prepare, move, and integrate data from multiple sources. Amazon Glue will connect to an S3 bucket and search through semi-structured data. It can scan log files, json files with similar objects, parquet files, and [more](https://docs.aws.amazon.com/databrew/latest/dg/supported-data-file-sources.html). - Relyance can trigger the scan of the S3 bucket using an AWS Glue crawler but Relyance will not have read access to the S3 bucket itself. - The results of these scans are then loaded into an ElasticSearch (Lucene) search database; this database of metadata is then accessed by the Relyance service account. No real data therefore leaves the customer's AWS environment, only metadata about detected privacy data types. WARNING: The customer will incur a cost from Amazon for the Glue processing job. Customers need to be aware of the potential cost. Relyance recommends each customer do an initial integration scan using 1 crawler running for <2 hours to get an initial idea of real costs. ## Integration Instructions The Glue integration has multiple operation modes that exist as options in the Relyance AWS Glue configuration settings: ![Glue-1.png](https://assets.relyanceuat.xyz/images/docs/10565024243213/10564222083341.png) 1. **Pre-existing Glue Database (unmanaged)** This option allows you to scan a pre-existing database, where Glue crawls have already been carried out in your environment of a semi-structured data source. The Glue database is therefore not managed by Relyance. In this method, Relyance needs read-only access to the database and ingests and processes whatever info is currently in the database. This option will most likely be used by customers that already use AWS Glue and are familiar with it. In this read-only interaction, the customer must handle the entire process of populating the Glue database. Any Glue crawlers must be managed and run by the customer; this also means they have the freedom to crawl data sources other than S3 bucket such as Dynamo DB, Delta Lake, DocumentDB, and [more](https://docs.aws.amazon.com/glue/latest/dg/crawler-data-stores.html). 2. **S3 Crawl and Scan (managed)** This option allows Relyance to manage the Glue database and Glue crawler. In this method, Relyance will need read and write access to the database and crawler. AWS Glue will also need access to read data from the S3 bucket (or buckets). While Relyance can manage and direct the Glue crawler, it will not have access to read the S3 buckets. This option will most likely be used by customers that have not used AWS Glue before. There are six steps to configuring and then operating the Amazon Glue integration: - [Creating the Glue database (managed)](#h_01GHKTRQK21M887H07EAJBP9YE) - [Creating the Glue crawler (managed)](#h_01GHKYPHVDM6ZKYA6903TSBND0) - [Create the IAM role for Relyance](#h_01GHKYPZCJ19TH0CR69AX99ETH) - [Configuring Relyance](#h_01GHM3ECJ7V2DNXCKMVWFHBR4G) - [Running the integration job](#h_01GDX4NYSSHXSGA6JZQQAV3EKB) - [Apply additional security](#h_01GHVEDFGRMPP1DWE0QKE56YR1) ### Creating the Glue database (managed) In a managed setup, a Glue database must be pre-configured. This database does not need to be populated with anything. *This section can be skipped for an unmanaged configuration*. 1. From the **Databases** tab in the **AWS Glue** console choose **Add database**. 2. Provide a meaningful **Name** and **Description**. 3. Click the **Create database button**. More info can be found here: [https://docs.aws.amazon.com/glue/latest/dg/console-databases.html](https://docs.aws.amazon.com/glue/latest/dg/console-databases.html) ### Creating the Glue crawler (managed) Crawlers are used to inspect the S3 buckets and return results into the ElasticSearch (Lucene) database. *This section can be skipped for an unmanaged configuration*. This crawler doesn't need to be configured with real values: it will be auto-managed by the Relyance integration to use the correct parameters. 1. From the **Crawlers** tab in the *AWS Glue* console choose **Create crawler**. 2. Provide a meaningful **Name** and **Description** and click **Next**. 3. In the **Data source** step, choose ***any*** available datasource. Click the **Next** button. **Note:** The settings in this view do not matter as the crawler will be reconfigured with the settings entered in the Relyance UI. 4. In the *Configure Security settings* step, you'll need to pick an IAM role to use here. This is *not* the role that Relyance uses. Rather, this role is used by AWS Glue itself when reading data to deduce which tables and columns to create. As such, this role must grant: a) read access to the buckets that are to be scanned, and b) read and write access to the database to which it will send output. The easiest way to do this is to use the **Create new IAM role** button. This role can be tweaked to be even less restrictive, but the defaults from the creation button will suffice. Once created, take note of the role's name or **ARN**. The **ARN** will be needed later in the Relyance configuration. 5. Click the **Next** button to continue. 6. In the *Output and Scheduling* screen, make sure you select the correct Glue Target database that you previously created. Click the **Next** button to continue. 7. Review your settings and click the **Create crawler** button. ![Glue-2.png](https://assets.relyanceuat.xyz/images/docs/10565024243213/10566487408013.png) ### Create the IAM role for Relyance Now we'll need to provide Relyance access by creating the IAM role and setting the trust relationship. 1. From the **Roles** tab in the *IAM* console click the **Create role** button. 2. Choose **AWS Account** and click **Next**. 3. In the next screen, you'll be prompted to choose or create a role policy with permissions. Use one of the following policies: **Unmanaged:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "glue:GetDatabase", "glue:GetTableVersion", "glue:GetTables", "glue:GetTableVersions", "glue:GetTable" ], "Resource": [ "arn:aws:glue:<< Region >>:<< Account >>:catalog", "arn:aws:glue:<< Region >>:<< Account >>:table/<< Glue Database >>/*", "arn:aws:glue:<< Region >>:<< Account >>:database/<< Glue Database >>" ] } ] } ``` **Managed:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "glue:GetDatabase", "glue:GetCrawler", "glue:GetTableVersion", "glue:GetTables", "glue:GetTableVersions", "glue:StartCrawler", "glue:UpdateCrawler", "glue:GetTable" ], "Resource": [ "arn:aws:glue:<< Region >>:<< Account>>:catalog", "arn:aws:glue:<< Region >>:<< Account>>:table/<< Glue Database >>/*", "arn:aws:glue:<< Region >>:<< Account>>:database/<< Glue Database >>", "arn:aws:glue:<< Region >>:<< Account>>:crawler/<< Crawler Name >>" ] } ] } ``` Update the policy ARNs with your region, account, and table/database/crawler names in brackets. The region and account number may be replaced with a '*' if you're only using one account and one region. Make sure you give the policy a meaningful name like **GlueReadOnly-<< dbname >>** or **GlueScan-<< dbname >>**. These policies are not Relyance-specific. The configuration setting is just a set of rules and can be added inline to a pre-existing policy. 4. Click **Next** to continue. 5. In the final screen provide a **Role name** (e.g. *RelyanceGlueIntegrationReadOnly* or *RelyanceGlueIntegrationScan* ) and **Description**. 6. Click the **Create role** button. 7. Set the trust relationship by going into your newly configured role's configuration and click on the **Trust Relationships** tab. 8. Click **Edit trust policy**. 9. Update the default trust policy with: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "Statement1", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::580082088342:user/tenant-prod-access" }, "Action": "sts:AssumeRole" } ] } ``` The line with *"AWS": "arn:aws:iam::580082088342:user/tenant-prod-access"* grants permission to the Relyance Service Account access to the resources outlined above. 10. Click the **Update policy** button. More info can be found here: [https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) ### Configuring Relyance 1. Login to your Relyance account. 2. Navigate to the **Settings** Menu in the bottom left-hand side. 3. Select **Integrations**. 4. Click on the **Vendors and Data Stores** filter. 5. Find the **Amazon Glue** integration card and click it to open its connections. 6. On the **Authentication** step, pick the method under **Authentication Method**. If using the *unmanaged* configuration select **Pre-existing Glue Database (IAM Roles)**. If using the *managed* configuration selection **S3 Crawl and Scan (IAM Roles)**. 7. Enter the relevant information in the *Create Connection* form. #### Unmanaged (Pre-existing Glue Database) - Input the **AWS Region**. - Copy the **Relyance Service Role ARN** from the AWS IAM Console. It will be something like ' *arn:aws:iam::123456789:role/RelyanceGlueIntegrationReadOnly* '. - Enter the **Glue Database Name** (not the ARN) of the database to scan. ![Glue-3.png](https://assets.relyanceuat.xyz/images/docs/10565024243213/10569538344333.png) #### Managed (Relyance-managed crawler) - Input the **AWS Region**. - Copy the **Relyance Service Role ARN** from the AWS IAM Console. It will be something like ' *arn:aws:iam::123456789:role/RelyanceGlueIntegration-S3Crawl* '. - Copy the **AWS Glue Service Role ARN** from the AWS IAM Console. It will be something like ' *arn:aws:iam::123456789:role/AWSGlueServieRole-RelyanceIntegrationCrawl* '. - Enter the **Glue Database Name** (not the ARN) the crawler writes to. - Provide the **Crawler Name** (not the ARN) that you used when creating the crawler. - The **Glue/S3 Connector Name** is optional (it may be used in more elaborate setups where custom connectors internal to AWS to bridge Glue and S3 are necessary). - Enter the S3 buckets you wish to scan in the **Buckets to Scan** input box. Make sure you enter one bucket per line. - The **Crawler Run Interval** is separate from the rate Relyance scans are re-run. If a Relyance scan occurs and the last scan time is shorter than this interval time, then the crawler is not triggered and a simple data scan is done (other crawlers may have output data). This is because crawlers can take a long time to complete and do incur cost. This field can take a number of time formats; here are some examples (text to use in bold): - The **Crawl all files** is a significant option to consider. Glue S3 crawlers have an option to either sample the target buckets (up to 249 files per " *prefix* ", or folder), or to look at every single file in every single folder. If it's unticked the sampling is set to to 249 files per folder. If it's ticked, everything is scanned. Scanning everything will be slower to bring results, but it may require trial and error in a specific customer environment to determine whether it actually improves results. The recommendation is to initially leave this setting off to lower processing time and costs; you may re-run it later scanning everything if you wish. 8. Click **Authenticate**. This will trigger the first integration job. ![Glue-4.png](https://assets.relyanceuat.xyz/images/docs/10565024243213/10569538343949.png) ### Running the Integration Job On the Amazon Glue integration configuration page, click the **Save** button to create a job. When **Save** is clicked the AWS Glue integration will take all the information that is provided and create, then start, the first integration job. To add additional AWS Glue integrations, use the **+** icon at the top of the page. ### Apply Additional Security Once the first scan has run, it is suggested to apply additional security to the integration. This is done by using the unique ID that is generated upon the first scan. 1. After the first scan, look at the integration audit logs and look for a single line that looks like this: ``` Authenticated with role arn:aws:iam::12345678:role/RelyanceGlueIntegrationReadOnly. To ensure secure role assumption, make sure the role's trust policy includes: "Condition": { "StringEquals": { "sts:ExternalId": "c6098e1201bb5bdf2ffe1c688700177eccb97125" } }. For more details, see: https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html ``` 2. The important part of this message is: ```json "Condition": { "StringEquals": { "sts:ExternalId": "c6098e1201bb5bdf2ffe1c688700177eccb97125" } } ``` 3. Once you have this, you can go into your AWS IAM console, find the newly created Relyance service role trust relationships, and add the condition clause. The external ID ('c6098e1201bb5bdf2ffe1c688700177eccb97125' in the example above) is unique to each tenant and using it in the trust conditions further hardens the security of the integration. ## Background --- ### What is Amazon Glue? AWS Glue is a serverless data integration service that makes it easy for analytics users to discover, prepare, move, and integrate data from multiple sources. It is an ETL system built to load semi-structured (machine-comprehensible data) into an ElasticSearch (Lucene) search database. For more information see [What is AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html). ### How does the Relyance integration work? Relyance only scans the Glue database for any PII type results. **Managed** When Relyance manages the S3 Scan, it will trigger a Glue crawler to perform the necessary S3 scan using Glue. The crawler configuration is also performed by Relyance. ![AWS_Integration_-_Managed.png](https://assets.relyanceuat.xyz/images/docs/10565024243213/10617464359181.png) **Unmanaged** When Relyance is unmanaged, it will simply scan the Glue database. Any results will need to be created by the customer. ![AWS_Integration_-_Unmanaged.png](https://assets.relyanceuat.xyz/images/docs/10565024243213/10617464353165.png) ### Pricing guide WARNING: The customer will incur a cost from Amazon for the Glue processing job. Customers need to be aware of the potential cost. Relyance recommends each customer do an initial integration scan using 1 crawler running for <2 hours to get an initial idea of real costs. If you have an Amazon account manager you may wish to speak to them about negotiating prices for data processing. ### Verify the connection is really working The two methods need different things, and the S3 permissions are the part that surprises people: - **Scan only** reads an existing Glue Data Catalog. Relyance's role needs the `glue:Get*`/`glue:List*` reads and nothing on S3 — the catalog already holds the schemas. - **Crawl S3** provisions a Glue crawler that reads your buckets. The crawler runs under **your** role (the **Vendor Role ARN** field), not Relyance's, so the S3 read permissions for the target buckets belong to *that* role. This is why no `s3:*` action appears in the permission list on this page. 1. **Crawl S3 completes and finds no tables.** The crawler's own role is missing `s3:GetObject`/`s3:ListBucket` on the target buckets, or the buckets use SSE-KMS and that role has no `kms:Decrypt` on the key. The crawler reports success either way and writes an empty database. 2. **Scan only returns nothing.** **Glue Database Name** must match an existing database in the same region as the connection; a name that does not exist is not an error, just an empty result. 3. **The crawler costs money on every run.** **Crawler Run Interval** controls how often, and **Crawl All** widens what it reads. Both change your AWS bill rather than anything in Relyance. 4. **A failure at Authenticate** is the trust policy or the external ID on the Relyance service role. ## Manage this integration with Terraform Connections for this integration can be managed as code with the [Relyance Terraform provider](https://registry.terraform.io/providers/Relyance/relyance/latest). Non-secret fields go in `auth.params`; secret fields go in `auth.secrets_wo`, which is write-only — never stored in Terraform state. Rotate secrets by bumping `auth.secrets_wo_version`. ### AWS IAM role (scan only) ```hcl resource "relyance_integration_connection" "aws_glue_0" { vendor = "aws_glue" name = "" auth = { method = "iam-role-scan-only" params = { aws_region = "us-east-1" relyance_service_role_arn = "" glue_database_name = "" } } scans = { "property-inspection" = { enabled = true } } } ``` ### AWS IAM role (crawl S3) ```hcl resource "relyance_integration_connection" "aws_glue_1" { vendor = "aws_glue" name = "" auth = { method = "iam-role-crawl-s3" params = { aws_region = "us-east-1" relyance_service_role_arn = "" vendor_role_arn = "" glue_crawler_name = "" glue_database_name = "" crawler_run_interval = "" } # Secret fields are write-only: sent to Relyance, never stored in state. secrets_wo = { s3_target_buckets = var.aws_glue_s3_target_buckets enable_crawl_all = "false" } secrets_wo_version = 1 } scans = { "property-inspection" = { enabled = true } } } ```