Amazon S3 + Macie Integration Guide

Contents

1. Overview
2. Integration Instructions
   1. Configuring Amazon S3 buckets
   2. Configuring Amazon Macie and the role
   3. Configuring Relyance
   4. Running the Integration Job
3. Background
   1. What is Amazon Macie?
   2. How does the Relyance integration work?
   3. Pricing guide
   4. Increasing Macie data processing limits
   5. Explanation of sampling

Overview

The Relyance AI integration with Amazon S3 + Macie is different from most of our other Vendor integrations.

For most of our Vendor & Data Store Integrations, Relyance queries specific API endpoints of third-party SaaS vendor applications to detect and report privacy datatypes being stored in those apps.

For the Amazon S3 unstructured integration:

• Relyance uses Amazon Macie, Amazon’s own data-classification tool, to do the scanning in the customer’s AWS S3 bucket and then send metadata about privacy data types back to the Relyance UI.
• Many types of data files can be read rather than requiring vendor app-specific integrations (i.e., “unstructured”).

WARNING: The customer will incur a cost from Amazon for the Macie processing job. The Relyance integration makes use of sampling techniques to minimize the size, but customers need to be aware of the potential cost. Relyance recommends each customer do an initial integration scan of a bucket no larger than 3TB in size to get a real cost value.

Integration Instructions

There are four steps to configuring and then operating the Amazon S3 unstructured data integration:

• Configuring Amazon S3 buckets.
• Configuring Amazon Macie and the role.
• Configuring Relyance.
• Running the integration job.

Configuring Amazon S3 buckets

1. Background if needed: Getting started with Amazon Macie.
2. Enable Macie.
3. Create a S3 bucket to store Macie discovery results.
   1. Storing and retaining sensitive data discovery results with Amazon Macie - Amazon Macie
   2. Ensure you have proper permissions as denoted in the link on step “a”. You will need certain permissions for Macie, S3, and KMS.
   3. The bucket will need to be encrypted with an AWS KMS Key. It can be an existing Key or a new one. The Key will need to give Macie permission to use it. Add the following to the Key’s policy, replacing the '111122223333' and Region with your account and desired region respectively. Key policy | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | { "Sid": "Allow Macie to use the key", "Effect": "Allow", "Principal": { "Service": "macie.amazonaws.com" }, "Action": [ "kms:GenerateDataKey", "kms:Encrypt" ], "Resource": "", "Condition": { "StringEquals": { "aws:SourceAccount": "111122223333" }, "ArnLike": { "aws:SourceArn": [ "arn:aws:macie2:Region:111122223333:export-configuration:", "arn:aws:macie2:Region:111122223333:classification-job/*" ] } } } |
4. Once the Key is set up, you can create a bucket or use an existing one to store results. To create a new bucket for Macie findings, go to Macie → Settings → Discovery Results.
   1. Choose Create bucket. The name must be unique.
   2. Under KMS Encryption choose the key you created or want to use.

Configuring Amazon Macie and the role

1. Navigate to Identity Access Manager (IAM) in Amazon.
2. On the next page click Create Policy.
3. Select Macie as the service and the following permissions:

4. Two additional permissions will need to be selected in the “write” section which will require a bit more work. These are the CreateClassificationJob and CreateCustomIdentifier permissions.

5. You will see red text in the Resources section saying you need to specify resources.
6. Click on Add ARN and input the region and account number for your account and check the Any box for Resource ID. Do the same for CreateCustomIdentifier.

7. Macie creates a Resource ID upon creating a job or identifier, which is why we select any. Here is an example of an ARN for a job:

8. Click Next to go to the next page. This is the Tag page which can be ignored for now. Click Next.
9. Give the policy a name and Create.
10. Select Roles on the left side menu and click Create Role.

11. From the options choose AWS Account.

12. Enter 580082088342 for the account ID (This is Relyance’s Account ID) and choose a value for the External ID.

13. On the Add Permissions page, add the policy you just created, and click Next.
14. Give the role a name and Create.
15. Go back to the Roles page and go to the Role you just created. Click on the Trust Relationships tab.
16. Click on Edit Trust Policy.
17. Change the line with: “Principal”: { “AWS”: "arn:aws:iam::580082088342:root" } to “Principal”: {"AWS": "arn:aws:iam::580082088342:user/tenant-prod-access" }

Configuring Relyance

1. Login to your Relyance account.

2. Navigate to the Settings Menu in the bottom left-hand side.

3. Select Integrations.

4. Click on the Vendors and Data Stores filter.

5. Find the Amazon S3 integration card and click it to open its connections.

6. Under Authentication Method, choose AWS Macie.

7. Input the values from the previous sections for the Role ARN of the role you created, the external ID you created and the Region. For the Job name field a unique name will be required in order to run and for the Bucket name field input the name of a bucket of data you wish to process.

   1. Multiple buckets can be read if they are separated by commas
   2. Ensure there are no spaces in this field, either in bucket names or between names and commas.

   

Running the Integration Job

1. On the Amazon S3 integration configuration page, click the Connect button to create a Macie job for each bucket. When Save is clicked Macie will take all the information that is provided (Job name and bucket names) and create, then start, a Classification job on those buckets. It will be named JobName_BucketName. When Rescan is clicked (and also once a day automatically) Relyance will query the findings that Macie has generated and stored. We then process them and display them in Relyance. No jobs are triggered from this action, and no new Macie costs are incurred; Relyance is simply reading data that has already been populated. To add additional S3 buckets, use the + icon at the top of the Amazon S3 integration page to add further integrations.

Background

What is Amazon Macie?

Amazon Macie is a data security and data privacy service that uses machine learning and pattern matching to help you discover, monitor, and protect sensitive data in your AWS environment. You can read more at What is Amazon Macie?

Macie can detect the following data type categories in data files:

For more details on each of these data types, see Using managed data identifiers in Amazon Macie.

The Relyance implementation uses a larger data type taxonomy than is available by default in Macie, however. We do this by supplying Macie with additional data identifiers for it to use in its analysis; for more information see Building custom data identifiers in Amazon Macie.

How does the Relyance integration work?

The Relyance integration for unstructured data in an Amazon S3 bucket works by using Amazon Macie as a proxy. Relyance does not see the customer’s S3 data directly, it stays safely and securely in its bucket. Instead, Relyance creates a Macie job, with additional custom data types beyond the Macie defaults; Macie runs its data classification job on the unstructured data in the S3 bucket with all processing happening on-prem in the customer’s Amazon environment; and Macie then passes the data type metadata back to relyance for use and presentation in the Relyance AI platform.

The following table lists and describes the types of file and storage formats that Macie can analyze to detect sensitive data. For each supported type, the table also lists the applicable file name extensions.

Macie does not analyze data in images or audio, video, and other types of multimedia content.

For more information see Supported file and storage formats in Amazon Macie.

Pricing guide

WARNING: The customer will incur a cost from Amazon for the Macie processing job. The Relyance integration makes use of sampling techniques to minimize the size, but customers need to be aware of the potential cost. Relyance recommends each customer do an initial integration scan of a bucket no larger than 3TB in size to get a real cost value.

If you have an Amazon account manager you may wish to speak to them about negotiating prices for data processing.

Increasing Macie data processing limits

If, after initial runs, you believe that you might need to increase your Macie data processing limits you should speak with your Amazon Account Manager. You can read more about this at Amazon Macie Quotas.

Explanation of sampling

For speed and minimization of AWS costs, this integration uses sampling on the Macie end.

Macie allows you to specify the percentage of eligible S3 objects that you want a sensitive data discovery job to analyze. If this value is less than 100%, Macie selects eligible objects to analyze at random, up to the specified percentage, and analyzes all the data in those objects. For example, if you configure a job to analyze 10,000 objects and you specify a sampling depth of 20%, the job analyzes approximately 2,000 randomly selected, eligible objects.

The sampling percentage in this integration is calculated automatically based on the size of the customer’s S3 buckets to get a sample size as close as possible to 100GB. So if a customer has a 200GB bucket, the integration will automatically configure sampling at 50%.