# Google Cloud Storage ![GoogleCloudStorageLogo.png](https://assets.relyanceuat.xyz/images/docs/14095278636685/14096474661133.png) Google Cloud Storage is a scalable, durable, and highly available object storage service provided by Google. It enables users to store, manage, and access vast amounts of data globally, supporting applications and workloads across various industries and use cases. ### Choosing a connection method This integration offers four, and they divide into two very different models. Pick first, because the prerequisites and the cost are not the same. | Method | How it scans | Needs | | --- | --- | --- | | **Direct Connection** *(recommended)* | Relyance lists and reads objects itself | Service-account impersonation. No key file, no DLP, no BigQuery | | **Inventory Reports** *(recommended for very large buckets)* | Reads published Storage Inventory Reports instead of listing objects | Impersonation, plus inventory reports configured on the buckets | | **Google Cloud Storage** | Hands the scan to Google's DLP service | A service-account **key JSON**, the DLP API, and a BigQuery findings table. **Google charges for DLP processing** | | **BigQuery** | DLP over BigQuery datasets rather than buckets | Same as above | Start with **Direct Connection** unless you specifically want DLP to do the classification: it needs no key material to paste, rotate or leak, and it carries no Google DLP charge. Options 1 and 2 below cover the DLP path and Options 3 and 4 the impersonation path. The DLP options need a **Service Account Details** key file, **Project ID**, **Dataset ID**, **Job Name**, **Bucket Location** and **Sampling** rate. The customer will incur a cost from Google for the DLP processing job. The Relyance integration makes use of sampling techniques to minimize the size, but customers need to be aware of the potential cost. #### Prerequisites: These apply to the DLP methods (Options 1 and 2). Direct Connection and Inventory Reports need none of them — only the impersonation grant described under Option 3. - Google's Data Loss Prevention (DLP) needs to be enabled. See [https://cloud.google.com/dlp](https://cloud.google.com/dlp) - Google BigQuery needs to be enabled. This is where your DLP results will be found. See [https://cloud.google.com/bigquery](https://cloud.google.com/bigquery) - Google's Cloud Storage Bucket needs to be enabled and is the source to be indexed. See [https://cloud.google.com/storage/docs/buckets](https://cloud.google.com/storage/docs/buckets) #### In Google Cloud Storage: 1. Login to your Google console. 2. In GCP's **IAM & Admin** page click **Roles** on the left sidebar and then click **+ CREATE ROLE**. 3. Fill out the role with the following values: **Title**: Provide a meaningful name **ID**: Provide an ID to reference this role by **Permissions**: - bigquery.datasets.get - bigquery.jobs.create - bigquery.tables.list - dlp.jobs.create - bigquery.tables.getData ![GoogleCloudStorage-1.png](https://assets.relyanceuat.xyz/images/docs/14095278636685/14936506805389.png) ![GoogleCloudStorage-2.png](https://assets.relyanceuat.xyz/images/docs/14095278636685/14936811926285.png) 4. Click **CREATE** to create the new role. 5. Once the role is created, click **Service Accounts** in the left-hand **IAM & Admin** menu. 6. Create a new service account by clicking the **+ CREATE SERVICE ACCOUNT** link at the top. 7. Fill out the service account details: 8. Once filled out, click **CREATE AND CONTINUE**. 9. Step 2 will give you the option to select a role. Select the role previously created. 10. Click **DONE**. 11. Once done, add keys to the newly created service account. In the service account settings, click the **KEYS** tab. 12. Click the **ADD KEY** dropdown and select **Create new key**. 13. Select **JSON** as the key type and click **CREATE**. 14. This will prompt you to download the JSON file. Save this file in a secure location. 15. Now go to **BigQuery** in GCP. 16. Using the relevant workspace create a new BigQuery table for your results. Find your project in the explorer list and click the three dots beside it. Select **Create dataset**. 17. Create a new dataset and provide a meaningful name (e.g. dlp_results). 18. Find that newly created dataset in your list, click on the three dots, and select **Share > Manage Permissions**. 19. In the resulting page, click **ADD PRINCIPAL**. 20. This will open up a side dialog to specify what Principal and roles should be used for this dataset. - **Principal**: Use the service account you created previously. - **Role**: Add BigQuery Admin. The BigQuery Admin role is what allows us to query the data and this permission is restricted to only this dataset. ![GoogleCloudStorage-3.png](https://assets.relyanceuat.xyz/images/docs/14095278636685/14955898942221.png) #### In the Relyance AI application: 1. Login to your Relyance account. 2. Navigate to the **Settings** Menu in the bottom left-hand side. 3. Select **Integrations**. 4. Click on the **Vendor Integration** tab. 5. Find the **Google Cloud Storage** integration card and click it to open its connections. 6. On the **Authentication** step, pick the method under **Authentication Method**. 7. **Option 1:** Choose **Google Cloud Storage** under **Authentication Method**. - Enter the following details: - **Service Account Details**: Copy the contents of the JSON Service Account file yodownloaded earlier. - **Save findings to table**: Enter the dataset info in the form of ProjectID.DatasetID.TableID - **Job Name**: The name of the job Relyance will create. - **Bucket Location**: The bucket location of your source data. This needs to be a gs url (e.g. *gs://bucket_of_PII*) - **Sampling**: Put a number for your desired amount of sampling. This takes in integers 1-100. This field if set lower than 100 will randomly scan that percentage of files in the bucket. ![Screenshot](https://assets.relyanceuat.xyz/images/docs/14095278636685/24697588966157.png) - Click **Authenticate**. - At this point, you should see the following result on the **Vendor** Integrations page: - Congratulations, you are now connected to **Google Cloud Storage**. 1. **Option 2:** Choose **BigQuery** under **Authentication Method**. - Enter the following details: - **Service Account Details**: Copy the contents of the JSON Service Account file you downloaded earlier. - **Project ID:** Project ID of the project where you want **to read the data from** - **Save findings to table**: Enter the info in the form of ProjectID.DatasetID.TableID. **If your scan results are stored in a different project,** you can specify that project ID here if needed. Otherwise, the project ID would remain the same. See the [end of the article](#h_01HR663DJ6TYQQ4P8DVB0V124Z) to set up permissions appropriately for this setup - **DLP Config Options**: required — which datasets and tables the DLP job covers, and how it is tuned. Deny wins over allow: ```json { "allow_list_datasets": [], "allow_list_tables": [], "deny_list_datasets": ["your_dataset_name"], "max_rows": "10", "min_likelihood": "likely", "processing_location": "" } ``` - Click **Authenticate**. - At this point, you should see the following result on the **Vendor** Integrations page: ![Screenshot](https://assets.relyanceuat.xyz/images/docs/14095278636685/24697580345741.png) ![GoogleCloudStorage-5.png](https://assets.relyanceuat.xyz/images/docs/14095278636685/14965256826381.png) - Congratulations, you are now connected to **Google Cloud Storage**. ### Option 3: Direct Connection The two options above hand the scan to Google's DLP service, so they need a service-account key JSON and a findings table. **Direct Connection** uses service-account impersonation instead — no key material to paste, rotate or leak — and is the better choice for a new connection. - **Service Account Email** — the account Relyance impersonates, e.g. `rely--int-@relyance-prod.iam.gserviceaccount.com`. Grant `Workload Identity User` on it to `tenant-gcp-integrations@relyance-prod.iam.gserviceaccount.com`, exactly as in the [Google Cloud Platform guide](/docs/vendor-integrations/google-cloud-platform/). - **Project ID to scan** — the project holding the buckets. - **Bucket Configuration** (optional) — scope to specific buckets and prefixes. Left empty, every reachable bucket is in scope. ```json [ { "bucket_name": "", "allow_file_prefixes": [""] } ] ``` ### Option 4: Inventory Reports The same impersonation model as Direct Connection, but it reads [GCS Storage Inventory Reports](https://cloud.google.com/storage/docs/insights/inventory-reports) instead of listing objects — the right choice for buckets holding millions of objects, where listing is slow and expensive. **Service Account Email** and **Project ID to scan** are as above. The **Bucket** Configuration **points at the inventory report and filters what is read from it:** ```json [ { "inventory_bucket_path": "", "source_bucket_allow_list": [""], "source_bucket_block_list": [""], "allow_file_prefixes": [""], "filter_last_accessed_days": "", "storage_class": "STANDARD" } ] ``` Lower **Maximum Concurrent API Calls** if the scan needs to be gentler on the project's API quota. ##### **Note: If using DLP in a project other than the Project you are saving the results to:** 1. On the project where the findings are being saved, you will need to grant the Google managed service account the role DLP API Service Agent on the results dataset. 2. First we need to find the Service account. Navigate to the IAM page in your DLP project. In the filter type “dlp”. And on the right side of the page make sure the checkbox for “Include Google-provided role grants” is checked. 3. Scroll until you find something that looks like this: ( [service-12345123@dlp-api.iam.gserviceaccount.com](mailto:service-12345123@dlp-api.iam.gserviceaccount.com) ) ![Screenshot](https://assets.relyanceuat.xyz/images/docs/14095278636685/24697580349965.png) 1. Copy that value. Navigate to bigquery in the project you are saving the results and for the dataset you are using click the button with 3 dots and click share. 2. Add principal, paste in the value of the service account. Assign the role “DLP API Service Agent”. 3. The service account created in project B can be granted the role of Bigquery data viewer, or some role that has the ability to read and create jobs on this dataset. 4. If ever a new results dataset when integrating then these roles will need to be added to that new dataset as well. This is to limit our access to these specific datasets that only contain the findings of the DLP job. ![The Authentication step for Google Cloud Storage](https://assets.relyanceuat.xyz/images/docs/ui/gcloud_storage/07-wizard-authentication.png) ### Verify the connection is really working The impersonation methods request five permissions, and they fail in ways that look alike from the outside: | Permission | Used for | | --- | --- | | `storage.buckets.list` | discovering which buckets exist in the project | | `storage.buckets.get` | reading a bucket's location and configuration | | `storage.buckets.getIamPolicy` | reporting who can reach the bucket | | `storage.objects.list` | enumerating objects within a bucket | | `storage.objects.get` | reading object content to classify it | `roles/storage.objectViewer` plus `roles/iam.securityAuditor` covers all five, and is what the Terraform module below grants. Watch out for `roles/storage.legacyBucketReader`, which looks like the right role but covers neither `storage.buckets.list` nor `storage.buckets.getIamPolicy`. 1. **A failure at Authenticate** is almost always the impersonation grant. Check that `Workload Identity User` is on the service account itself — its **Principals with access** tab — and not on the project. A grant at project level looks correct in the IAM list and does not work. 2. **Connected, but no buckets appear.** Either `storage.buckets.list` is missing, or **Bucket Configuration** names buckets that do not match. An empty Bucket Configuration means every reachable bucket, so an empty result there points at the permission. 3. **Buckets appear but nothing is classified.** This is `storage.objects.get`. Listing an object needs only `storage.objects.list`, so the inventory looks complete while no content is ever read — the same shape as the SSE-KMS problem on S3, and the most common "connected but empty" report on this integration. 4. **Inventory Reports finds nothing.** The reports have to exist before Relyance can read them: confirm the bucket actually publishes them, that at least one report has been generated since the configuration was created, and that `inventory_bucket_path` points at the report location rather than at the source bucket. 5. **A DLP job fails to write findings.** The findings dataset must already exist, and when DLP runs in a different project from the one storing results, both grants in the note above are required. A missing DLP API Service Agent grant on the results dataset fails the job after the scan has already run. ## Manage this integration with Terraform Connections for this integration can be managed as code with the [Relyance Terraform provider](https://registry.terraform.io/providers/Relyance/relyance/latest). Non-secret fields go in `auth.params`; secret fields go in `auth.secrets_wo`, which is write-only — never stored in Terraform state. Rotate secrets by bumping `auth.secrets_wo_version`. ### Account credentials — Google Cloud Storage ```hcl resource "relyance_integration_connection" "gcloud_storage_0" { vendor = "gcloud_storage" name = "" auth = { method = "account-credentials-google-cloud-storage" # NOTE: account_details carries credentials (private_key) but is not flagged # secret, so it goes in params and IS written to Terraform state in # cleartext. Keep state in an encrypted remote backend with restricted # access, and rotate the credential if state was ever committed. params = { account_details = jsonencode({ auth_provider_x509_cert_url = "" auth_uri = "" client_email = "" client_id = "" client_x509_cert_url = "" private_key = "" private_key_id = "" project_id = "" token_uri = "" type = "service_account" }) save_path = "ProjectID.DatasetID.TableID" job_name = "" bucket = "" sampling_rate = "1" } } scans = { "data-inspection" = { enabled = true } } } ``` ### Account credentials — BigQuery ```hcl resource "relyance_integration_connection" "gcloud_storage_1" { vendor = "gcloud_storage" name = "" auth = { method = "account-credentials-big-query" # NOTE: account_details carries credentials (private_key) but is not flagged # secret, so it goes in params and IS written to Terraform state in # cleartext. Keep state in an encrypted remote backend with restricted # access, and rotate the credential if state was ever committed. params = { account_details = jsonencode({ auth_provider_x509_cert_url = "" auth_uri = "" client_email = "" client_id = "" client_x509_cert_url = "" private_key = "" private_key_id = "" project_id = "" token_uri = "" type = "service_account" }) project_id_scan = "" save_path = "ProjectID.DatasetID.TableID" config_options = jsonencode({ allow_list_datasets = [] allow_list_tables = [] deny_list_datasets = [ "your_dataset_name" ] max_rows = "10" min_likelihood = "likely" processing_location = "" }) } } scans = { "data-inspection" = { enabled = true } } } ``` ### Service account — Direct Connection ```hcl resource "relyance_integration_connection" "gcloud_storage_2" { vendor = "gcloud_storage" name = "" auth = { method = "service-account-direct" # Secret fields are write-only: sent to Relyance, never stored in state. secrets_wo = { service_account_email = var.gcloud_storage_service_account_email project_id_scan = var.gcloud_storage_project_id_scan } secrets_wo_version = 1 } scans = { "data-inspection" = { enabled = true } } } ``` ### Service account — Inventory Reports ```hcl resource "relyance_integration_connection" "gcloud_storage_3" { vendor = "gcloud_storage" name = "" auth = { method = "service-account-inventory-reports" # Secret fields are write-only: sent to Relyance, never stored in state. secrets_wo = { service_account_email = var.gcloud_storage_service_account_email project_id_scan = var.gcloud_storage_project_id_scan } secrets_wo_version = 1 } scans = { "data-inspection" = { enabled = true } } } ```