Google Cloud Storage

Google Cloud Storage is a scalable, durable, and highly available object storage service provided by Google. It enables users to store, manage, and access vast amounts of data globally, supporting applications and workloads across various industries and use cases.

Choosing a connection method

This integration offers four, and they divide into two very different models. Pick first, because the prerequisites and the cost are not the same.

Start with Direct Connection unless you specifically want DLP to do the classification: it needs no key material to paste, rotate or leak, and it carries no Google DLP charge. Options 1 and 2 below cover the DLP path and Options 3 and 4 the impersonation path.

The DLP options need a Service Account Details key file, Project ID, Dataset ID, Job Name, Bucket Location and Sampling rate. The customer will incur a cost from Google for the DLP processing job. The Relyance integration makes use of sampling techniques to minimize the size, but customers need to be aware of the potential cost.

Prerequisites:

These apply to the DLP methods (Options 1 and 2). Direct Connection and Inventory Reports need none of them — only the impersonation grant described under Option 3.

• Google's Data Loss Prevention (DLP) needs to be enabled. See https://cloud.google.com/dlp
• Google BigQuery needs to be enabled. This is where your DLP results will be found. See https://cloud.google.com/bigquery
• Google's Cloud Storage Bucket needs to be enabled and is the source to be indexed. See https://cloud.google.com/storage/docs/buckets

In Google Cloud Storage:

1. Login to your Google console.
2. In GCP's IAM & Admin page click Roles on the left sidebar and then click + CREATE ROLE.
3. Fill out the role with the following values: Title: Provide a meaningful name ID: Provide an ID to reference this role by Permissions:

• bigquery.datasets.get
• bigquery.jobs.create
• bigquery.tables.list
• dlp.jobs.create
• bigquery.tables.getData

4. Click CREATE to create the new role.

5. Once the role is created, click Service Accounts in the left-hand IAM & Admin menu.

6. Create a new service account by clicking the + CREATE SERVICE ACCOUNT link at the top.

7. Fill out the service account details:

8. Once filled out, click CREATE AND CONTINUE.

9. Step 2 will give you the option to select a role. Select the role previously created.

10. Click DONE.

11. Once done, add keys to the newly created service account. In the service account settings, click the KEYS tab.

12. Click the ADD KEY dropdown and select Create new key.

13. Select JSON as the key type and click CREATE.

14. This will prompt you to download the JSON file. Save this file in a secure location.

15. Now go to BigQuery in GCP.

16. Using the relevant workspace create a new BigQuery table for your results. Find your project in the explorer list and click the three dots beside it. Select Create dataset.

17. Create a new dataset and provide a meaningful name (e.g. dlp_results).

18. Find that newly created dataset in your list, click on the three dots, and select Share > Manage Permissions.

19. In the resulting page, click ADD PRINCIPAL.

20. This will open up a side dialog to specify what Principal and roles should be used for this dataset.

    • Principal: Use the service account you created previously.
    • Role: Add BigQuery Admin. The BigQuery Admin role is what allows us to query the data and this permission is restricted to only this dataset.

    

In the Relyance AI application:

1. Login to your Relyance account.
2. Navigate to the Settings Menu in the bottom left-hand side.
3. Select Integrations.
4. Click on the Vendor Integration tab.
5. Find the Google Cloud Storage integration card and click it to open its connections.
6. On the Authentication step, pick the method under Authentication Method.
7. Option 1: Choose Google Cloud Storage under Authentication Method.

• Enter the following details:
• Service Account Details: Copy the contents of the JSON Service Account file yodownloaded earlier.
• Save findings to table: Enter the dataset info in the form of ProjectID.DatasetID.TableID
• Job Name: The name of the job Relyance will create.
• Bucket Location: The bucket location of your source data. This needs to be a gs url (e.g. gs://bucket_of_PII)
• Sampling: Put a number for your desired amount of sampling. This takes in integers 1-100. This field if set lower than 100 will randomly scan that percentage of files in the bucket.

• Click Authenticate.

• At this point, you should see the following result on the Vendor Integrations page:

• Congratulations, you are now connected to Google Cloud Storage.

1. Option 2: Choose BigQuery under Authentication Method.

• Enter the following details:

• Service Account Details: Copy the contents of the JSON Service Account file you downloaded earlier.

• Project ID: Project ID of the project where you want to read the data from

• Save findings to table: Enter the info in the form of ProjectID.DatasetID.TableID. If your scan results are stored in a different project, you can specify that project ID here if needed. Otherwise, the project ID would remain the same. See the end of the article to set up permissions appropriately for this setup

• DLP Config Options: required — which datasets and tables the DLP job covers, and how it is tuned. Deny wins over allow:

  {
    "allow_list_datasets": [],
    "allow_list_tables": [],
    "deny_list_datasets": ["your_dataset_name"],
    "max_rows": "10",
    "min_likelihood": "likely",
    "processing_location": ""
  }

• Click Authenticate.

• At this point, you should see the following result on the Vendor Integrations page:

• Congratulations, you are now connected to Google Cloud Storage.

Option 3: Direct Connection

The two options above hand the scan to Google's DLP service, so they need a service-account key JSON and a findings table. Direct Connection uses service-account impersonation instead — no key material to paste, rotate or leak — and is the better choice for a new connection.

• Service Account Email — the account Relyance impersonates, e.g. rely-<tenant_name>-int-<suffix>@relyance-prod.iam.gserviceaccount.com. Grant Workload Identity User on it to tenant-gcp-integrations@relyance-prod.iam.gserviceaccount.com, exactly as in the Google Cloud Platform guide.
• Project ID to scan — the project holding the buckets.
• Bucket Configuration (optional) — scope to specific buckets and prefixes. Left empty, every reachable bucket is in scope.

[
  {
    "bucket_name": "",
    "allow_file_prefixes": [""]
  }
]

Option 4: Inventory Reports

The same impersonation model as Direct Connection, but it reads GCS Storage Inventory Reports instead of listing objects — the right choice for buckets holding millions of objects, where listing is slow and expensive.

Service Account Email and Project ID to scan are as above. The Bucket Configuration points at the inventory report and filters what is read from it:

[
  {
    "inventory_bucket_path": "",
    "source_bucket_allow_list": [""],
    "source_bucket_block_list": [""],
    "allow_file_prefixes": [""],
    "filter_last_accessed_days": "",
    "storage_class": "STANDARD"
  }
]

Lower Maximum Concurrent API Calls if the scan needs to be gentler on the project's API quota.

Note: If using DLP in a project other than the Project you are saving the results to:

1. On the project where the findings are being saved, you will need to grant the Google managed service account the role DLP API Service Agent on the results dataset.
2. First we need to find the Service account. Navigate to the IAM page in your DLP project. In the filter type “dlp”. And on the right side of the page make sure the checkbox for “Include Google-provided role grants” is checked.
3. Scroll until you find something that looks like this: ( service-12345123@dlp-api.iam.gserviceaccount.com )

1. Copy that value. Navigate to bigquery in the project you are saving the results and for the dataset you are using click the button with 3 dots and click share.
2. Add principal, paste in the value of the service account. Assign the role “DLP API Service Agent”.
3. The service account created in project B can be granted the role of Bigquery data viewer, or some role that has the ability to read and create jobs on this dataset.
4. If ever a new results dataset when integrating then these roles will need to be added to that new dataset as well. This is to limit our access to these specific datasets that only contain the findings of the DLP job.

Verify the connection is really working

The impersonation methods request five permissions, and they fail in ways that look alike from the outside:

roles/storage.objectViewer plus roles/iam.securityAuditor covers all five, and is what the Terraform module below grants. Watch out for roles/storage.legacyBucketReader, which looks like the right role but covers neither storage.buckets.list nor storage.buckets.getIamPolicy.

1. A failure at Authenticate is almost always the impersonation grant. Check that Workload Identity User is on the service account itself — its Principals with access tab — and not on the project. A grant at project level looks correct in the IAM list and does not work.
2. Connected, but no buckets appear. Either storage.buckets.list is missing, or Bucket Configuration names buckets that do not match. An empty Bucket Configuration means every reachable bucket, so an empty result there points at the permission.
3. Buckets appear but nothing is classified. This is storage.objects.get. Listing an object needs only storage.objects.list, so the inventory looks complete while no content is ever read — the same shape as the SSE-KMS problem on S3, and the most common "connected but empty" report on this integration.
4. Inventory Reports finds nothing. The reports have to exist before Relyance can read them: confirm the bucket actually publishes them, that at least one report has been generated since the configuration was created, and that inventory_bucket_path points at the report location rather than at the source bucket.
5. A DLP job fails to write findings. The findings dataset must already exist, and when DLP runs in a different project from the one storing results, both grants in the note above are required. A missing DLP API Service Agent grant on the results dataset fails the job after the scan has already run.

Manage this integration with Terraform

Connections for this integration can be managed as code with the Relyance Terraform provider. Non-secret fields go in auth.params; secret fields go in auth.secrets_wo, which is write-only — never stored in Terraform state. Rotate secrets by bumping auth.secrets_wo_version.

Account credentials — Google Cloud Storage

resource "relyance_integration_connection" "gcloud_storage_0" {
  vendor = "gcloud_storage"
  name   = "<your connection name>"

  auth = {
    method = "account-credentials-google-cloud-storage"
    # NOTE: account_details carries credentials (private_key) but is not flagged
    # secret, so it goes in params and IS written to Terraform state in
    # cleartext. Keep state in an encrypted remote backend with restricted
    # access, and rotate the credential if state was ever committed.
    params = {
      account_details = jsonencode({
        auth_provider_x509_cert_url = ""
        auth_uri = ""
        client_email = ""
        client_id = ""
        client_x509_cert_url = ""
        private_key = ""
        private_key_id = ""
        project_id = ""
        token_uri = ""
        type = "service_account"
      })
      save_path = "ProjectID.DatasetID.TableID"
      job_name = "<job_name>"
      bucket = "<bucket>"
      sampling_rate = "1"
    }
  }

  scans = { "data-inspection" = { enabled = true } }
}

Account credentials — BigQuery

resource "relyance_integration_connection" "gcloud_storage_1" {
  vendor = "gcloud_storage"
  name   = "<your connection name>"

  auth = {
    method = "account-credentials-big-query"
    # NOTE: account_details carries credentials (private_key) but is not flagged
    # secret, so it goes in params and IS written to Terraform state in
    # cleartext. Keep state in an encrypted remote backend with restricted
    # access, and rotate the credential if state was ever committed.
    params = {
      account_details = jsonencode({
        auth_provider_x509_cert_url = ""
        auth_uri = ""
        client_email = ""
        client_id = ""
        client_x509_cert_url = ""
        private_key = ""
        private_key_id = ""
        project_id = ""
        token_uri = ""
        type = "service_account"
      })
      project_id_scan = "<project_id_scan>"
      save_path = "ProjectID.DatasetID.TableID"
      config_options = jsonencode({
        allow_list_datasets = []
        allow_list_tables = []
        deny_list_datasets = [
          "your_dataset_name"
        ]
        max_rows = "10"
        min_likelihood = "likely"
        processing_location = ""
      })
    }
  }

  scans = { "data-inspection" = { enabled = true } }
}

Service account — Direct Connection

resource "relyance_integration_connection" "gcloud_storage_2" {
  vendor = "gcloud_storage"
  name   = "<your connection name>"

  auth = {
    method = "service-account-direct"
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      service_account_email = var.gcloud_storage_service_account_email
      project_id_scan = var.gcloud_storage_project_id_scan
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}

Service account — Inventory Reports

resource "relyance_integration_connection" "gcloud_storage_3" {
  vendor = "gcloud_storage"
  name   = "<your connection name>"

  auth = {
    method = "service-account-inventory-reports"
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      service_account_email = var.gcloud_storage_service_account_email
      project_id_scan = var.gcloud_storage_project_id_scan
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}