Google BigQuery

BigQuery is a managed, serverless data warehouse product by Google, offering scalable analysis over large quantities of data. It is a Platform as a Service (PaaS) that supports querying using a dialect of SQL.

Google BigQuery can be connected with Relyance AI using the following Authentication methods:

• Service Account The customer will incur a cost from Google for the DLP processing job. The Relyance integration makes use of sampling techniques to minimize the size, but customers need to be aware of the potential cost.
• Direct Connection The customer will not incur any additional costs from Google for DLP processing jobs since Relyance uses its own DLP service

Which method to choose

Direct Connection is the simpler choice unless you specifically need findings produced by Google's own DLP service: it needs no key file to create, store or rotate, and no findings table. Choose Service Account when your organisation requires Google-native inspection — and expect the Google bill that comes with it.

Authentication Method - Service Account:

The Service Account method involves setting up a dedicated Google account, or service account, to let Relyance AI access your BigQuery data. You'd use this primarily because it allows you to utilize Google's Sensitive Data Protection (formerly DLP) service to inspect your data. This is great if your organization needs to rely on Google's native security and compliance tools. The crucial point here is that using this option means you'll have to pay Google directly for the data processed by their Sensitive Data Protection service.

Prerequisites:

• Google's Data Loss Prevention (AKA Sensitive Data Protection) (DLP API) needs to be enabled. See https://cloud.google.com/sensitive-data-protection/docs/sensitive-data-protection-overview
• Google BigQuery needs to be enabled. This is where your DLP results will be found. See https://cloud.google.com/bigquery
• BigQuery Storage API needs to be enabled. See https://cloud.google.com/bigquery

In Google Cloud Console:

1. Login to your Google console
2. In GCP's IAM & Admin page click Roles on the left sidebar and then click + CREATE ROLE
3. Fill out the role with the following values: Title: Provide a meaningful name ID: Provide an ID to reference this role by Permissions:
   • bigquery.datasets.get
   • bigquery.jobs.create
   • bigquery.jobs.get
   • bigquery.tables.list
   • bigquery.tables.get
   • bigquery.tables.getData
   • dlp.jobs.create
   • dlp.jobs.get
4. Click CREATE to create the new role
5. Once the role is created, click Service Accounts in the left-hand IAM & Admin menu
6. Create a new service account by clicking the + CREATE SERVICE ACCOUNT link at the top
7. Fill out the service account details:

If using DLP in a project other than the Project you are saving the results to please see the appendix

. 8. Once filled out, click CREATE AND CONTINUE 9. Step 2 will give you the option to select a role. Select the role previously created. 10. Click DONE 11. Once done, add keys to the newly created service account. In the service account settings, click the KEYS tab 12. Click the ADD KEY dropdown and select Create new key 13. Select JSON as the key type and click CREATE 14. This will prompt you to download the JSON file. Save this file in a secure location 15. Now go to BigQuery in GCP 16. Using the relevant workspace create a new BigQuery dataset for your results. Find your project in the explorer list and click the three dots beside it. Select Create dataset 17. Create a new dataset and provide a meaningful name (e.g. dlp_results) 18. Find that newly created dataset in your list, click on the three dots, and select Share > Manage Permissions 19. In the resulting page, click ADD PRINCIPAL 20. This will open up a side dialog to specify what Principal and roles should be used for this dataset The BigQuery Admin role is what allows us to query the data and this permission is restricted to only this dataset. - Principal: Use the service account you created previously. - Role: Add BigQuery Admin and Bigquery MetadataViewer

In the Relyance AI application:

1. Login to your Relyance AI account.

2. Navigate to the Settings (bottom-left corner).

3. Select Integrations.

4. Search and locate the Google BigQuery integration card and click on it.

5. Click on the Add Connection button on the top right.

6. Provide a meaningful name for the integration and click on the Add button.

7. In the Overview section, select the integration features you wish to enable for the integration, review the Scope and Permission, and Endpoint details and click on Continue. For more details, seeIntegration Features.

8. In the Connection section, provide the appropriate values and click on Continue,

   • Connection Name: This property allows you to update the integration specified in Step 6. If you have multiple integrations for the same vendor, you may want to assign distinct names to each. This helps streamline filtering by Discovery Source across the Inventory, Visual Maps, Assets, and Data Flow Analysis pages.
   • Rescan Frequency: This property allows you to configure how often Relyance executes scans against this Vendor connection.
   • Business Atlas Associations (required): the business entities or products that newly discovered third parties, services and assets from this integration are attributed to. Choose at least one from the Select Associations dropdown — the wizard will not advance past this step without one. For more details, see Business Atlas.

9. In the Authentication section, provide the following information retrieved from the In Google Cloud Console section and click on Continue.

   • Service Account Details — the contents of the service-account key JSON.

   • Project ID to scan — the project holding the datasets to scan.

   • Store findings in this table — fully qualified, ProjectID.DatasetID.TableID.

   • Optional Project ID that runs DLP Analysis — only when DLP runs in a different project from the one storing findings; see the note at the end of this page for the grants that setup needs.

   • DLP Config Options — which datasets and tables are in scope, and how the DLP job is tuned:

     {
       "allow_list_datasets": [],
       "allow_list_tables": [],
       "deny_list_datasets": ["your_dataset_name"],
       "max_rows": "10",
       "processing_location": ""
     }

   

The table name should be provided as a fully qualified table name {DATASET_ID}.{TABLE_NAME} 10. This step applies only if the Data Inspection check box was enabled in Step 7. In the Data Inspection section, provide the appropriate values and click on Continue, - Sampling Percentage: This property adjusts the percentage of records using the table's row count (1000 records max) to return from a table. 11. Review the configuration summary from the Completion section and click on Finish. 12. Confirm the integration Status reflects Connected

Authentication Method - Direct Connection:

The Direct Connection method also uses a service account for BigQuery access, but it relies on Relyance AI's own internal DLP service for the data inspection, completely bypassing Google's Sensitive Data Protection service. You would choose this option specifically to avoid the usage-based fees from Google for running their data inspection jobs. Since Relyance AI handles the DLP processing internally, you won't incur those extra Google Cloud costs, making it a more cost-effective choice.

In Google Cloud Console:

1. Login to your Google console

2. In GCP's IAM & Admin page click Roles on the left sidebar and then click + CREATE ROLE

3. Fill out the role with the following values: Title: Provide a meaningful name ID: Provide an ID to reference this role by Permissions:

   • bigquery.datasets.get
   • bigquery.tables.list
   • bigquery.tables.get
   • bigquery.tables.getData (reading rows; tables.get returns only metadata)
   • bigquery.jobs.create (we need this to execute queries)

4. Click CREATE to create the new role

5. Once the role is created, click Service Accounts in the left-hand IAM & Admin menu

6. Create a new service account by clicking the + CREATE SERVICE ACCOUNT link at the top

7. Fill out the service account details: The service account ID needs to be in the format: rely-<your-tenant-subdomain>-int

8. Once filled out, click CREATE AND CONTINUE

9. Step 2 will give you the option to select a role. Select the role previously created.

10. Click DONE

11. Go to your created service account and go to the tab “Principal with Access” and click on “+ Grant Access”

12. Paste the Relyance service account (tenant-gcp-integrations@relyance-prod.iam.gserviceaccount.com) on “New principals”, select “Workload Identity User” as the role and click on Save.

13. Find that dataset you want to scan in your list, click on the three dots, and select Share>Manage Permissions

14. In the resulting page, click ADD PRINCIPAL

15. This will open up a side dialog to specify what Principal and roles should be used for this dataset The BigQuery Admin role is what allows us to query the data and this permission is restricted to only this dataset.

    • Principal: Use the service account you created previously.
    • Role: Add BigQuery Admin and Bigquery MetadataViewer

    

    

    

In the Relyance AI application:

1. Login to your Relyance AI account.
2. Navigate to the Settings (bottom-left corner).
3. Select Integrations.
4. Search and locate the Google BigQuery integration card and click on it.
5. Click on the Add Connection button on the top right.
6. Provide a meaningful name for the integration and click on the Add button.
7. In the Overview section, select the integration features you wish to enable for the integration, review the Scope and Permission, and Endpoint details and click on Continue. For more details, see Integration Features.
8. In the Connection section, provide the appropriate values and click on Continue,
   • Connection Name: This property allows you to update the integration specified in Step 6. If you have multiple integrations for the same vendor, you may want to assign distinct names to each. This helps streamline filtering by Discovery Source across the Inventory, Visual Maps, Assets, and Data Flow Analysis pages.
   • Rescan Frequency: This property allows you to configure how often Relyance executes scans against this Vendor connection.
   • Business Atlas Associations (required): the business entities or products that newly discovered third parties, services and assets from this integration are attributed to. Choose at least one from the Select Associations dropdown — the wizard will not advance past this step without one. For more details, see Business Atlas.
9. In the Authentication section, provide the following information retrieved from the Google Cloud Console section and click on Continue.
   • Service Account Email
   • Project ID The table name should be provided as a fully qualified table name {DATASET_ID}.{TABLE_NAME}
10. This step applies only if the Data Inspection check box was enabled in Step 7. In the Data Inspection section, provide the appropriate values and click on Continue,
    • Sampling Percentage: This property adjusts the percentage of records using the table's row count (1000 records max) to return from a table.
11. Review the configuration summary from the Completion section and click on Finish.
12. Confirm the integration Status reflects Connected

Appendix

Alternate Account

If using DLP in a project other than the Project you are saving the results to:

1. On the project where the findings are being saved, you will need to grant the Google managed service account the role DLP API Service Agent on the results dataset
2. First we need to find the Service account. Navigate to the IAM page in your DLP project. In the filter type “dlp”. And on the right side of the page make sure the checkbox for “Include Google-provided role grants” is checked
3. Scroll until you find something that looks like this: ( service-12345123@dlp-api.iam.gserviceaccount.com )
4. Copy that value. Navigate to bigquery in the project you are saving the results and for the dataset you are using click the button with 3 dots and click share
5. Add principal, paste in the value of the service account. Assign the role “DLP API Service Agent”
6. The service account created in project B can be granted the role of Bigquery data viewer, or some role that has the ability to read and create jobs on this dataset
7. If ever a new results dataset when integrating then these roles will need to be added to that new dataset as well. This is to limit our access to these specific datasets that only contain the findings of the DLP job.

Verify the connection is really working

1. Datasets and tables appear, but no data types are classified. This is bigquery.tables.getData. Relyance samples rows to classify them — the Sampling Percentage field above is exactly that — and bigquery.tables.get returns only metadata, so the schema arrives while every query is denied. The inventory looks complete and the classification is empty.
2. The role is easier to get right with predefined roles. Instead of the custom role, roles/bigquery.dataViewer plus roles/bigquery.jobUser covers everything the Direct Connection method needs — dataViewer carries tables.get, tables.list and tables.getData, jobUser carries jobs.create. That is what the Terraform module below grants.
3. A failure at Authenticate on Direct Connection is the impersonation grant: Workload Identity User must be on the service account itself, not on the project.
4. A DLP job that never writes findings. The findings dataset must exist before the scan, and when DLP runs in a different project from the results project, both grants in the note above are required.

Manage this integration with Terraform

Connections for this integration can be managed as code with the Relyance Terraform provider. Non-secret fields go in auth.params; secret fields go in auth.secrets_wo, which is write-only — never stored in Terraform state. Rotate secrets by bumping auth.secrets_wo_version.

Account credentials

resource "relyance_integration_connection" "gcloud_bigquery_0" {
  vendor = "gcloud_bigquery"
  name   = "<your connection name>"

  auth = {
    method = "account-credentials"
    # NOTE: account_details carries credentials (private_key) but is not flagged
    # secret, so it goes in params and IS written to Terraform state in
    # cleartext. Keep state in an encrypted remote backend with restricted
    # access, and rotate the credential if state was ever committed.
    params = {
      account_details = jsonencode({
        auth_provider_x509_cert_url = ""
        auth_uri = ""
        client_email = ""
        client_id = ""
        client_x509_cert_url = ""
        private_key = ""
        private_key_id = ""
        project_id = ""
        token_uri = ""
        type = "service_account"
      })
      project_id_scan = "example-project"
      save_path = "ProjectID.DatasetID.TableID"
      config_options = jsonencode({
        allow_list_datasets = []
        allow_list_tables = []
        deny_list_datasets = [
          "your_dataset_name"
        ]
        max_rows = "10"
        processing_location = ""
      })
    }
  }

  scans = { "data-inspection" = { enabled = true } }
}

Service account

resource "relyance_integration_connection" "gcloud_bigquery_1" {
  vendor = "gcloud_bigquery"
  name   = "<your connection name>"

  auth = {
    method = "service-account"
    params = {
      config_options = jsonencode({
        allow_list_datasets = []
        allow_list_tables = []
        deny_list_datasets = [
          "your_dataset_name"
        ]
      })
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      service_account_email = var.gcloud_bigquery_service_account_email
      project_id_scan = var.gcloud_bigquery_project_id_scan
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}