# Google BigQuery ![BigQuery And Its Amazing Features(Try ...](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877076937357.png) BigQuery is a managed, serverless data warehouse product by Google, offering scalable analysis over large quantities of data. It is a Platform as a Service (PaaS) that supports querying using a dialect of SQL. Google BigQuery can be connected with Relyance AI using the following Authentication methods: - [Service Account](#h_01K6DPFQZVM2VMCTJTQTFMCRZK) The customer will incur a cost from Google for the DLP processing job. The Relyance integration makes use of sampling techniques to minimize the size, but customers need to be aware of the potential cost. - [Direct Connection](#01K6DPMS50T9BKHXBM7ZZVSB3E) The customer will not incur any additional costs from Google for DLP processing jobs since Relyance uses its own DLP service ### Which method to choose | Method | Inspection runs in | Credential | Cost | | --- | --- | --- | --- | | **Direct Connection** | Relyance | service-account **impersonation** — no key file | no Google DLP charge | | Service Account | Google Sensitive Data Protection (DLP) | service-account **key JSON** | you pay Google for the data DLP processes | **Direct Connection is the simpler choice** unless you specifically need findings produced by Google's own DLP service: it needs no key file to create, store or rotate, and no findings table. Choose Service Account when your organisation requires Google-native inspection — and expect the Google bill that comes with it. ### Authentication Method - Service Account: The **Service Account** method involves setting up a dedicated Google account, or service account, to let Relyance AI access your BigQuery data. You'd use this primarily because it allows you to utilize **Google's Sensitive Data Protection** (formerly DLP) service to inspect your data. This is great if your organization needs to rely on Google's native security and compliance tools. The crucial point here is that using this option means you'll have to **pay Google** directly for the data processed by their Sensitive Data Protection service. #### Prerequisites: - Google's **Data Loss Prevention** (AKA Sensitive Data Protection) (DLP API) needs to be enabled. See [https://cloud.google.com/sensitive-data-protection/docs/sensitive-data-protection-overview](https://cloud.google.com/sensitive-data-protection/docs/sensitive-data-protection-overview) - Google BigQuery needs to be enabled. This is where your DLP results will be found. See [https://cloud.google.com/bigquery](https://cloud.google.com/bigquery) - **BigQuery Storage API** needs to be enabled. See [https://cloud.google.com/bigquery](https://cloud.google.com/bigquery) #### In Google Cloud Console: 1. Login to your Google console 2. In GCP's **IAM & Admin** page click **Roles** on the left sidebar and then click **+** **CREATE ROLE** 3. Fill out the role with the following values: **Title**: Provide a meaningful name **ID**: Provide an ID to reference this role by **Permissions**: - bigquery.datasets.get - bigquery.jobs.create - bigquery.jobs.get - bigquery.tables.list - bigquery.tables.get - bigquery.tables.getData - dlp.jobs.create - dlp.jobs.get 4. Click **CREATE** to create the new role 5. Once the role is created, click **Service Accounts** in the left-hand **IAM & Admin** menu 6. Create a new service account by clicking the **+** **CREATE SERVICE ACCOUNT** link at the top 7. Fill out the service account details: ![GoogleCloudStorage-1.png](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877076938765.png) ![GoogleCloudStorage-2.png](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877087860493.png) ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/40005674965261.png) If using DLP in a project other than the Project you are saving the results to please see the appendix . 8. Once filled out, click **CREATE AND CONTINUE** 9. Step 2 will give you the option to select a role. Select the role previously created. 10. Click **DONE** 11. Once done, add keys to the newly created service account. In the service account settings, click the **KEYS** tab 12. Click the **ADD KEY** dropdown and select **Create new key** 13. Select **JSON** as the key type and click **CREATE** 14. This will prompt you to download the JSON file. Save this file in a secure location 15. Now go to **BigQuery** in GCP 16. Using the relevant workspace create a new BigQuery dataset for your results. Find your project in the explorer list and click the three dots beside it. Select **Create dataset** 17. Create a new dataset and provide a meaningful name (e.g. dlp_results) 18. Find that newly created dataset in your list, click on the three dots, and select **Share > Manage Permissions** 19. In the resulting page, click **ADD PRINCIPAL** 20. This will open up a side dialog to specify what Principal and roles should be used for this dataset The BigQuery Admin role is what allows us to query the data and this permission is restricted to only this dataset. - **Principal**: Use the service account you created previously. - **Role**: Add *BigQuery Admin* and *Bigquery MetadataViewer* ![GoogleCloudStorage-3.png](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877087861517.png) #### In the Relyance AI application: 1. Login to your Relyance AI account. 2. Navigate to the **Settings** (bottom-left corner). 3. Select **Integrations**. 4. Search and locate the **Google BigQuery** integration card and click on it. 5. Click on the **Add Connection** button on the top right. 6. Provide a meaningful name for the integration and click on the **Add** button. 7. In the **Overview** section, select the integration features you wish to enable for the integration, review the **Scope and Permission**, and **Endpoint** details and click on **Continue**. For more details, see[Integration Features](/docs/introduction-to-relyance-ai/integration-features/). 8. In the **Connection** section, provide the appropriate values and click on **Continue**, - **Connection Name**: This property allows you to update the integration specified in Step 6. If you have multiple integrations for the same vendor, you may want to assign distinct names to each. This helps streamline filtering by **Discovery Source** across the **Inventory**, **Visual** **Maps**, **Assets**, and **Data Flow Analysis** pages. - **Rescan Frequency**: This property allows you to configure how often Relyance executes scans against this Vendor connection. - **Business Atlas Associations** (required): the business entities or products that newly discovered third parties, services and assets from this integration are attributed to. Choose at least one from the **Select Associations** dropdown — the wizard will not advance past this step without one. For more details, see [Business Atlas](/docs/other-settings/business-atlas-overview/). 9. In the **Authentication** section, provide the following information retrieved from the **In Google Cloud Console** section and click on **Continue**. - **Service Account Details** — the contents of the service-account key JSON. - **Project ID to scan** — the project holding the datasets to scan. - **Store findings in this table** — fully qualified, `ProjectID.DatasetID.TableID`. - **Optional Project ID that runs DLP Analysis** — only when DLP runs in a different project from the one storing findings; see the note at the end of this page for the grants that setup needs. - **DLP Config Options** — which datasets and tables are in scope, and how the DLP job is tuned: ```json { "allow_list_datasets": [], "allow_list_tables": [], "deny_list_datasets": ["your_dataset_name"], "max_rows": "10", "processing_location": "" } ``` ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/40005122645517.png) The table name should be provided as a fully qualified table name {DATASET_ID}.{TABLE_NAME} 10. This step applies only if the **Data Inspection** check box was enabled in Step 7. In the Data Inspection section, provide the appropriate values and click on **Continue**, - **Sampling Percentage:** This property adjusts the percentage of records using the table's row count (1000 records max) to return from a table. 11. Review the configuration summary from the **Completion** section and click on **Finish.** 12. Confirm the integration Status reflects **Connected** ### Authentication Method - Direct Connection: The **Direct Connection** method also uses a service account for BigQuery access, but it relies on **Relyance AI's own internal DLP service** for the data inspection, completely bypassing Google's Sensitive Data Protection service. You would choose this option specifically to **avoid the usage-based fees from Google** for running their data inspection jobs. Since Relyance AI handles the DLP processing internally, you won't incur those extra Google Cloud costs, making it a more cost-effective choice. #### In Google Cloud Console: 1. Login to your Google console 2. In GCP's **IAM & Admin** page click **Roles** on the left sidebar and then click **+** **CREATE ROLE** 3. Fill out the role with the following values: **Title**: Provide a meaningful name **ID**: Provide an ID to reference this role by **Permissions**: - bigquery.datasets.get - bigquery.tables.list - bigquery.tables.get - bigquery.tables.getData (reading rows; `tables.get` returns only metadata) - bigquery.jobs.create (we need this to execute queries) 4. Click **CREATE** to create the new role 5. Once the role is created, click **Service Accounts** in the left-hand **IAM & Admin** menu 6. Create a new service account by clicking the **+** **CREATE SERVICE ACCOUNT** link at the top 7. Fill out the service account details: The service account ID needs to be in the format: `rely--int` 8. Once filled out, click **CREATE AND CONTINUE** 9. Step 2 will give you the option to select a role. Select the role previously created. 10. Click **DONE** 11. Go to your created service account and go to the tab “Principal with Access” and click on “+ Grant Access” 12. **Paste** the Relyance service account (tenant-gcp-integrations@relyance-prod.iam.gserviceaccount.com) on “New principals”, select “Workload Identity User” as the role and click on **Save.** 13. Find that dataset you want to scan in your list, click on the three dots, and select **Share>Manage Permissions** 14. In the resulting page, click **ADD PRINCIPAL** 15. This will open up a side dialog to specify what Principal and roles should be used for this dataset The BigQuery Admin role is what allows us to query the data and this permission is restricted to only this dataset. - **Principal**: Use the service account you created previously. - **Role**: Add *BigQuery Admin* and *Bigquery MetadataViewer* ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/39977982322573.png) ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/39977982327437.png) ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/40005674973197.png) #### In the Relyance AI application: 1. Login to your Relyance AI account. 2. Navigate to the **Settings** (bottom-left corner). 3. Select **Integrations**. 4. Search and locate the **Google BigQuery** integration card and click on it. 5. Click on the **Add Connection** button on the top right. 6. Provide a meaningful name for the integration and click on the **Add** button. 7. In the **Overview** section, select the integration features you wish to enable for the integration, review the **Scope and Permission**, and **Endpoint** details and click on **Continue**. For more details, see [Integration Features](/docs/introduction-to-relyance-ai/integration-features/). 8. In the **Connection** section, provide the appropriate values and click on **Continue**, - **Connection Name**: This property allows you to update the integration specified in Step 6. If you have multiple integrations for the same vendor, you may want to assign distinct names to each. This helps streamline filtering by **Discovery Source** across the **Inventory**, **Visual** **Maps**, **Assets**, and **Data Flow Analysis** pages. - **Rescan Frequency**: This property allows you to configure how often Relyance executes scans against this Vendor connection. - **Business Atlas Associations** (required): the business entities or products that newly discovered third parties, services and assets from this integration are attributed to. Choose at least one from the **Select Associations** dropdown — the wizard will not advance past this step without one. For more details, see [Business Atlas](/docs/other-settings/business-atlas-overview/). 9. In the **Authentication** section, provide the following information retrieved from the **Google Cloud Console** section and click on **Continue**. - **Service Account Email** - **Project ID** **The table name should be provided as a fully qualified table name {DATASET_ID}.{TABLE_NAME}** 10. This step applies only if the **Data Inspection** check box was enabled in Step 7. In the Data Inspection section, provide the appropriate values and click on **Continue**, - **Sampling Percentage:** This property adjusts the percentage of records using the table's row count (1000 records max) to return from a table. 11. Review the configuration summary from the **Completion** section and click on **Finish.** 12. Confirm the integration Status reflects **Connected** ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/40850334752141.png) ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/40005122642701.png) ![Google_BigQuery_connection_20250224_195002.png](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877087866893.png) ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/39977982331661.png) ![Google_BigQuery_data_inspection_20250224_195011.png](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877076949517.png) ![Google_BigQuery_completion_20250224_195006.png](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877087875981.png) ### Appendix #### Alternate Account If using DLP in a project other than the Project you are saving the results to: 1. On the project where the findings are being saved, you will need to grant the Google managed service account the role DLP API Service Agent on the results dataset 2. First we need to find the Service account. Navigate to the IAM page in your DLP project. In the filter type “dlp”. And on the right side of the page make sure the checkbox for “Include Google-provided role grants” is checked 3. Scroll until you find something that looks like this: ( [service-12345123@dlp-api.iam.gserviceaccount.com](mailto:service-12345123@dlp-api.iam.gserviceaccount.com) ) 4. Copy that value. Navigate to bigquery in the project you are saving the results and for the dataset you are using click the button with 3 dots and click share 5. Add principal, paste in the value of the service account. Assign the role “DLP API Service Agent” 6. The service account created in project B can be granted the role of Bigquery data viewer, or some role that has the ability to read and create jobs on this dataset 7. If ever a new results dataset when integrating then these roles will need to be added to that new dataset as well. This is to limit our access to these specific datasets that only contain the findings of the DLP job. ![Screenshot](https://assets.relyanceuat.xyz/images/docs/34877087883149/34877076951309.png) ### Verify the connection is really working 1. **Datasets and tables appear, but no data types are classified.** This is `bigquery.tables.getData`. Relyance samples rows to classify them — the **Sampling Percentage** field above is exactly that — and `bigquery.tables.get` returns only metadata, so the schema arrives while every query is denied. The inventory looks complete and the classification is empty. 2. **The role is easier to get right with predefined roles.** Instead of the custom role, `roles/bigquery.dataViewer` plus `roles/bigquery.jobUser` covers everything the Direct Connection method needs — dataViewer carries `tables.get`, `tables.list` and `tables.getData`, jobUser carries `jobs.create`. That is what the Terraform module below grants. 3. **A failure at Authenticate on Direct Connection** is the impersonation grant: `Workload Identity User` must be on the service account itself, not on the project. 4. **A DLP job that never writes findings.** The findings dataset must exist before the scan, and when DLP runs in a different project from the results project, both grants in the note above are required. ## Manage this integration with Terraform Connections for this integration can be managed as code with the [Relyance Terraform provider](https://registry.terraform.io/providers/Relyance/relyance/latest). Non-secret fields go in `auth.params`; secret fields go in `auth.secrets_wo`, which is write-only — never stored in Terraform state. Rotate secrets by bumping `auth.secrets_wo_version`. ### Account credentials ```hcl resource "relyance_integration_connection" "gcloud_bigquery_0" { vendor = "gcloud_bigquery" name = "" auth = { method = "account-credentials" # NOTE: account_details carries credentials (private_key) but is not flagged # secret, so it goes in params and IS written to Terraform state in # cleartext. Keep state in an encrypted remote backend with restricted # access, and rotate the credential if state was ever committed. params = { account_details = jsonencode({ auth_provider_x509_cert_url = "" auth_uri = "" client_email = "" client_id = "" client_x509_cert_url = "" private_key = "" private_key_id = "" project_id = "" token_uri = "" type = "service_account" }) project_id_scan = "example-project" save_path = "ProjectID.DatasetID.TableID" config_options = jsonencode({ allow_list_datasets = [] allow_list_tables = [] deny_list_datasets = [ "your_dataset_name" ] max_rows = "10" processing_location = "" }) } } scans = { "data-inspection" = { enabled = true } } } ``` ### Service account ```hcl resource "relyance_integration_connection" "gcloud_bigquery_1" { vendor = "gcloud_bigquery" name = "" auth = { method = "service-account" params = { config_options = jsonencode({ allow_list_datasets = [] allow_list_tables = [] deny_list_datasets = [ "your_dataset_name" ] }) } # Secret fields are write-only: sent to Relyance, never stored in state. secrets_wo = { service_account_email = var.gcloud_bigquery_service_account_email project_id_scan = var.gcloud_bigquery_project_id_scan } secrets_wo_version = 1 } scans = { "data-inspection" = { enabled = true } } } ```