Databricks

Databricks is a cloud-based data engineering tool used for processing and transforming massive quantities of data and exploring the data through machine learning models. Developed by the creators of Apache Spark, it provides a unified, open platform for all your data.

In order to integrate Databricks with Relyance AI you will need a Databricks Instance URL and Personal Access Token.

Token based authentication must be enabled by the administrator.

In Databricks:

1. The Databricks instance URL will be the custom Databricks URL for your organization.
2. Click Settings in the lower left corner of your Databricks workspace.
3. Click User Settings.
4. Go to the Access Tokens tab.
5. Click the Generate New Token button.
6. Click the Generate button and copy the generated token.

In the Relyance AI application:

1. Login to your Relyance account.
2. Navigate to the Settings Menu in the bottom left-hand side.
3. Select Integrations.
4. Find the Databricks integration card and click on it.
5. In the Databricks integration page, click Add Connection.
6. Follow the wizard, adjusting any defaults as needed.
7. Choose Custom for the Authentication Method.
8. Paste the Databricks Instance URL and Personal Access Token into their respective fields. You can also specify the Clusters and SQL Warehouses to scan. For Asset Discovery its important and necessary to provide the warehouse or cluster name.

9. Click Authenticate and finish the wizard.
10. At this point, you should see the following result on the Vendor Integrations page:
11. Congratulations, you are now connected to Databricks.

If the connection reports Connected but returns nothing

These are the ways this integration comes back empty without reporting an error. Generated from the integration catalog, so it tracks what the connection actually asks for.

1. A credential rotated at the vendor is not picked up here. Personal Access Token is stored when you save the connection, so regenerating the value at the vendor breaks the next scan until it is re-pasted here. Recording the expiry on the connection means Relyance warns you before it lapses.
2. Check the address fields before suspecting the credentials. Databricks Instance URL identifies which tenant, region or host to talk to. A wrong value there fails authentication and looks exactly like a bad secret.

Authentication methods and fields

Pick one of these under Authentication Method on the connection wizard's Authentication step. This table is generated from the integration catalog, so it always matches what the form actually asks for.

Manage this integration with Terraform

Connections for this integration can be managed as code with the Relyance Terraform provider. Non-secret fields go in auth.params; secret fields go in auth.secrets_wo, which is write-only — never stored in Terraform state. Rotate secrets by bumping auth.secrets_wo_version.

resource "relyance_integration_connection" "databricks" {
  vendor = "databricks"
  name   = "<your connection name>"

  auth = {
    method = "api-token"
    params = {
      instance_url = "<instance_url>"
      clusters = jsonencode({
        cluster_names = [
          ""
        ]
      })
      sql_warehouses = jsonencode({
        sql_warehouse_names = [
          ""
        ]
      })
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      api_token = var.databricks_api_token
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}