# ClickHouse

ClickHouse Cloud is a cloud-based distributed column-oriented database service designed for online analytical processing (OLAP) that allows generation of analytical data reports in real-time. The ClickHouse integration enables Relyance AI to discover and analyze data across your ClickHouse Cloud organizations, services, and metrics.

The ClickHouse integration provides:

- **Organization Discovery**: Scan all organizations, members, and invitations
- **Service Monitoring**: Discover services and their configurations
- **Metrics Analysis**: Extract Prometheus metrics from ClickHouse services
- **Data Inspection**: Analyze data schemas and identify personal information types
- **Service Mapping**: Visualize service-to-service communication flows

### Outpost on AWS

For **In Host BYOK**, complete [Outpost deployment](/docs/introduction-to-relyance-ai/how-outpost-works/) and store the [ClickHouse Cloud credential JSON](/docs/introduction-to-relyance-ai/outpost-connection-credentials/#clickhouse-cloud-basic-auth) in AWS Secrets Manager. Enter only its ARN on the connection. The credential form and Terraform secret-value examples below describe hosted credentials; use the BYOK reference-only setup for Outpost.

Validate two independent access paths from your EKS network:

| Path | Credential | Validation |
|---|---|---|
| ClickHouse Cloud API at `api.clickhouse.cloud` | API Key ID and Key Secret | The intended organizations and services are discovered. The connection authentication check tests this API path. |
| Each discovered service's HTTPS SQL endpoint | Database User and Database Password | The approved databases/tables can be enumerated and a bounded table read succeeds; enabled inspection then produces classifications. |

API authentication does not prove SQL sampling. Permit the EKS egress addresses in the service's IP access list or use your approved private connectivity. Use the HTTPS host and port from the service's **Connect** details; ClickHouse Cloud HTTPS commonly uses TLS port 8443. The scanner uses HTTP over TLS, so opening native TCP port 9440 alone is insufficient. See the [ClickHouse HTTPS connection details](https://clickhouse.com/docs/integrations/language-clients/python/index#gather-your-connection-details).

Use a dedicated database user with read permissions on the approved database/table scope and the metadata visibility needed to list that scope. Keep `db_user`, `db_password`, `key_id`, `key_secret` and `database_config` in the secret. Validate the service list and the SQL schema/table results separately before expanding scan scope.

### Prerequisites

Before connecting ClickHouse to Relyance AI, you must:

1. Have an active ClickHouse Cloud account
2. Have administrative access to create API keys
3. Create a dedicated API key for Relyance AI integration
4. (Optional) Set up database user credentials if enabling direct database inspection

### Authentication Setup

#### Step 1: Create API Credentials in ClickHouse Cloud

ClickHouse uses API key-based authentication to secure access to your organizations and services.

1. Log in to your [ClickHouse Cloud Console](https://console.clickhouse.cloud/)
2. Navigate to **Organization Settings** in the left-hand menu
3. Click on **API Keys** in the settings navigation
4. Click **Create New API Key**
5. Configure the API key:
    - **Name**: Enter a descriptive name (e.g., "Relyance AI Integration")
    - **Expiration date**: Set to "Never" or choose an appropriate expiration
    - **Roles**: Select "Admin" role (or appropriate role with read permissions)
    - **Allow access from**: Choose "Anywhere" for simplicity, or "Specific locations" for enhanced security
6. Click **Generate API Key**
7. **Important**: Copy both the **Key ID** and **Key Secret** immediately. The secret will not be shown again.

Make sure to securely store both values:

```
Key ID: (copy from the screen)
Key Secret: (copy from the screen - click the eye icon to reveal)
```

#### Step 2: (Optional) Create Database User for Data Inspection

If you want Relyance AI to perform data inspection on your ClickHouse databases:

1. Connect to your ClickHouse service using a SQL client
2. Create a dedicated read-only user for Relyance:

```sql
CREATE USER relyance_user IDENTIFIED WITH sha256_password BY 'your_secure_password';
```

1. Grant necessary read permissions:

```sql
-- Grant read access to specific databases
GRANT SELECT ON database_name.* TO relyance_user;

-- Or grant read access to all databases
GRANT SELECT ON *.* TO relyance_user;

-- Grant SHOW permissions to list databases and tables
GRANT SHOW DATABASES ON *.* TO relyance_user;
GRANT SHOW TABLES ON *.* TO relyance_user;
```

1. Verify the user can connect and query:

```sql
-- Test with:
SHOW DATABASES;
SHOW TABLES FROM database_name;
SELECT count(*) FROM database_name.table_name LIMIT 1;
```

Security Note:

This user only needs READ permissions. Do not grant INSERT, UPDATE, DELETE, or administrative privileges.

### Integration Configuration in Relyance AI

#### Step 3: Add ClickHouse Integration

1. Log in to your Relyance AI account
2. Navigate to **Settings**→ **Integrations**
3. In the **Vendor Integrations** tab, search for "ClickHouse"
4. Click the ClickHouse integration card to open its connections.

#### Step 4: Configure Connection Settings

In the **Basic Auth** configuration form, fill in the following fields:

**Product** (auto-filled)

```
from_clickhouse
```

**Key ID** (required)
Paste the Key ID you copied from ClickHouse Cloud Console.

**Key Secret** (required)
Paste the Key Secret you copied from ClickHouse Cloud Console.

**Database User** (optional)
Enter the database username if you created one for data inspection (e.g., `relyance_user` or `default`). Leave empty if you only want to scan organization and service metadata.

**Database Password** (optional)
Enter the database password for the user above. Leave empty if not using database inspection.

**Database Configuration** (optional)
Configure which databases to scan using JSON format:

```json
{
  "allow_list": ["production_db", "analytics_db"],
  "block_list": ["test_db", "development_db"]
}
```

Configuration options:

- **allow_list**: Only scan databases in this list (empty array means scan all)
- **block_list**: Exclude specific databases from scanning

Example configurations:

**Scan all databases:**

```json
{
  "allow_list": [
  ],
  "block_list": [
  ]
}
```

**Scan only specific databases:**

```json
{
  "allow_list": [
    "user_data",
    "transactions"
  ],
  "block_list": [
  ]
}
```

**Scan all except specific databases:**

```json
{
  "allow_list": [
  ],
  "block_list": [
    "temp_data",
    "cache"
  ]
}
```

#### Step 5: Configure Data Inspection (Optional)

The Data Inspection feature allows Relyance AI to analyze actual data content to identify personal information types and categories.

- **Enable Data Inspection**: Check this box to enable deep data scanning. Relyance will sample data from tables to identify PI types.
- **Minimum Confidence Level**: Set the minimum confidence threshold for PI detection. Possible values: `Very Low`, `Low`, `Medium`, `High`, `Very High`. Recommended: `High` for production environments.

Note:

Data inspection requires database credentials to be configured in the previous step.

#### Step 6: Configure Asset Discovery

**Enable Asset Discovery**: Check this box to enable automatic discovery of ClickHouse services. This creates a service map showing inter-service communication.

#### Step 7: Save and Connect

1. Click **Authenticate** to finalize the integration
2. Relyance will begin the initial scan of your ClickHouse environment
3. You'll see a confirmation on the Vendor Integrations page showing the connection is active

The integration will show a green "Connected" status with the connection date.

### What Data Does Relyance AI Collect?

Relyance AI will discover and analyze:

The **Authentication** step asks for:

- **Product**: Enter the product name for the APM service; required.
- **Key ID**: required; held as a secret.
- **Key Secret**: required; held as a secret.
- **Database User**: optional.
- **Database password**: optional; held as a secret.
- **Database Configuration**: required; a JSON value; the field is pre-filled with the shape to complete.

#### Organization Level

- Organization IDs and names
- Organization members and their roles
- Pending invitations
- API keys metadata (not secrets)

#### Service Level

- Service names and IDs
- Service configurations
- Service providers (AWS, GCP, Azure)
- Service regions and endpoints

#### Metrics Level

- Prometheus metrics from services
- Service-to-service communication patterns
- Performance metrics
- Health status indicators

#### Database Level (if data inspection enabled)

- Database names
- Table schemas
- Column names and data types
- Sample data for PI classification
- Data volume statistics

Important:

Relyance AI reads metadata and samples data for classification. For Outpost, completed metadata and classification results are copied to hosted GCS as described in [What Relyance receives](/docs/introduction-to-relyance-ai/how-outpost-works/#what-relyance-receives).

### Discovered Assets

After connecting ClickHouse, you'll see the following assets in Relyance AI:

#### In the Data Map

- **ClickHouse Organizations**: Top-level container assets
- **ClickHouse Services**: Individual database services
- **Service Hosts**: Cloud provider infrastructure
- **Service Names**: Logical service identifiers
- **Data Flows**: Service-to-service communication paths

#### In Asset Explorer

- Tables and columns from scanned databases
- Personal Information classifications
- Data category mappings
- Service dependencies

### Permissions Reference

The ClickHouse API key requires the following permissions:

| Resource | Permission | Purpose |
| --- | --- | --- |
| Organizations | Read | List all organizations |
| Services | Read | Discover ClickHouse services |
| Members | Read | Identify organization members |
| Invitations | Read | Track pending access |
| Metrics | Read | Access Prometheus endpoints |

For database inspection, the database user needs:

| Permission | Purpose |
| --- | --- |
| SELECT | Read table data |
| SHOW DATABASES | List available databases |
| SHOW TABLES | List tables in databases |

### Troubleshooting

#### Connection Fails

**Error: "Invalid API credentials"**

- Verify Key ID and Key Secret are correct
- Check that the API key hasn't been revoked
- Ensure the key has sufficient permissions

**Error: "Organization access denied"**

- Verify the API key has organization-level read permissions
- Check that your ClickHouse account is active

#### Database Inspection Fails

**Error: "Authentication failed for database user"**

- Verify database username and password are correct
- Test the credentials directly using a SQL client
- Check that the user hasn't been dropped or disabled

**Error: "Permission denied on database"**

- Verify the user has SELECT permissions
- Check database allow_list/block_list configuration
- Ensure SHOW permissions are granted

#### Metrics Collection Issues

**Error: "Cannot retrieve Prometheus metrics"**

- Verify services are running and healthy
- Check that Prometheus metrics are enabled on services
- Ensure API key has metrics read permissions

#### Slow Scan Performance

If the initial scan takes too long:

- Reduce the number of databases in allow_list
- Add large temporary databases to block_list
- Adjust sampling percentage in Asset Discovery settings
- Consider disabling data inspection for very large databases

### Security Best Practices

1. **Use Dedicated API Keys**: Create a separate API key specifically for Relyance AI
2. **Principle of Least Privilege**: Only grant read permissions required for scanning
3. **Rotate Credentials**: Periodically rotate API keys and database passwords
4. **Monitor Access**: Review ClickHouse audit logs for Relyance AI access patterns
5. **Restrict Database Access**: Use allow_list to limit which databases are scanned
6. **Secure Secrets**: Never commit API keys or passwords to source control

### Updating Configuration

To modify the integration settings:

1. Navigate to **Settings**→ **Integrations**
2. Find the ClickHouse integration
3. Click **Edit**
4. Update the desired fields
5. Click **Re-Authenticate** to apply updates

Changes take effect on the next scheduled scan or when you manually trigger a sync.

### Terraform with an Outpost credential ARN

Use provider `Relyance/relyance` 1.1.1 or a later release verified with Relyance, configured as in [Outpost connection setup](/docs/introduction-to-relyance-ai/how-outpost-works/#step-4-add-connections). This example references an existing AWS Secrets Manager secret and leaves the connection unconnected while you verify the deployment. Keep credential values out of `auth.params`, `auth.secrets_wo` and Terraform state; use the [BYOK secret schema](/docs/introduction-to-relyance-ai/outpost-connection-credentials/#aws-inventory-rds-and-clickhouse-secret-examples).

```hcl
variable "clickhouse_secret_arn" {
  type        = string
  description = "Full credential secret ARN in the Outpost deployment account"
}

resource "relyance_integration_connection" "clickhouse_outpost" {
  vendor       = "clickhouse"
  name         = "ClickHouse Cloud (Outpost)"
  runtime_mode = "IN_HOST_BYOK"
  connect_on_apply = false
  secret_ref   = var.clickhouse_secret_arn

  auth = {
    method = "username-password"
    params = {
      data_storage_location = "|United States|"
    }
  }

  scans = {
    "data-inspection" = { enabled = false }
  }
}
```

The ARN is a reference, not a credential value. It must belong to the registered deployment account. Save the raw-sample storage policy and review the separate sanitized-context setting under [What Relyance receives](/docs/introduction-to-relyance-ai/how-outpost-works/#what-relyance-receives). Enable/connect only after the [deployment and source checks](/docs/introduction-to-relyance-ai/how-outpost-works/#step-5-check-the-deployment) pass. The generated examples below use hosted credential handling.

<!-- terraform-examples:begin (generated from the integration catalog; do not hand-edit) -->

## Manage this integration with Terraform

Connections for this integration can be managed as code with the [Relyance Terraform provider](https://registry.terraform.io/providers/Relyance/relyance/latest). Non-secret fields go in `auth.params`; secret fields go in `auth.secrets_wo`, which is write-only — never stored in Terraform state. Rotate secrets by bumping `auth.secrets_wo_version`.

```hcl
resource "relyance_integration_connection" "clickhouse" {
  vendor = "clickhouse"
  name   = "<your connection name>"

  auth = {
    method = "username-password"
    params = {
      product = "from_clickhouse"
      database_config = jsonencode({
        allow_list = [
          ""
        ]
        block_list = [
          ""
        ]
      })
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      key_id = var.clickhouse_key_id
      key_secret = var.clickhouse_key_secret
      db_password = var.clickhouse_db_password
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}
```

<!-- terraform-examples:end -->