# ClickHouse

ClickHouse Cloud is a cloud-based distributed column-oriented database service designed for online analytical processing (OLAP) that allows generation of analytical data reports in real-time. The ClickHouse integration enables Relyance AI to discover and analyze data across your ClickHouse Cloud organizations, services, and metrics.

The ClickHouse integration provides:

- **Organization Discovery**: Scan all organizations, members, and invitations
- **Service Monitoring**: Discover services and their configurations
- **Metrics Analysis**: Extract Prometheus metrics from ClickHouse services
- **Data Inspection**: Analyze data schemas and identify personal information types
- **Service Mapping**: Visualize service-to-service communication flows

### Prerequisites

Before connecting ClickHouse to Relyance AI, you must:

1. Have an active ClickHouse Cloud account
2. Have administrative access to create API keys
3. Create a dedicated API key for Relyance AI integration
4. (Optional) Set up database user credentials if enabling direct database inspection

### Authentication Setup

#### Step 1: Create API Credentials in ClickHouse Cloud

ClickHouse uses API key-based authentication to secure access to your organizations and services.

1. Log in to your [ClickHouse Cloud Console](https://console.clickhouse.cloud/)
2. Navigate to **Organization Settings** in the left-hand menu
3. Click on **API Keys** in the settings navigation
4. Click **Create New API Key**
5. Configure the API key:
    - **Name**: Enter a descriptive name (e.g., "Relyance AI Integration")
    - **Expiration date**: Set to "Never" or choose an appropriate expiration
    - **Roles**: Select "Admin" role (or appropriate role with read permissions)
    - **Allow access from**: Choose "Anywhere" for simplicity, or "Specific locations" for enhanced security
6. Click **Generate API Key**
7. **Important**: Copy both the **Key ID** and **Key Secret** immediately. The secret will not be shown again.

Make sure to securely store both values:

```
Key ID: (copy from the screen)
Key Secret: (copy from the screen - click the eye icon to reveal)
```

#### Step 2: (Optional) Create Database User for Data Inspection

If you want Relyance AI to perform data inspection on your ClickHouse databases:

1. Connect to your ClickHouse service using a SQL client
2. Create a dedicated read-only user for Relyance:

```sql
CREATE USER relyance_user IDENTIFIED WITH sha256_password BY 'your_secure_password';
```

1. Grant necessary read permissions:

```sql
-- Grant read access to specific databases
GRANT SELECT ON database_name.* TO relyance_user;

-- Or grant read access to all databases
GRANT SELECT ON *.* TO relyance_user;

-- Grant SHOW permissions to list databases and tables
GRANT SHOW DATABASES ON *.* TO relyance_user;
GRANT SHOW TABLES ON *.* TO relyance_user;
```

1. Verify the user can connect and query:

```sql
-- Test with:
SHOW DATABASES;
SHOW TABLES FROM database_name;
SELECT count(*) FROM database_name.table_name LIMIT 1;
```

Security Note:

This user only needs READ permissions. Do not grant INSERT, UPDATE, DELETE, or administrative privileges.

### Integration Configuration in Relyance AI

#### Step 3: Add ClickHouse Integration

1. Log in to your Relyance AI account
2. Navigate to **Settings**→ **Integrations**
3. In the **Vendor Integrations** tab, search for "ClickHouse"
4. Click the ClickHouse integration card to open its connections.

#### Step 4: Configure Connection Settings

In the **Basic Auth** configuration form, fill in the following fields:

**Product** (auto-filled)

```
from_clickhouse
```

**Key ID** (required)
Paste the Key ID you copied from ClickHouse Cloud Console.

**Key Secret** (required)
Paste the Key Secret you copied from ClickHouse Cloud Console.

**Database User** (optional)
Enter the database username if you created one for data inspection (e.g., `relyance_user` or `default`). Leave empty if you only want to scan organization and service metadata.

**Database Password** (optional)
Enter the database password for the user above. Leave empty if not using database inspection.

**Database Configuration** (optional)
Configure which databases to scan using JSON format:

```json
{
  "allow_list": ["production_db", "analytics_db"],
  "block_list": ["test_db", "development_db"]
}
```

Configuration options:

- **allow_list**: Only scan databases in this list (empty array means scan all)
- **block_list**: Exclude specific databases from scanning

Example configurations:

**Scan all databases:**

```json
{
  "allow_list": [
  ],
  "block_list": [
  ]
}
```

**Scan only specific databases:**

```json
{
  "allow_list": [
    "user_data",
    "transactions"
  ],
  "block_list": [
  ]
}
```

**Scan all except specific databases:**

```json
{
  "allow_list": [
  ],
  "block_list": [
    "temp_data",
    "cache"
  ]
}
```

#### Step 5: Configure Data Inspection (Optional)

The Data Inspection feature allows Relyance AI to analyze actual data content to identify personal information types and categories.

- **Enable Data Inspection**: Check this box to enable deep data scanning. Relyance will sample data from tables to identify PI types.
- **Minimum Confidence Level**: Set the minimum confidence threshold for PI detection. Possible values: `Very Low`, `Low`, `Medium`, `High`, `Very High`. Recommended: `High` for production environments.

Note:

Data inspection requires database credentials to be configured in the previous step.

#### Step 6: Configure Asset Discovery

**Enable Asset Discovery**: Check this box to enable automatic discovery of ClickHouse services. This creates a service map showing inter-service communication.

#### Step 7: Save and Connect

1. Click **Authenticate** to finalize the integration
2. Relyance will begin the initial scan of your ClickHouse environment
3. You'll see a confirmation on the Vendor Integrations page showing the connection is active

The integration will show a green "Connected" status with the connection date.

### What Data Does Relyance AI Collect?

Relyance AI will discover and analyze:

The **Authentication** step asks for:

- **Product**: Enter the product name for the APM service; required.
- **Key ID**: required; held as a secret.
- **Key Secret**: required; held as a secret.
- **Database User**: optional.
- **Database password**: optional; held as a secret.
- **Database Configuration**: required; a JSON value; the field is pre-filled with the shape to complete.

#### Organization Level

- Organization IDs and names
- Organization members and their roles
- Pending invitations
- API keys metadata (not secrets)

#### Service Level

- Service names and IDs
- Service configurations
- Service providers (AWS, GCP, Azure)
- Service regions and endpoints

#### Metrics Level

- Prometheus metrics from services
- Service-to-service communication patterns
- Performance metrics
- Health status indicators

#### Database Level (if data inspection enabled)

- Database names
- Table schemas
- Column names and data types
- Sample data for PI classification
- Data volume statistics

Important:

Relyance AI only reads metadata and samples data. It does not store or transmit your actual business data.

### Discovered Assets

After connecting ClickHouse, you'll see the following assets in Relyance AI:

#### In the Data Map

- **ClickHouse Organizations**: Top-level container assets
- **ClickHouse Services**: Individual database services
- **Service Hosts**: Cloud provider infrastructure
- **Service Names**: Logical service identifiers
- **Data Flows**: Service-to-service communication paths

#### In Asset Explorer

- Tables and columns from scanned databases
- Personal Information classifications
- Data category mappings
- Service dependencies

### Permissions Reference

The ClickHouse API key requires the following permissions:

| Resource | Permission | Purpose |
| --- | --- | --- |
| Organizations | Read | List all organizations |
| Services | Read | Discover ClickHouse services |
| Members | Read | Identify organization members |
| Invitations | Read | Track pending access |
| Metrics | Read | Access Prometheus endpoints |

For database inspection, the database user needs:

| Permission | Purpose |
| --- | --- |
| SELECT | Read table data |
| SHOW DATABASES | List available databases |
| SHOW TABLES | List tables in databases |

### Troubleshooting

#### Connection Fails

**Error: "Invalid API credentials"**

- Verify Key ID and Key Secret are correct
- Check that the API key hasn't been revoked
- Ensure the key has sufficient permissions

**Error: "Organization access denied"**

- Verify the API key has organization-level read permissions
- Check that your ClickHouse account is active

#### Database Inspection Fails

**Error: "Authentication failed for database user"**

- Verify database username and password are correct
- Test the credentials directly using a SQL client
- Check that the user hasn't been dropped or disabled

**Error: "Permission denied on database"**

- Verify the user has SELECT permissions
- Check database allow_list/block_list configuration
- Ensure SHOW permissions are granted

#### Metrics Collection Issues

**Error: "Cannot retrieve Prometheus metrics"**

- Verify services are running and healthy
- Check that Prometheus metrics are enabled on services
- Ensure API key has metrics read permissions

#### Slow Scan Performance

If the initial scan takes too long:

- Reduce the number of databases in allow_list
- Add large temporary databases to block_list
- Adjust sampling percentage in Asset Discovery settings
- Consider disabling data inspection for very large databases

### Security Best Practices

1. **Use Dedicated API Keys**: Create a separate API key specifically for Relyance AI
2. **Principle of Least Privilege**: Only grant read permissions required for scanning
3. **Rotate Credentials**: Periodically rotate API keys and database passwords
4. **Monitor Access**: Review ClickHouse audit logs for Relyance AI access patterns
5. **Restrict Database Access**: Use allow_list to limit which databases are scanned
6. **Secure Secrets**: Never commit API keys or passwords to source control

### Updating Configuration

To modify the integration settings:

1. Navigate to **Settings**→ **Integrations**
2. Find the ClickHouse integration
3. Click **Edit**
4. Update the desired fields
5. Click **Re-Authenticate** to apply updates

Changes take effect on the next scheduled scan or when you manually trigger a sync.

<!-- terraform-examples:begin (generated from the integration catalog; do not hand-edit) -->

## Manage this integration with Terraform

Connections for this integration can be managed as code with the [Relyance Terraform provider](https://registry.terraform.io/providers/Relyance/relyance/latest). Non-secret fields go in `auth.params`; secret fields go in `auth.secrets_wo`, which is write-only — never stored in Terraform state. Rotate secrets by bumping `auth.secrets_wo_version`.

```hcl
resource "relyance_integration_connection" "clickhouse" {
  vendor = "clickhouse"
  name   = "<your connection name>"

  auth = {
    method = "username-password"
    params = {
      product = "from_clickhouse"
      database_config = jsonencode({
        allow_list = [
          ""
        ]
        block_list = [
          ""
        ]
      })
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      key_id = var.clickhouse_key_id
      key_secret = var.clickhouse_key_secret
      db_password = var.clickhouse_db_password
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}
```

<!-- terraform-examples:end -->