Elasticsearch

Elasticsearch is a distributed, cloud-friendly search and analytics engine that combines the power of full-text search, real-time data indexing, and scalable querying into a single flexible platform for logs, metrics, and structured data..

In order to integrate Elasticsearch to Relyance AI, you will need an API key.

In Elasticsearch portal:

1- Login to your Elasticsearch account and go to Management on the left side bar. Then select Stack Management from the menu.

2 - Still on the new left side bar, go for Security and select API keys. Then select Create API key from from the menu.

3 - In this moment, it’s important to set the same indices names that will appear in the Relyance settings. Below are some example of the default. The key needs to have access to the same indexes used by the integration, defined in the integration configuration page. Then click in Create API Key

4 - Copy the API key from the new menu.

In the Relyance application:.

1. Login to your Relyance AI account.
2. Navigate to the Settings Menu in the bottom left-hand side.
3. Select Integrations.
4. Search for Elasticsearch and click it to open its connections.
5. In Authentication, select APM and paste the the API key in the field api_key inside the text area. Then click on Ok.

5. In Completion, click on Finish. Congratulations! You are now connected to Elasticsearch.

   The Authentication step asks for:

   • Using Custom —
     • API Key: API key for the Elastic Cloud API at api.elastic-cloud.com; required; held as a secret.
   • Using API Key —
     • API Key: API key for your custom Elasticsearch endpoint; required; held as a secret.
     • Elasticsearch endpoint: Elasticsearch endpoint; optional.
     • Indices: required; a JSON value; the field is pre-filled with the shape to complete.
   • Using APM —
     • Account Details: Add details about the Elasticsearch accounts; required; a JSON value; the field is pre-filled with the shape to complete.

If the connection reports Connected but returns nothing

These are the ways this integration comes back empty without reporting an error. Generated from the integration catalog, so it tracks what the connection actually asks for.

1. A feature you enabled returns nothing. Some scopes belong to a feature rather than to the connection: Data subject requests needs API Key. Turning the feature on after the connection exists does not widen the grant it already holds, so re-authenticate the connection.
2. A credential rotated at the vendor is not picked up here. API Key is stored when you save the connection, so regenerating the value at the vendor breaks the next scan until it is re-pasted here. Recording the expiry on the connection means Relyance warns you before it lapses.
3. Check the address fields before suspecting the credentials. Elasticsearch endpoint identifies which tenant, region or host to talk to. A wrong value there fails authentication and looks exactly like a bad secret.
4. One entry can be empty while the others work. Account Details is a list, and each entry is authorised independently, so a missing grant on one account or subscription empties just that entry -- silently.

Manage this integration with Terraform

Connections for this integration can be managed as code with the Relyance Terraform provider. Non-secret fields go in auth.params; secret fields go in auth.secrets_wo, which is write-only — never stored in Terraform state. Rotate secrets by bumping auth.secrets_wo_version.

API key

resource "relyance_integration_connection" "elasticsearch_0" {
  vendor = "elasticsearch"
  name   = "<your connection name>"

  auth = {
    method = "api-key"
    params = {
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      api_key = var.elasticsearch_api_key
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}

API key — API Key

resource "relyance_integration_connection" "elasticsearch_1" {
  vendor = "elasticsearch"
  name   = "<your connection name>"

  auth = {
    method = "api-key-search"
    params = {
      indices = jsonencode({
        indices = [
          ""
        ]
      })
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      api_key = var.elasticsearch_api_key
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}

Account credentials

resource "relyance_integration_connection" "elasticsearch_2" {
  vendor = "elasticsearch"
  name   = "<your connection name>"

  auth = {
    method = "account-credentials"
    # NOTE: accounts carries credentials (api_key) but is not flagged
    # secret, so it goes in params and IS written to Terraform state in
    # cleartext. Keep state in an encrypted remote backend with restricted
    # access, and rotate the credential if state was ever committed.
    params = {
      accounts = jsonencode([
        {
          api_key = ""
          index = "traces-apm*,apm-*,logs-apm*,apm-*,metrics-apm*,apm-*"
          product = ""
          url = ""
        }
      ])
      data_storage_location = "us"
    }
  }

  scans = { "data-inspection" = { enabled = true } }
}