Custom Data Import

Custom Data Import

Overview

The Relyance AI Custom Data Import enables the ability to scan schema files from any system and map the information to a vendor (and, correspondingly, its ROPA) in your data map in Relyance. To review a schema file, Relyance needs four (4) critical pieces of information:

1. The location where the schema files are located.
2. An "Endpoint" name: This corresponds to the "entity" in the source system the data should be attributed. For example, the endpoint will be a table name for a database or a data warehouse. For a sales for marketing SaaS system, the endpoint will be the "entity" or the "object" name that stores data types.
3. "Endpoint Field" name: This corresponds to the field name that Relyance should scan to extract the data type. For example, the property will be the table's column names for a database or a data warehouse. For a sales for marketing SaaS system, the properties will be the "keys" of the object specified in the endpoint.
4. Vendor: The vendor name in the Relyance that data from the CSV/TSV files should be attributed to.

What does Relyance support?

• CSV and TSV files. Compressed files for these formats are supported ( .tsv.gzip ).
• The two (2) primary ways you can import data in Relyance; are via an Amazon S3 bucket or a shared folder on Google Drive.

End-to-End Process Example:

1. Create your CSV/TSV file by exporting it from the source system. In the example below, the column "TABLE_NAME" represents the Endpoint, while the column "COLUMN_NAME" represents the Endpoint Field:
2. Upload the CSV/TSV file in Google Drive or an S3 bucket. The example below uploads the file to a folder called "Test" in "My Drive" in Google Drive.
3. Navigate to the Custom Data Import Integration Page in Relyance and select the desired connection setting. In the example below, since our file resides in My Drive in Google Drive, that's the connection option chosen. Please look at the other configuration values in the "Edit Connection" page below.
4. Once the integration runs successfully, you can view the data tied to the vendor selected in step 3 above, along with the data types and endpoint values.

Individual Configuration Steps:

In your source system:

1. Export the desired schema in a ".csv" or a ".tsv" format with at least two (2) columns; the endpoint and the endpoint field.
2. Place the schema file(s) either in an S3 bucket or a Google Drive folder.

In the Relyance AI application: AWS S3 - Role

You can use this option if you intend to attach the S3 buckets to a role and then give Relyance access to that Role.

1. Login to your Relyance account.
2. Navigate to the Settings menu on the bottom left-hand side.
3. Select Integrations.
4. Click on the Vendor Integration tab.
5. Find the Custom Data Import integration card and click it to open its connections.
6. Under Authentication Method, choose AWS S3 - Role.
7. Select a Vendor from the "Vendor" dropdown. Relyance will map the data type predictions to the selected vendor entity. Only vendors available in your data map will be displayed in this dropdown.
8. Input your Account ID, Role Name, External ID, Region, and Bucket Name for Relyance to connect. Check out the Appendix below for more information on creating buckets in S3.
9. Input the "Endpoint Column Name" and the "Endpoint Field Column Name." as they appear in the CSV / TSV file.
10. Click Authenticate.

In the Relyance AI application: AWS S3 - User

Use this option if you intend to grant Relyance's user access to an S3 bucket(s).

Work with your Customer Success team to get credentials for the Relyance AWS User.

1. Navigate to the Settings menu on the bottom left-hand side.
2. Select Integrations.
3. Click on the Vendor Integration tab.
4. Find the Custom Data Import integration card and click it to open its connections.
5. Under Authentication Method, choose AWS S3 - User.
6. Select a Vendor from the "Vendor" dropdown. Relyance will map the data type predictions to the selected vendor entity. Only vendors available in your data map will be displayed in this dropdown.
7. Input an appropriate value for "Resources." This will be a link to the S3 bucket.
8. Input the "Endpoint Column Name" and the "Endpoint Field Column Name." as they appear in the CSV / TSV file.
9. Click Authenticate.

In the Relyance AI application: Google Drive - My Drive

You can use this option if you grant Relyance access to the CSV/TSV files via a folder in "My Drive."

1. Navigate to the Settings menu on the bottom left-hand side.
2. Select Integrations.
3. Click on the Vendor Integration tab.
4. Find the Custom Data Import integration card and click it to open its connections.
5. Under Authentication Method, choose Google - My Drive.
6. Select a Vendor from the "Vendor" dropdown. Relyance will map the data type predictions to the selected vendor entity. Only vendors available in your data map will be displayed in this dropdown.
7. Enter the folder name in "My Drive folders containing Contracts delimited by; (Eg: folder1; folder2)". For more information on folder naming conventions etc., check out this article (search for "For My Drive or Shared Folder")
8. Input the "Endpoint Column Name" and the "Endpoint Field Column Name." as they appear in the CSV / TSV file.
9. Click Authenticate.

In the Relyance AI application: Google Drive - Shared Folder

You can use this option if you grant Relyance access to the CSV/TSV files via a shared folder in Google Drive.

1. Navigate to the Settings menu on the bottom left-hand side.
2. Select Integrations.
3. Click on the Vendor Integration tab.
4. Find the Custom Data Import integration card and click it to open its connections.
5. Under Authentication Method, choose Google Drive - Shared Folder.
6. Select a Vendor from the "Vendor" dropdown. Relyance will map the data type predictions to the selected vendor entity. Only vendors available in your data map will be displayed in this dropdown.
7. Enter the folder name in "Shared folders containing Contracts delimited by; (Eg: folder1; folder2)". For more information on folder naming conventions etc., check out this article (search for "For My Drive or Shared Folder")
8. Input the "Endpoint Column Name" and the "Endpoint Field Column Name." as they appear in the CSV / TSV file.
9. Click Authenticate.

In the Relyance AI application: Google Drive - Shared Drive

You can use this option if you grant Relyance access to the CSV/TSV files via a folder in a Shared Drive in Google Drive.

1. Navigate to the Settings menu on the bottom left-hand side.
2. Select Integrations.
3. Click on the Vendor Integration tab.
4. Find the Custom Data Import integration card and click it to open its connections.
5. Under Authentication Method, choose Google Drive - Shared Drive.
6. Select a Vendor from the "Vendor" dropdown. Relyance will map the data type predictions to the selected vendor entity. Only vendors available in your data map will be displayed in this dropdown.
7. Input the configuration by using the JSON editor next to "Shared Drive Folders". For more information on folder naming conventions etc., check out this article (search for "For Shared Drive")
8. Input the "Endpoint Column Name" and the "Endpoint Field Column Name." as they appear in the CSV / TSV file.
9. Click Authenticate.

Appendix: Giving Relyance permissions in AWS S3

1. Create a standard S3 Bucket. Specify a bucket name and an AWS Region while keeping all other settings to default.

2. Navigate to IAM in AWS.
3. Select Roles on the left side menu and click Create Role.

4. From the options, choose AWS Account.

5. Enter the account ID (Relyance's account id), choose a value for the External ID, and click next.

6. On the Add permissions page, search for " AmazonS3ReadOnlyAccess " and select that policy and click Next. As a side note, feel free to create your own custom policy based on your requirements.

7. Give the Role a name and create.
8. Go to the Roles page and the Role you just created. Click on the "Trust Relationships" tab.
9. Click on "Edit Trust Policy"
10. Change the line with "Principal": {"AWS": "arn:aws:iam::580082088342:root"} To “Principal”: {"AWS": "arn:aws:iam::580082088342:user/tenant-prod-access" }
11. Drop the schema files in the AWS S3 bucket created above.

If the connection reports Connected but returns nothing

These are the ways this integration comes back empty without reporting an error. Generated from the integration catalog, so it tracks what the connection actually asks for.

1. Authorised, then empty. The connection carries an authorisation, not a password: it stops returning data if the account that granted it loses access, its own permissions narrow, or the grant is revoked at the vendor. None of that reports an error here -- the connection keeps its last status until the next scan.
2. A credential rotated at the vendor is not picked up here. External ID is stored when you save the connection, so regenerating the value at the vendor breaks the next scan until it is re-pasted here. Recording the expiry on the connection means Relyance warns you before it lapses.
3. Check the address fields before suspecting the credentials. Endpoint Column Names, Account ID and Region identify which tenant, region or host to talk to. A wrong value there fails authentication and looks exactly like a bad secret.

Authentication methods and fields

Pick one of these under Authentication Method on the connection wizard's Authentication step. This table is generated from the integration catalog, so it always matches what the form actually asks for.

Manage this integration with Terraform

Connections for this integration can be managed as code with the Relyance Terraform provider. Non-secret fields go in auth.params; secret fields go in auth.secrets_wo, which is write-only — never stored in Terraform state. Rotate secrets by bumping auth.secrets_wo_version.

OAuth (browser authorization) — Google - My Drive

The OAuth (browser authorization) — Google - My Drive method uses a browser authorization flow, so the connection is created in the Relyance app. Manage it in Terraform afterwards by importing it (terraform import relyance_integration_connection.example custom_data_import/<connection_id>) or reading it with the relyance_integration_connection data source.

OAuth (browser authorization) — Google - Shared Folder

The OAuth (browser authorization) — Google - Shared Folder method uses a browser authorization flow, so the connection is created in the Relyance app. Manage it in Terraform afterwards by importing it (terraform import relyance_integration_connection.example custom_data_import/<connection_id>) or reading it with the relyance_integration_connection data source.

OAuth (browser authorization) — Google - Shared Drive

The OAuth (browser authorization) — Google - Shared Drive method uses a browser authorization flow, so the connection is created in the Relyance app. Manage it in Terraform afterwards by importing it (terraform import relyance_integration_connection.example custom_data_import/<connection_id>) or reading it with the relyance_integration_connection data source.

OAuth (browser authorization) — Data Inspection - My Drive

The OAuth (browser authorization) — Data Inspection - My Drive method uses a browser authorization flow, so the connection is created in the Relyance app. Manage it in Terraform afterwards by importing it (terraform import relyance_integration_connection.example custom_data_import/<connection_id>) or reading it with the relyance_integration_connection data source.

OAuth (browser authorization) — Data Inspection - Shared Drive

The OAuth (browser authorization) — Data Inspection - Shared Drive method uses a browser authorization flow, so the connection is created in the Relyance app. Manage it in Terraform afterwards by importing it (terraform import relyance_integration_connection.example custom_data_import/<connection_id>) or reading it with the relyance_integration_connection data source.

OAuth (browser authorization) — Data Inspection - Shared Folders

The OAuth (browser authorization) — Data Inspection - Shared Folders method uses a browser authorization flow, so the connection is created in the Relyance app. Manage it in Terraform afterwards by importing it (terraform import relyance_integration_connection.example custom_data_import/<connection_id>) or reading it with the relyance_integration_connection data source.

Amazon S3 - User

resource "relyance_integration_connection" "custom_data_import_6" {
  vendor = "custom_data_import"
  name   = "<your connection name>"

  auth = {
    method = "s3-user"
    params = {
      vendor = "<vendor>"
      resources = "<resources>"
      endpoint_columns = "<endpoint_columns>"
      property_column = "<property_column>"
      data_storage_location = "us"
    }
  }

  scans = { "property-inspection" = { enabled = true } }
}

AWS IAM role

resource "relyance_integration_connection" "custom_data_import_7" {
  vendor = "custom_data_import"
  name   = "<your connection name>"

  auth = {
    method = "iam-role"
    params = {
      vendor = "<vendor>"
      account_id = "<account_id>"
      role_name = "<role_name>"
      region = "<region>"
      bucket_name = "<bucket_name>"
      endpoint_columns = "<endpoint_columns>"
      property_column = "<property_column>"
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      external_id = var.custom_data_import_external_id
    }
    secrets_wo_version = 1
  }

  scans = { "property-inspection" = { enabled = true } }
}