Source Code Integration: Overview and Technical Details

Source Code Integration: Overview and Technical Details

Overview

The Source Code Inspector (SCI) a.k.a Source Code Integration is Relyance AI’s proprietary software that performs Natural Language Processing (NLP) and Static Code Analysis, a core feature of Relyance AI, to continuously detect and monitor data flows across internal microservices, machine learning (ML) pipelines, and third-parties for your privacy and data protection program.

The following topics are covered in this article:

• How to set up SCI?
• What data does SCI gather and send back to Relyance AI?
• What security controls does Relyance AI have in place?
• Common myths answered
• Integration Technical Details

Where does the SCI live?

The Source Code Inspector typically lives in a CI/CD pipeline with results being sent back to Relyance. It is suggested to create a separate CI pipeline for the Relyance scan as it is a passive scan and would impact build time. The SCI only needs access to the source code (or Code Repository).

Here is a typical high-level workflow:

How to set up SCI?

SCI is run on-prem and is designed to be deployed in customer environments because Relyance AI does not pull customers’ source code into the Relyance AI environment.

SCI has two primary setup steps:

1. 1 configuration file: This file describes the name of the code repository, the source code directories to scan, and source code directories to ignore. See the technical details section for additional information about the YAML file. If the Relyance configuration file is absent in the source code repository, the Source Code Inspector will search for a '.git/config' file. In such a case, it will automatically create a YAML file, using the repository url as the repository name. If neither the relyance.yaml nor .git/config file is present, the source code inspector will produce an error message.
2. 1 line of code: A single line of code (docker pull command), that can be added to your existing CI (continuous integration) pipeline enables the code scan every time new code is deployed.
   • The docker pull command for the tenant is available under "Settings > Integrations > Source Code Integration > Step 2" (screenshot above).
   • We recommend setting up a nightly isolated job in your CI pipeline for Relyance. If that isn't feasible, a nightly CRON job will also work. Windows users should replace pwd:/repo with ${PWD}:/repo in the docker run command

This is a non-intrusive text-based scan, meaning Relyance AI does not inject anything into your source code and you are not required to make any code changes on your side. This line of code ensures that the latest SCI is pulled from Relyance AI’s servers as a secure docker container. It is then authorized, authenticated, securely executed after authentication and then the docker container is deleted automatically until the build process runs again.

This allows Relyance AI to offer a continuous compliance management and monitoring platform for privacy and data protection at the point when new data flows arise.

What data does SCI gather and send back to Relyance AI?

The SCI scan creates an analysis report which is sent back to Relyance AI and this only includes metadata around:

• Function and API calls
• Third-party SDK or endpoint usages
• Function formal parameters
• Call graph skeletons

Relyance AI analyzes this metadata to build a graph of any Personal Information and sensitive data in use. This graph is then used to identify data communications with third-party vendors, and internal services.

For example, SCI can determine that personal information like customer email addresses is being sent to different vendors via API calls from the source code but it never collects actual customer email addresses. Also, note that the source code is not transmitted to Relyance AI.

What security controls does Relyance AI have in place?

At Relyance AI, we provide enterprise-grade controls to ensure security at every level.

Software Protection Controls: How do we protect the SCI?

Data Security & Access Controls: How do we protect data access via the SCI?

Physical Controls: How do we protect SCI during implementation?

In addition to the controls outlined above, customers can also allow-list outbound connections only to Relyance AI to ensure the binary never communicates with any other entity. Overall, Relyance AI has implemented a robust privacy and security program, which places maximal emphasis on the protection of customer data. Additional details about our overall security program are detailed in Relyance AI's Privacy and Security Paper.

Common myths answered

Common myths about Relyance AI’s SCI are addressed below:

1. SCI does not pull source code and transmit it to Relyance AI.
2. SCI doesn’t require customers to make changes in their source code unlike other Application Performance Monitoring (APM) tools in the market.
3. SCI is not an SDK.
4. SCI doesn’t show in your Production code, system, or logs in any way.
5. Issues in the SCI will not prevent or impact any customer systems or processes. SCI can be run as a separate nightly job so it never interferes with the build process. If for any reason the SCI fails, it doesn't affect our customers’ product build, unless it’s configured as such.
6. SCI is agnostic of build tools or CI/CD infrastructure.

Integration Technical Details:

i. Prerequisites

• Docker needs to be installed on the machine that runs the Relyance AI job.
• Docker needs to be configured to use Linux containers.
• Host needs to be x86_AMD64 architecture (or be able to emulate it). The latest version of Docker Desktop can emulate different architectures.
• Relyance needs to be able to communicate with GCP and Relyance. Please add the following sites to your firewall's allow-list:
  • *.gcr.io
  • *.relyance.ai
  • *.googleapis.com

ii. Which repos should be integrated with Relyance AI?

We recommend using the following approach to identifying source code repos:

• Core backend/microservices that talk to tools and third parties (via 3rd party SDK or API calls).
• Component/services that talk to databases, data-lakes, data-warehouses, and the core backend.
• Repos that have ETL pipelines / Machine Learning DAGs to push or pull data from business applications.

The Relyance Source Code Inspector does not scan third-party directories by default. These third-party directories include:

• node_modules
• vendor
• lib/python2
• lib/python3

This is to prevent over-reporting and over-scanning. If you wish to include this folder, you'll need to explicitly specify it in the Relyance configuration file.

iii. How to update the configuration file (relyance.yaml) and what is its purpose?

The configuration file is a .yaml file. Therelyance.yaml file is used to provide build dependencies and additional information about your code repository.

The configuration file (relyance.yaml) needs to be placed in the root directory of your Github repository. You can edit the default relyance.yaml file to change it. Below is the list of configuration areas.

Configuration Area #1: Product and repository name

This configuration is used to specify the name of the product built by the repository and the name of the repository.

Example:

repository: Acme/BoutiqueApp

Configuration Area #2: Build dependency table (services, source, and product)

This configuration is used to ascertain the services impacted by the APIs. In the example configuration below, the service 'adservice' is built using the code in src/adservice and src/dbutils. Please note that the directory paths are relative to the repository's root directory.

In addition, the product field associates a set of services with a product name which can then be used as a filter in the platform. For example, in the example below, adservice and cartservice are associated with the product Boutique while checkoutservice is associated with product Boutique2.

Example:

services:

- service: adservice
  product: 
     - Boutique
  source:
     - src/adservice
     - src/dbutils

- service: cartservice
  product: 
     - Boutique
  source:
     - src/cartservice

- service: checkoutservice
  product: 
     - Boutique2
  source:
     - src/checkoutservice
     - src/dbutils

*Configuration Area #3:*Source directories to exclude

This configuration is used to exclude the directories you specify from Relyance AI’s SCI processing. All directory names are relative to the repository's root.

Example:

exclude:
   - src/experiments
   - src/adservice/playground

Configuration Area #4: Restricted to theseSource directories

This setting restricts the scan to only the directories you specify. Think of it like a spotlight: it will only illuminate and process what's within those specific areas, ignoring everything else. All directory names must be relative to the repository's starting point.

Example:

restrict:
   - src/libraries
   - src/custom_code

Relyance AI will work with your technical team to update this file.

The name of the yaml file should be “relyance. yaml”. If it does not exist, the source code inspector will search for a "relyance. yml" and failing that will check if this is a git repo to grab the repo name. If the configuration file does not exist and it is not a git repo, you will get a missing configuration file error.

Appendix:

Running the Source Code Inspector in test mode

It's possible to run the source code scan without sending data back to Relyance. Use this feature when you want an example of what the source code inspector relays back to Relyance.

Append --dry-run to your docker run command to see example output of a randomly picked file:

docker run --rm -v `pwd`:/repo --env "API_KEY=example:1234567890"
gcr.io/relyance-ext/compliance_inspector:release --dry-run

Currently supported for all languages exceptJavascript/Typescript.

The output will look something like:

**************************** Relyance AI™ Inspector ****************************
* Privacy. It's in the Code™
********************************************************************************
Scanning source code...
100%|████████████████████████████████████████| 136/136 [00:01<00:00, 88.68it/s]
Elapsed: 4s
Module 'Pagerduty', version: 1, path:
  'src/Pagerduty/python/EVENTS_API_v2/trigger/trigger_with_incident_key.py',
  language: 'python', datetime: '2024-03-25 23:00:35', next_id: 87
  Import '<built_in>'
    Reference '<built_in>', kind: 'module'
  Module 'python'
.....