dbt Cloud Integration

Introduction

This document provides information on how to use dbt integration feature to set up automatic metadata sync between dbt Cloud and Holistics.

Pre-requisites

• You need to have access to the dbt Cloud API.

How it works

• dbt Service Account Token: This token enables secure authentication with the dbt Cloud API given that you grant each token a narrow, controlled set of permissions. You can learn more about it here.
• dbt Cloud job: a set of dbt commands to execute against a data source. This is created in dbt Cloud.
• dbt integration job: an operation in Holistics to ensure that the metadata between dbt models and Holistics models are in sync
  • By linking a dbt Cloud job with a data source in Holistics, we can get the metadata after dbt commands are run on that data source

Step-by-step instructions

Step 1: Generate dbt Cloud Service Token with read-only permissions

Remember to copy the Service Token that you have generated!

Step 2: Configure dbt integration in Holistics

• Go to Holistics Settings > Integrations > dbt Integration.

• Add dbt Cloud connection.
  • Add the Service Account Token that you have created in Step 1.
  • Add Account ID (can be retrieved in the URL https://cloud.getdbt.com/next/settings/accounts/{{account_id}}.
  • Add Custom URL (or Access URL in dbt): can be found via this guide.
• Add dbt Integration job.
  • Add the dbt Job ID that you want to sync dbt metadata with Holistics
    • There could be multiple Job IDs that run on different environments and target different data sources/schemas. So you need to choose which Job ID you want to sync with Holistics
    • You can get the dbt Cloud job by going to that Job in dbt Cloud and checking via the URL https://cloud.getdbt.com/next/deploy/{{account_id}}/projects/{{project_id}}/jobs/{{job_id}}
• Link that dbt Integration Job with a Data Source that you have already connected in Holistics.
• Connect and Execute.
• After the dbt Integration Job executes successfully, all the metadata (descriptions, dependencies graph) in dbt will be synced automatically with related Holistics Table Models via the Data Source name (done in step 5.2) and Source Table Name. This is the same with dbt CLI integration.

Removing dbt Cloud Connection

info

Please note that removing the dbt Cloud connection only discontinues the automatic sync between dbt Cloud and Holistics models. Therefore, any previously synced metadata will continue to exist in Holistics. We do not support an option to entirely erase the synced metadata from dbt at the moment.

To remove dbt Cloud connection, go to Settings > Integrations > Update dbt Cloud connection > Remove dbt Cloud connection

Using your metadata

If you prefer not to use the metadata from dbt, you can easily supersede it by adding your descriptions to the model. By inputting descriptions for models and/or dimensions, the system will automatically overwrite the existing dbt description.