dbt Core Integration

High-level Concept

How this works on the high-level:

• dbt is run and generates a manifest.json file that contains metadata related to the dbt project.
• Users can push dbt's manifest.json file to Holistics using Holistics CLI
• Holistics links metadata to the corresponding Holistics data models with the same underlying data tables generated by dbt.

Some of the metadata from dbt that can be presented in Holistics includes Tables description, columns description, dbt's lineage diagrams.

Setting up

Step 1: Install & Configure Holistics CLI

Currently, Holistics only support pushing dbt metadata to Holistics via CLI.

Please follow this doc to set up the CLI for Holistics. Make sure you generate an API Access Key and authenticate the CLI with that key.

Step 2: Pushing manifest.json file to Holistics

manifest.json is a JSON file generated by dbt build process. The file contains all metadata related to your dbt project.

Depending on your situation, use an appropriate DBT command to generate your manifest.json file:

What dbt command should I use to generate manifest.json?

• If this is the first time running the project, and your dbt models have not been materialized, use dbt run or dbt build.

• If you have just made a change to your dbt project, use dbt run or dbt build.

• If you have run your dbt project, materialized dbt models and there is nothing you want to change, use dbt compile.

Once you have the latest manifest file generated, run:

$ holistics dbt upload --file-path manifest.json --data-source=your_ds_name

The above command requires 2 params:

• --file-path: path to your manifest.json file
• --data-source: your data source name in Holistics

Each manifest file will be linked to a specific data source in Holistics.

If you have successfully pushed the file manifest.json to Holistics, you can go to our admin panel to check the status of your manifest.json file.

Step 3: Create the table models in Holistics UI

At this stage, all of your metadata from dbt has been propagated into Holistics. Table Models that share the same data tables with dbt will automatically display the metadata (defined in dbt docs)

If you don't have any models from dbt tables yet, you can create the table model as normal. Holistics will auto-detect if that table is linked with a dbt model.

Consider the following users_summary Holistics model that sits on top of users_summary table (created by dbt)

Model users_summary {
  data_source: ''
  table: 'bi.users_summary'

  dimension id {
    type: 'number'
  }
  dimension username {
    type: 'text'
  }
  dimension full_name {
    type: 'text'
  }
}

When viewed in Holistics UI, the descriptions of the fields are automatically pulled in from dbt metadata

Overriding field's description

If you want to set a custom description and override the ones defined in dbt, just set the value of description attribute of the field.

Model users_summary {
  table: 'bi.users_summary'
  ...

  dimension full_name {
    ...
    description: 'Full name of the user' # This will override the value from dbt
  }
}

FAQs

If I update my model's metadata in Holistics, will it propagate back to dbt docs?

No. The direction is only one way (from dbt docs → Holistics).

What is the relationship between the dbt model and Holistics' table model for the same underlying database table?

No direct relationship. Remember that there are 2 different concepts:

• dbt model: A SQL query that gets persisted into a database table.
• Holistics' Table Model: An abstraction on top of an existing database table.

I have updated the dbt's metadata, will they automatically sync with Holistics?

No. Currently, we don't support this yet.

If you have updated your dbt's metadata, you will need to re-push your latest manifest.json to Holistics. Once done, the dbt's metadata in Holistics is automatically updated.

We recommend that you should set up a re-sync strategy from your side so the file manifest.json is automatically pushed to Holistics on a regular (or trigger) basis.

I am using dbt Cloud, how can I integrate with Holistics?

Check out this guide to integrate dbt Cloud with Holistics.

Why my metadata from dbt is not populated in Holistics, although I have set up dbt integration successfully

Sometimes, when you have done setting up the dbt integration, your dbt descriptions and lineage diagram are not populated in the related Holistics model.

This is probably because your targeted schema when executing dbt run is different from the schema of the Holistics Model.

This issue happens quite frequently when you have 2 tables with the same name but in different schema in your Database (for e.g, prod.users table and dev.users table). And then when creating Holistics, you have accidentally selected the users table in dev schema, while your dbt model has targeted to prod schema.

Since we detect the metadata from dbt by mapping our models with dbt models via schema and database name, you would need to ensure that the schema (table name) and database name match between dbt model and Holistics model.