Skip to main content

Continuous Integration (CI/CD)

Introduction

Advanced users may want to create a CI/CD workflow in Holistics so that they can develop and manage Holistics project in a formal software development manner.

Your current setup

This document assumes you have a setup as the diagram below:

CI/CD Overview Setup

  • An ETL setup for your Data Warehouse transformation (dbt)
  • A BI layer (Holistics) to build reports and dashboards on top of the data in your Data Warehouse
  • The codes are hosted by a Git Service Provider (Github/Gitlab) that integrated with a CI/CD system (Github Action, CircleCI)

Overview of CI/CD Use Cases

This document discusses the 4 main use cases:

  1. dbt to Holistics Validation: When making changes to dbt models, analysts want to validate if the changes break or affect any of the BI reports.
    • Synchronized Development Preview: If possible, set it up such that analysts can synchronize & preview edits in dbt and BI at the same time.
  2. Dynamic Dev/Prod Environment
    • Configure your environment so that, during development, all models, datasets, and dashboards point to your development data sources or schema.
    • When you publish changes from development to production, these items will automatically switch to the production data sources or schema.
  3. Holistics validation (from models → dashboards)
    • Detect which dashboards, datasets are broken when you make a change in a Holistics data model
  4. PR Workflow and Auto-deployment:
    • Auto create a PR when developing in Holistics
    • Auto deploy new codes when the PR is merged (in Github/Gitlab)

Process Overview

Data Validation

Read more about how to validate dbt changes to Holistics

Dynamic Dev/Prod Environment

Read more about Dynamic Dev/Prod environment setup in Holistics

PR Workflow and Auto-deployment

Read more about PR Workflow and Auto-Deployment


Let us know what you think about this document :)