Continuous Integration (CI/CD)
Introduction
Advanced users may want to create a CI/CD workflow in Holistics so that they can develop and manage Holistics project in a formal software development manner.
Your current setup
This document assumes you have a setup as the diagram below:
- An ETL setup for your Data Warehouse transformation (dbt)
- A BI layer (Holistics) to build reports and dashboards on top of the data in your Data Warehouse
- The codes are hosted by a Git Service Provider (Github/Gitlab) that integrated with a CI/CD system (Github Action, CircleCI)
Overview of CI/CD Use Cases
This document discusses the 4 main use cases:
- dbt to Holistics Validation: When making changes to dbt models, analysts want to validate if the changes break or affect any of the BI reports.
- Synchronized Development Preview: If possible, set it up such that analysts can synchronize & preview edits in dbt and BI at the same time.
- Dynamic Dev/Prod Environment
- Configure your environment so that, during development, all models, datasets, and dashboards point to your development data sources or schema.
- When you publish changes from development to production, these items will automatically switch to the production data sources or schema.
- Holistics validation (from models → dashboards)
- Detect which dashboards, datasets are broken when you make a change in a Holistics data model
- PR Workflow and Auto-deployment:
- Auto create a PR when developing in Holistics
- Auto deploy new codes when the PR is merged (in Github/Gitlab)
Process Overview
Data Validation
Read more about how to validate dbt changes to Holistics
Dynamic Dev/Prod Environment
Read more about Dynamic Dev/Prod environment setup in Holistics