Dynamic Development/Production environment

Coming Soon

This feature is under development and will be coming soon!

Introduction

In a typical development workflow, you will have multiple environments

• Production Environment: is the most stable environment where your End-users consume the Dashboards
• Development Environment(s): isolated environment for your Data Analysts to develop and test their changes
• (Optional) Staging Environment: is the environment where all of the work of your Data Analysts are merged together for the final review before publishing to Production

In this process, you might have multiple Data Sources (or Schemas) for each Environments

• When you’re in Production Environment, all of your Dashboards (with Models + Datasets) point to the Production Data Source (or Schema).
• When you’re in Development Environment, all of your Dashboards (with Models + Datasets) point to the Development Data Source (or Schema)
  • You might have multiple Development Data Sources (or Schemas) for your Data Analysts
• When you’re in Staging Environment, all of your Dashboards (with Models + Datasets) point to the Staging Data Source (or Schema).

High-level Solution

info

The Data Source (or Schema) will be dynamically switched based on branch.

For example:

• branch develop: auto switch the database (or schema) to dev
• branch staging: auto switch the database (or schema) to stg
• branch master (or production): auto switch the database (or schema) to prod

Note that the pre-requisite of the flow is that you need to connect to all of your databases first (development, staging, production). Please refer to this link for more information

At a high level, you will need to create a database.aml file to define the logic for dynamically switching databases based on the current branch.

Holistics provides a global dynamic variable, H_ENV.git.current_branch, which returns the name of the current branch.

You can then use conditional statements (if-else) to dynamically switch the database or schema name based on the branch.

note

One key note: the file database.aml will be stored in git, so please don’t store sensitive information / credentials there

Detailed Setup

1. Create a databases.aml file to map the datasource based on the current branch

// H_ENV.git.current_branch is a dynamic variable, injected by Holistics

const current_branch = H_ENV.git.current_branch

// This is for dynamic database use case by branch
const dynamic_db = if (current_branch == 'master') {
                     'production'
                   } else if (current_branch == 'staging') {
                     'staging'
                   } else {
                     'develop'
                   }

// This is for dynamic schema use case by branch                
const dynamic_schema = if (current_branch == 'master') {
                        'prod'
                        } else if (current_branch == 'staging') {
                        'stg'
                        }
                        else {'dev'
                        }

2. Use the dynamic_db and dynamic_schema in datasets/models

Model users {
    label: 'Users'
    description: 'Info about Users'
    data_source_name: dynamic_db  
    table_name: '${dynamic_schema}.revenue'
}

Dataset ecommerce {
    label: 'Ecommerce'
    description: 'Dynamic Data Source Ecom Dataset'
    data_source_name: dynamic_db
    models: [users, orders]
    relationships: [orders.user_id > users.id, true]
}

3. Switching branch will switch the data source name of the related models and datasets

For example:

• If you change to develop branch => your database will be develop
• If you change to staging branch => your database will be staging
• If you deploy => your project on master branch will use the database production