Dynamic Data Sources for Development/Production Environment

Coming Soon

This feature is under development and will be coming soon! If you want to request for beta testing, please sign up here

info

Holistics support this thanks to our ability to define dynamic data sources.

Introduction

In a typical development workflow, you'll have several environments: Production, Development, and Staging, each using different data sources.

This document will show you how to set up this workflow in Holistics.

How it works

At a high level, the data source will dynamically switch based on which branch users are on.

Holistics provides a global dynamic variable, H.git.current_branch, which returns the name of the current branch.

You can then use conditional statements (if-else) to dynamically switch the database or schema name based on the branch.

1. Connect databases

Note that the pre-requisite of the flow is that you need to connect to all of your databases first (development, staging, production). Please refer to this link for more information.

2. Create a databases.aml file to map the datasource based on the current branch

// H.git.current_branch is a dynamic variable, injected by Holistics
const current_branch = H.git.current_branch

// This is for dynamic database use case by branch
const dynamic_db = if (H.git.current_branch == 'master') {
                     'production'
                   } else if (H.git.current_branch == 'staging') {
                     'staging'
                   } else {
                     'develop'
                   }

// This is for dynamic schema use case by branch                
const dynamic_schema = if (H.git.current_branch == 'master') {
                         'prod'
                       } else if (H.git.current_branch == 'staging') {
                         'stg'
                       } else {
                         'dev'
                       }

Note

The file database.aml will be stored in git, so please don’t store sensitive information credentials there

3. Use the dynamic_db and dynamic_schema in datasets/models

Model users {
    label: 'Users'
    description: 'Info about Users'
    data_source_name: dynamic_db  
    table_name: '${dynamic_schema}.revenue'
}

Dataset ecommerce {
    label: 'Ecommerce'
    description: 'Dynamic Data Source Ecom Dataset'
    data_source_name: dynamic_db
    models: [users, orders]
    relationships: [orders.user_id > users.id, true]
}

4. Switching branch will switch the data source name of the related models and datasets

For example:

• If you change to develop branch => your database will be develop
• If you change to staging branch => your database will be staging
• If you deploy => your project on master branch will use the database production

Available Variables

While not exhaustive, here are some common system variables you can use:

• H.git.current_branch → this will return the name of the current branch
• H.git.is_production → this will return true when in Reporting tab, or in Development tab with production mode enabled.

Requesting for beta

We are kicking off a beta and would love for you to be involved! Sign up here if you want to join in.