Dynamic Data Sources for Development/Production Environment

Coming Soon

This feature is under development and will be coming soon!

info

Holistics support this thanks to our ability to define dynamic data sources.

Introduction

In a typical development workflow, you'll have several environments: Production, Development, and Staging, each using different data sources.

This document will show you how to set up this workflow in Holistics.

How it works

At a high level, the data source will dynamically switch based on which branch users are on.

Holistics provides a global dynamic variable, H.git.current_branch, which returns the name of the current branch.

You can then use conditional statements (if-else) to dynamically switch the database or schema name based on the branch.

1. Connect databases

Note that the pre-requisite of the flow is that you need to connect to all of your databases first (development, staging, production). Please refer to this link for more information.

2. Create a databases.aml file to map the datasource based on the current branch

// H.git.current_branch is a dynamic variable, injected by Holistics

const current_branch = H.git.current_branch

// This is for dynamic database use case by branch
const dynamic_db = if (current_branch == 'master') {
                     'production'
                   } else if (current_branch == 'staging') {
                     'staging'
                   } else {
                     'develop'
                   }

// This is for dynamic schema use case by branch                
const dynamic_schema = if (current_branch == 'master') {
                        'prod'
                        } else if (current_branch == 'staging') {
                        'stg'
                        }
                        else {'dev'
                        }

Note

The file database.aml will be stored in git, so please don’t store sensitive information credentials there

3. Use the dynamic_db and dynamic_schema in datasets/models

Model users {
    label: 'Users'
    description: 'Info about Users'
    data_source_name: dynamic_db  
    table_name: '${dynamic_schema}.revenue'
}

Dataset ecommerce {
    label: 'Ecommerce'
    description: 'Dynamic Data Source Ecom Dataset'
    data_source_name: dynamic_db
    models: [users, orders]
    relationships: [orders.user_id > users.id, true]
}

4. Switching branch will switch the data source name of the related models and datasets

For example:

• If you change to develop branch => your database will be develop
• If you change to staging branch => your database will be staging
• If you deploy => your project on master branch will use the database production