Holistics Docs - End-to-End Business Intelligence Platform

Holistics Documentation

Welcome to the Holistics Documentation page. You'll find comprehensive guides and documentation to help you start working with Holistics as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started

Storage Settings

What is Storage Settings?

Every time you explore or query a SQL model, the whole query is run again. These operations can have low performance, put unnecessary strains on your resources, or become costly (in case of pay-per-use data warehouses like BigQuery) if the query is unoptimized.

To avoid this situation, Storage Settings helps you periodically run the model's query and write the result to a physical table in your database.

With Storage Settings enabled, when users explore or query from a SQL model, they will interact with the data table instead. This will give you more power to optimize query performance and cost.

Storage Settings can be coupled with dependencies to create a flow-based storage schedule to ensure data freshness and consistency.

Setting up Storage

At any SQL model or third-party model, you can toggle on Storage Settings:

The following window will popup:

  • Schema Name, Table Name: Choose the destination schema and name for the physical table. Note that you must have write access to the schema you chose.
  • Schedule: Schedule to run the storage process.
  • Mode:
    • Full: Whenever the storage process is run, the whole result set of the SQL will be used to replace the previously created table.
      This should be used if your data is small, the records change regularly and you do not need to retain the history of your data.
    • Append: This should be used if you want to retain the history of your data.
      When the storage is run in Append Mode, all records from the source table at run time will be appended to the destination table, and old records are left untouched.
    • Incremental: This should be used if your data is large, but past records do not change.
      New records from the model's query will be appended to the destination table. For this to work, you need to specify an increment column so Holistics can decide on the correct data to extract.
    • Upsert: In case your data is large and past records change, this should be used.
      You will need to specify a Primary Key and an increment column. How it works:
      • If the new records from the model's SQL carry new Primary Key values, those records will be appended to the table
      • If the new records' Primary Key values already existed, and the increment column's values are new, then the old records will be replaced.
  • Advanced Settings (Column Format, Indexing, Flow-based Schedule...): coming soon

Updated about 20 hours ago


What's Next

Dataset

Storage Settings


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.