AML Persistence

Persistence can be found in

• Query Model
• Pre-Aggregate

Persistence settings

Syntax format

persistence: FullPersistence | IncrementalPersistence {
  schema: 'schema_name'
  view_name: 'view_name'

  // Parameters for IncrementalPersistence
  incremental_column: 'updated_at'
  primary_key: 'id'

  on_cascade: 'rebuild' | 'reuse'

  // Non-select query to optimize persisted table
  custom_ddl: @sql ... ;; 
  }

Parameter name	Requirement	Description
schema	required	Schema name that the persisted table will be written into
view_name	optional	If specified, the persisted table will have this name. If not, Holistics will automatically generate a name for the table.
incremental_column	optional/required	Values of this column will be used to check for new records. Only required in IncrementalPersistence
primary_key	optional/required	Holistics will refer to this column to replaced changed records in the current persisted table.Only required in IncrementalPersistence
on_cascade	optional	Specify the behavior of a model persistence when being triggered by a downstream model. Options: rebuild, reuse. Default: rebuild. Learn more
custom_ddl	optional	By declaring the custom_ddl (Custom Data Definition Language) in a persistence config, you can customize the way the persistence table is created in your Data Warehouse. Learn more

FullPersistence

Example:

persistence: FullPersistence {
  schema: 'scratch'       // persisted table will be written into this schema
  view_name: 'pqm_orders' //  Optional
}

IncrementalPersistence

Example:

persistence: IncrementalPersistence {
  // Incremental only settings
  incremental_column: 'updated_at'
  primary_key: 'id'

  // other options are the same as FullPersitence
}

Schedule settings

Steps

To set schedules to run persistences

1. Create a schedules.aml file at the root of your AML project.
2. Inside schedules.aml, define the schedules in a constant named schedules.

Schedule for Query Model Persistence

AML Type	Schedule

Parameter name	Requirement	Description
cron	required	Cron schedule expression
models	required	List of Query Models to persist

Sample content of a schedules.aml file for scheduling Query Model Persistences:

const schedules = [
  // Schedule to persist the Model `model_a` every 10 minutes
  Schedule { cron: '0,10,20,30,40,50 * * * *', models: [model_a] }

  // We can define another schedule using a different interval
  Schedule { cron: '0 * * * *', models: [model_b] }

  // We can also set multiple models to use the same schedule
  Schedule { cron: '0 * * * *', models: [model_c, model_d] }
]

Schedule for PreAggregate Persistence

AML Type	PreAggregateSchedule

Parameter name	Requirement	Description
cron	required	Cron schedule expression
object	required	Object containing the PreAggregates (Currently, can only be a Dataset)
pre_aggregates	optional	List of PreAggregate names (in the object) to persist. Omit or leave this empty to persist all PreAggregates in the object.

Sample content of a schedules.aml file for scheduling Pre-Aggregate Persistences:

const schedules = [
  // Schedule for specific PreAggregates in a Dataset
  PreAggregateSchedule {
    cron: '15 20 * * *'
    object: ecommerce_dataset
    pre_aggregates: ['agg_transactions'] // persist the PreAggregate 'agg_transactions' only
  }

  // Schedule for all PreAggregates in a Dataset
  PreAggregateSchedule {
    cron: '15 20 * * *'
    object: ecommerce_dataset
  }
]

Schedule for multiple types of persistences

You can schedule multiple types of persistences within the schedules list.

Sample content of a schedules.aml file with multiple types of schedules:

const schedules = [
  Schedule {
    cron: '0 * * * *'
    models: [model_c, model_d]
  }

  PreAggregateSchedule {
    cron: '15 20 * * *'
    object: ecommerce_dataset
  }
]

Cron schedule expression

Cron schedule expression

Here are a few links to help you get used to cron schedule expression:

• https://crontab.guru: translate cron expression to natural language
• https://crontab.guru/examples.htm: examples of frequently used expressions