Skip to main content

AML Persistence

Persistence can be found in

Persistence settings

Syntax format

persistence: FullPersistence | IncrementalPersistence {
schema: 'schema_name'
view_name: 'view_name'

// Parameters for IncrementalPersistence
incremental_column: 'updated_at'
primary_key: 'id'

on_cascade: 'rebuild' | 'reuse'

// Non-select query to optimize persisted table
custom_ddl: @sql ... ;;
}
Parameter nameRequirementDescription
schemarequiredSchema name that the persisted table will be written into
view_nameoptionalIf specified, the persisted table will have this name. If not, Holistics will automatically generate a name for the table.
incremental_columnoptional/requiredValues of this column will be used to check for new records. Only required in IncrementalPersistence
primary_keyoptional/requiredHolistics will refer to this column to replaced changed records in the current persisted table.Only required in IncrementalPersistence
on_cascadeoptionalSpecify the behavior of a model persistence when being triggered by a downstream model. Options: rebuild, reuse. Default: rebuild. Learn more
custom_ddloptionalBy declaring the custom_ddl (Custom Data Definition Language) in a persistence config, you can customize the way the persistence table is created in your Data Warehouse. Learn more

FullPersistence

Example:

persistence: FullPersistence {
schema: 'scratch' // persisted table will be written into this schema
view_name: 'pqm_orders' // Optional
}

IncrementalPersistence

Example:

persistence: IncrementalPersistence {
// Incremental only settings
incremental_column: 'updated_at'
primary_key: 'id'

// other options are the same as FullPersitence
}

Schedule settings

Steps

To set schedules to run persistences

  1. Create a schedules.aml file at the root of your AML project.
  2. Inside schedules.aml, define the schedules in a constant named schedules.

Schedule for Query Model Persistence

AML TypeSchedule
Parameter nameRequirementDescription
cronrequiredCron schedule expression
modelsrequiredList of Query Models to persist

Sample content of a schedules.aml file for scheduling Query Model Persistences:

const schedules = [
// Schedule to persist the Model `model_a` every 10 minutes
Schedule { cron: '0,10,20,30,40,50 * * * *', models: [model_a] }

// We can define another schedule using a different interval
Schedule { cron: '0 * * * *', models: [model_b] }

// We can also set multiple models to use the same schedule
Schedule { cron: '0 * * * *', models: [model_c, model_d] }
]

Schedule for PreAggregate Persistence

AML TypePreAggregateSchedule
Parameter nameRequirementDescription
cronrequiredCron schedule expression
objectrequiredObject containing the PreAggregates
(Currently, can only be a Dataset)
pre_aggregatesoptionalList of PreAggregate names (in the object) to persist.
Omit or leave this empty to persist all PreAggregates in the object.

Sample content of a schedules.aml file for scheduling Pre-Aggregate Persistences:

const schedules = [
// Schedule for specific PreAggregates in a Dataset
PreAggregateSchedule {
cron: '15 20 * * *'
object: ecommerce_dataset
pre_aggregates: ['agg_transactions'] // persist the PreAggregate 'agg_transactions' only
}

// Schedule for all PreAggregates in a Dataset
PreAggregateSchedule {
cron: '15 20 * * *'
object: ecommerce_dataset
}
]

Schedule for multiple types of persistences

You can schedule multiple types of persistences within the schedules list.

Sample content of a schedules.aml file with multiple types of schedules:

const schedules = [
Schedule {
cron: '0 * * * *'
models: [model_c, model_d]
}

PreAggregateSchedule {
cron: '15 20 * * *'
object: ecommerce_dataset
}
]

Cron schedule expression

Cron schedule expression

Here are a few links to help you get used to cron schedule expression:


Let us know what you think about this document :)