Job Queue System and Workers
This document talks about Holistics Job Queue System: How Holistics processes requests whenever users open a dashboard.
High level Mechanism​
In Holistics, when a user opens a report, we construct a SQL query that is sent to the customer's data warehouse, wait for it to finish and visualize the results.
Since the analytical SQL queries take time (seconds to minutes), it is usually not a good idea to handle this using synchronous web requests. A more scalable solution is to use a background job queue system.
A typical flow would look like:
- When a user views a report, a job is created and pushed into a job queue.
- A worker picks up the job, constructs the SQL queries, then runs them against customer’s data warehouse
- Once the query is finished, the result set is visualized and presented to the user’s browser.
What kind of actions will create a job?
Usually actions that involve running a SQL against the customer’s data warehouse:
- Users viewing dashboards
- Email schedules triggered
- Etc.
What is a Worker/Concurrent Worker?​
A worker (or concurrent worker) is an actor that processes jobs pushed into the queue. Jobs in a queue are processed sequentially. An available worker will pick up the next job and process it. Once done, the worker releases the job and picks up the next one.
Think of workers as the staff behind the counters when you visit a bank: You queue in a line; and when it’s your turn, the next available bank officer will attend to you.
Each Holistics customer (tenant) has a fixed number of workers that can be adjusted/purchased. The more concurrent workers a tenant has, the higher volume of concurrent queries one can run.
Type of Job Queues​
Each Holistics customer has their own job queue and workers. This ensures one customer overloading the job queue will have zero to little effect to other customers’ system.
Furthermore, depending on the nature of the job, it will be classified into different queues (or pools). For example, a Report job runs in a different queue than a Data Transform job.
Life Cycle of a Job​
| State | Description |
|---|---|
| created | When a job is first triggered (whenever you click Submit or Create button), it will have created status. |
| queued | Depend on the default slot limit of a specific queue in your tenant/company, if the slot is currently not fully occupied, the created job will be queued. |
| running | Depend on the available slots of our concurrent running workers, Holistics will then execute/run jobs which are currently being queued. |
| success | If the job runs successfully, it will have success status. |
| failure | If the job runs unsuccessfully, it will have failure status. |
| cancelling | While a job is running, if you manually cancel the job, it will have cancelling status. |
| cancelled | If the job is cancelled successfully, it will have cancelled status. |
| already_existed | When a job have this status, it means that this job currently coincides with the previous jobs created / queued / being run. |
If you want to enabled our Embedded Analytics feature, please refer to our doc about Embedded Analytics for more information.
Default slots for specific job queues​
Below is the current list of queues and their default worker count for each account.
Normal Queue​
| Queue | Default Slot | Action included |
|---|---|---|
| Default | 5 | 1. Create/Update Custom Field 2. Refresh Models and Dependant Models 3. Run Adhoc Query 4. Explore Dataset |
| Filter | 3 | 1. Filter suggestion 2. Process filter in Dashboard |
| Report | 10 | Execute report/widget |
| Prefetch | 2 | 1. Synchronize Schema 2. Prefetch Filter Cache |
| Preview | 3 | 1. Validate Data Import 2. Preview Report/Query |
| Export | 10 | 1. Export Dashboard 2. Export Dashboard Widget/Report |
| Email Schedule | 2 | Executing schedule (Email, Slack, SFTP, Google Sheet) |
| Data Import | 2 | Executing Data Import |
| Data Transform | 2 | Executing Data Transform (or Storage Settings) |
| Validate | 5 | 1. Validate Table Structure in Data Transform 2. Validate Query in Data Transform 3. Preview Data Transform |
Embed Analytics Queue​
| Embed Analytics Queue | Default Slot | Action included |
|---|---|---|
| Embed | 0 | 1. Execute Embedded Dashboard Widget/Report 2. Export Embedded Dashboard 3. Export Embedded Dashboard Widget/Report |
Your account’s configuration might be different from the default above. Please contact us by sending an email to [email protected] to find out your current setup.
Do note that the Embedded Analytics feature utilizes a special type of worker called Embed Worker. They are separate and can be manually adjusted from the Embed Analytics Manager.