Holistics Docs - End-to-End Business Intelligence Platform

Holistics Documentation

Welcome to the Holistics Documentation page. You'll find comprehensive guides and documentation to help you start working with Holistics as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started
Suggest Edits

Holistics API

 

Holistics API

We allow you to programmatically work with Holistics API. For example:

  • You can programmatically retrieve CSV, XLSX and JSON data from any Holistics report.
  • You can use our API to trigger execution of an email schedule, transform or data import.
  • etc...

This is useful especially if you need to pass live data to your other applications. For example, you may want to feed a live CSV endpoint of your user segmentation list into your email marketing application so that you can always get fresh data upon retrieving them.

Allow user API access

First of all, you need to grant User permission to access API in User Management. Edit the User and tick on the checkbox Allow API access

API Key

To retrieve the data without the need to manually log in, you can generate a unique API key from each user's account.

Please visit your user settings and generate a user token.

New API Key

Then you can set the API key in request header and call our API. An example is our report data export API link.

API Request Authorization

When calling our API endpoints, we will authorize with your API key. Please set the custom X-Holistics-Key header to your API key.

Example:

curl -X POST \ 
  -H "Accept:application/json" \
  -H "Content-Type:application/json" \
  -H "X-Holistics-Key:api_key" \
  https://secure.holistics.io/data_imports/1/execute.json
Suggest Edits

Data Pipeline API

 

Data Pipeline API

This will be useful for setting up ETL workflow and controlling data pipeline on your side.

API Key

Please see guide to setup your API key.

Data Import

Execute Job

To execute a data import job, you will need to use the following HTTP endpoint:

POST /data_imports/<data_import_id>/execute.json

If successful, this endpoint will return the job ID of the job created.
To get the result, you will need to poll for the job data with Job API https://docs.holistics.io/reference#job-info-1

Data Transform

Execute Job

The same to Data Import, you can execute a data transform job with:

POST /data_transforms/<data_transform_id>/execute.json

Job Logs

You can view information about the running job using this endpoint:

GET jobs/<job_id>/logs.json

Check Last Run of Imports/Transforms

Sometimes you want to run a custom processing job daily at a particular time, but only if the ETL jobs ran successfully.
And you need a way to check the status of the ETL jobs.

In that case, you can use the jobs/get_last_jobs.json endpoint.

Parameters:

  • source_type: DataImport or DataTransform
  • ids: array of your transform/import IDs

Sample Request:

GET /jobs/last_run_jobs.json?source_type=DataTransform&ids[]=123&ids=[]456

Sample Responses:

{
  "123": {
    "id":11106256,
    "status":"running",
    "start_time":"2017-12-12T04:30:31Z",
    "end_time":null,
    "created_at":"2017-12-12T04:30:31Z"
  },
  "456": {
      "id":11106257,
      "status":"success",
      "start_time":"2017-12-12T04:30:31Z",
      "end_time":"2017-12-12T05:30:31Z",
      "created_at":"2017-12-12T04:30:31Z"
  }
}
Suggest Edits

Getting Report Data API

 

Reports API

This is useful especially if you need to pass live data to your other applications. For example, you may want to feed a live CSV endpoint of your user segmentation list into your email marketing application so that you can always get fresh data upon retrieving them.

API Key

Please see guide to setup your API key.

Submit Report Query

To submit a report query job, you will need to use the following HTTP endpoint:

GET /queries/<report_id>/submit_query.json?<filter_param1>=<filter_value1>...

If successful, this endpoint will return the job ID of the job created. To get the result, you will need to poll for the job status with

GET /queries/get_query_results.json?&job_id=<job_id>

Do note that the results are paginated, look into the paginated section of the response JSON for more information. To fetch the next page of data, append &_page=<page_number> to the URL.

Params:

  • _page_size: (optional) set the page size of the response
  • _page: (optional) set the page number of data to fetch

Submit Report Export Job

In order to download the CSV or Excel for a report's result set, you will need to use the following:

GET /queries/<report_id>/submit_export.<format>?<filter_param1>=<filter_value1>...
  • Supported format: csv/json/xlsx

Or if your query string is too long (> 500 letters), you can consider using a POST request:

POST /queries/<report_id>/submit_export.<format>

In the request body, specify the filter values in JSON format:

{
  "customer_ids": [1, 2, 3],
  "status": "active"
}
  • Supported format: csv/json

If successful, the endpoint will return the job ID for the export job created. To get the result, you will need to poll for the job status with either of these endpoints:

GET /queries/get_export_results.json?job_id=<job_id>

or this endpoint:

GET /jobs/<job_id>/get_results.json

Once the job status is 'success', you can download the result file via exports/download endpoint

GET /exports/download?job_id=<job_id>

Small dataset (< 10K rows)


This API is deprecated. Please use the asynchronous submit_export API instead.

For a small dataset (less than 10 thoudsands rows), you can use just this endpoint to download the data:

GET https://secure.holistics.io/queries/<report_id>.<format>?<filter_param1>=<filter_value1>...

How often does these API data refresh?

This is based on your report cache duration that you set in the report.

Suggest Edits

Permissions API

 

Permissions API

Share/Unshare Reports with Users

This allow you to share/unshare reports/dashboard with users

POST /permissions/share.json
POST /permissions/unshare.json
{
  "actions": [
    {
      "actor_class": "User",
      "actor_id": 123,
      "action": "read",
      "subject_class": "QueryReport",
      "subject_id": 456
    }
  ]
}
  • Available actor_class: User, Group
  • Available subject_class: QueryReport, Dashboard
  • Available action: read

The above example will simulate an action of current user (via API authentication), sharing report 456
with another user (id: 123), allowing user 123 to be able to read the report.

Suggest Edits

Package for Python

 

Our package (Holistics Package for Python) allows Python's user export report's data by inputting:

  • Your API-key
  • Report's ID
  • Dictionary of filters applied to that report

Notes:

Supported output format: DataFrame object, .CSV
Python version: >= 3, < 4


Installation

Package can be installed with pip:

pip install holistics

Alternatively, you can grab the latest source code from GitHub:

git clone git://github.com/holistics/holistics-python.git
cd holistics-python
python setup.py install


How to export data

Beginning by import holistics package

from holistics import HolisticsAPI


Next, creating an object of HolisticsAPI class with your API-key and Holistics server's url

obj = HolisticsAPI(api_key = 'aerg454hoiaKJGlgku', url = 'demo.holistics.io')

Args:


Finally, call export_data function with specific syntax:
    export_data (report_id, path, filters, page_size, page)

my_dataframe = obj.export_data(report_id='123456', path='C:/output.csv', 
                              filters={'date': '2017-04-28', 'vat': 1.1}, 
                              page_size = 12, page = 5)

Args:

  • report_id (str): ID of report. Get from URL.
  • path (str) (optional): If you want to store export data to local path, set path variable.
    • Default value: None
    • Ex: 'D:/Data/output.csv'
  • filters (dict) (optional): dictionary of filters that would be applied to report.
    • Default value: None
    • Ex: {'tenant': 'holistics', 'date': '2017-04-28'}
  • page_size (int) (optional): Set the page size of the response.
    • Default value: 10000000
  • page (int) (optional): Set the page number of data to fetch.
    • Default value: 10000000

Return:

A DataFrame object. If path is not None, save object as .csv file at that path.

Raises:

  • HTTPError: If the program can't connect to target site and get data.
    • You should check your API-key, url of Holistics and internet connection.
  • RuntimeError: If return status is Failure.
    • It could be caused by wrong SQL of your QueryReport.
  • ParserError: If program can't parse downloaded data as DataFrame object.
    • It could be caused when downloaded data is None.
Suggest Edits

Holistics CLI Commands

 

Holistics Command-line Tool

Holistics CLI is an interface to Holistics Data Preparation module (like data transport, data transform, import). This also allows you to perform tasks such as uploading CSV data back into Holistics using the command line, which can be helpful if you're working on datasets in Python for re-importing, for example.

Setting Up

Holistics CLI works with Ruby 2.0 onwards, if you don't have it installed on Linux/Mac OSX, we suggest you install it with RVM:

$ curl -sSL https://get.rvm.io | bash -s stable --ruby

After you have Ruby installed on your computer, then run:

$ gem install holistics

Help

For general help or a list of command, run:

$ holistics help

Here is a list of some Holistics CLI commands you can run:
Holistics CLI commands

Authentication (done once)

Next, perform authentication. You can get the token in Holistics (go to Settings, under API Key section). Specifiically, first grant user permissions to access API in User Management, then visit your user settings and generate a user token.

$ holistics login <token>
Authenticating token...
Authentication successful. Info:
- ID: 1
- Email: admin@you.com

Pushing CSV data back into Holistics via CLI (Example)

Once you have completed the steps to install your Holistics gem, you can perform Holistics operations without having to open the platform. For example, you can re-import CSV data that you have worked on into Holistics, using the following command:

holistics import csv -d, --dest-ds-id=DEST_DS_ID -f, --filepath=FILEPATH -t, --dest-table-name=DEST_TABLE_NAME  # Import a local CSV file to destination server

More details on importing via CLI can be found using help:
Holistics CLI import

Suggest Edits

Error status codes and responses

 

401 Unauthorized

This error indicates that you have provided an invalid API key.

{ "errors": ["Invalid API Key"] }

403 Forbidden

This error indicates that you have accessed a resource that you do not have permission.

{ "errors": ["You do not have permission to access this resource"] }

422 Unprocessable Entity

This error indicates that you have provided missing or invalid fields.

{ "errors": [ "Validation failed: Recipients can't be blank, Recipients list required" ] }
{
  "errors": [
    "param is missing or the value is empty: schedule"
  ]
}

404 Not Found

This error indicates that the resource you are retrieving is not found.

 { "errors": [ "Resource not found" ] }
 

Request

GET /jobs/:job_id.json

Parameters

  • job_id: Integer. Job ID

Response

{
  "id": 1138,
  "status": "success",
  "created_at": "2018-08-24T08:19:19.596Z",
  "source_id": 3,
  "source_type": "DataImport",
  "user_id": 3,
  "start_time": "2018-08-24T08:19:20.670Z",
  "end_time": "2018-08-24T08:19:20.924Z",
  "tenant_id": 2,
  "duration": 0.253676,
  "cancellable": true,
  "source_method": "execute"
}
Suggest Edits

Last Run Jobs

 

Request

GET /jobs/last_run_jobs.json

Parameters

  • source_type: Possible values: 'DataImport', 'DataTransform', 'EmailSchedule', 'ScheduleCache'
  • ids: IDs of the jobs' sources. These sources must have the same type specified in param source_type

Response

{
  "3": {
    "id": 1138,
      "status": "success",
        "start_time": "2018-08-24T08:19:20Z",
          "end_time": "2018-08-24T08:19:20Z",
            "created_at": "2018-08-24T08:19:19Z"
  }
}
Suggest Edits

Create Email Schedule

Create email schedule with settings.

 

A created email schedule detail would be responded if the payload is valid or an error if invalid or missing required fields.
By default, all email schedules in Holistics are based on UTC timezone.
The created email schedule will be matched with the assigned source ID and cannot change it later.

Request

POST /api/v1/email_schedules

Payload examples

  • Create email schedule with subject Company Report, start at 4:00 AM (UTC) every day for dashboard ID 5995, with format in Excel and date filter from 2019-03-25 only:
{
  "title": "Company Report",
  "recipients": [
    "khanh@holistics.io",
    "test@holistics.io"
  ],
  "bcc": [
    "bcc@holistics.io"
  ],
  "body_text": "sample text",
  "format": "excel",
  "source": {
    "type": "Dashboard",
    "id": 5995
  },
  "schedule": {
    "repeat": "0 4 * * *",
    "paused": false
  },
  "filter_values": {
    "start_date": "2019-03-25"
  }
}
  • Create email schedule start at 4:00 AM (UTC) every day for dashboard ID 5995, with format in Excel and password protected:
{
  "title": "Daily at 0:40 (UTC+0000)",
  "recipients": [
    "khanh@holistics.io",
    "test@holistics.io"
  ],
  "bcc": [
    "bcc@holistics.io"
  ],
  "body_text": "sample text",
  "format": "excel",
  "password": "1110291343546767",
  "source": {
    "type": "Dashboard",
    "id": 5995
  },
  "schedule": {
    "repeat": "0 4 * * *",
    "paused": true
  }
}

Fields

General

  • id: Numeric. Unique email schedule identifier
  • recipients: Array. List of email addresses.
  • bcc: Array. List of BCC addresses.
  • title: Text. Email subject. In default, source title will be used if title is not available.
  • body_text: Text. Text content inside email body.
  • format: Attachment format. Available values are txt, csv, excel, pdf or none. Default is excel.
  • source: Object. the source that this email schedule is referred to. Available only on retrieving data.
  • options: Object. Optional fields. Please see the Options for detail.
  • schedule: Object. Please see the Schedule for detail.
  • filter_values: Object. Filter values applied in this report/dashboard. Please see the Filter Values reference for detail.

Options

  • preview: Include preview inside email content. Default value true
  • include_pdf: Include a PDF of a report or the whole dashboard. The default value is false.
  • include_header: Include column headers in an attached file. The default is false. Not available if the format is none
  • password: false (or null) if Password Protection disabled, string value ≥ 4 characters if enabled. The delivery file format will be a password-protected zip file that contains the data generated for CSV attachment(s), or a password-protected Excel file for Excel attachment(s). Available only on creating and updating email schedule. Not available if format is none
  • selected_widgets: List of widget IDs are selected to include in the email. _all for selecting all widgets.
  • include_report_link: Include link to report / dashboard. The default is false.
  • dont_send_when_empty: Just send the email when data is available. The default is false.

Schedule

  • repeat: Crontab expression. Example: 0 4 * * * will start a job at 4:00 AM everyday. You can check your crontab expression by using https://crontab.guru

    We recommend that you should change the repeat time to prevent anti-spam filter from your email provider. For example 0 4 * * * (job start at 4:00 AM UTC) could be 1 4 * * * (job start at 4:01 AM UTC) or 5 4 * * *(job start at 4:05 AM UTC) . This will also help your database does not get overload when there are many email schedules that execute at the same time.

  • paused: Pause execution temporarily. The default is false.

Source

  • id: current source ID. You could find the Source ID via our docs.
  • type: current source type. You could find the Source Type via our docs.
  • title: source title, this will be used for field title if it is blank

Filter Values

This object is mainly matched by the filter variable that you created inside Holistics. The default value will be used if you do not input any specific data to it.

For example, we need to update an email schedule that have time_period filter with the default value is week, category filter with the default value is all and date_range filter default is 2 days ago. The filter_values in the payload would be:

"filter_values": {
    "time_period": "week",
    "category": "_all",
    "date_range": "2 days ago"
  }

Tips and Tricks

  • In case that you need to send the same dashboard/report to many recipients with custom filters, our recommendation is creating an email schedule using our UI first, then copy these values to create multiple email schedules with your desired filters.
Suggest Edits

Execute sending email

Trigger sending the email schedule by ID.

 

This is helpful for making sure the content is delivered correctly before going live.
An acknowledged message with Job ID as below would be returned if success or error if invalid email schedule ID.

Request

POST /api/v1/email_schedules/<email_schedule_id>/execute

Response

A job_id will be returned for the case you need Holistics support to have a look into your job.

{ "status": "OK", "job_id": 1 }
Suggest Edits

List Email Schedules

Retrieve all email schedules that you created via Holistics.

 

Request

GET /api/v1/email_schedules

Fields

General

  • id: Numeric. Unique email schedule identifier
  • recipients: Array. List of email addresses.
  • bcc: Array. List of BCC addresses.
  • title: Text. Email subject. In default, source title will be used if title is not available.
  • body_text: Text. Text content inside email body.
  • format: Attachment format. Available values are txt, csv, excel, pdf or none. Default is excel.
  • source: Object. the source that this email schedule is referred to. Available only on retrieving data.
  • options: Object. Optional fields. Please see the Options for detail.
  • schedule: Object. Please see the Schedule for detail.
  • filter_values: Object. Filter values applied in this report/dashboard. Please see the Filter Values reference for detail.

Options

  • preview: Include preview inside email content. Default value true
  • include_pdf: Include a PDF of a report or the whole dashboard. The default value is false.
  • include_header: Include column headers in an attached file. The default is false. Not available if the format is none
  • password_enabled: true this email schedule is protected by a password. Available only on retrieving data. Not available if format is none
  • selected_widgets: List of widget IDs are selected to include in the email. _all for selecting all widgets.
  • include_report_link: Include link to report / dashboard. The default is false.
  • dont_send_when_empty: Just send the email when data is available. The default is false.

Schedule

  • repeat: Crontab expression. Example: 0 4 * * * will start a job at 4:00 AM everyday. You can check your crontab expression by using https://crontab.guru

    We recommend that you should change the repeat time to prevent anti-spam filter from your email provider. For example 0 4 * * * (job start at 4:00 AM) could be 1 4 * * * (job start at 4:01 AM) or 5 4 * * *(job start at 4:05 AM) . This will also help your database does not get overload when there are many email schedules that execute at the same time.

  • paused: Pause execution temporarily. The default is false.

Source

  • id: current source ID.
  • type: current source type.
  • title: source title, this will be used for field title if it is blank

Filter Values

This object is mainly matched by the filter variable that you created inside Holistics. The default value will be used if you do not input any specific data to it.

Suggest Edits

Email Schedule Info

Get email schedule detail by specific email schedule ID.

 

The current email schedule detail would be responded if the payload is valid or an error if invalid email schedule ID.

Request

GET /api/v1/email_schedules/<email_schedule_id>

Fields

General

  • id: Numeric. Unique email schedule identifier
  • recipients: Array. List of email addresses.
  • bcc: Array. List of BCC addresses.
  • title: Text. Email subject. In default, source title will be used if title is not available.
  • body_text: Text. Text content inside email body.
  • format: Attachment format. Available values are txt, csv, excel, pdf or none. Default is excel.
  • source: Object. the source that this email schedule is referred to. Available only on retrieving data.
  • options: Object. Optional fields. Please see the Options for detail.
  • schedule: Object. Please see the Schedule for detail.
  • filter_values: Object. Filter values applied in this report/dashboard. Please see the Filter Values reference for detail.

Options

  • preview: Include preview inside email content. Default value true
  • include_pdf: Include a PDF of a report or the whole dashboard. The default value is false.
  • include_header: Include column headers in an attached file. The default is false. Not available if the format is none
  • password_enabled: true this email schedule is protected by a password. Available only on retrieving data. Not available if format is none
  • selected_widgets: List of widget IDs are selected to include in the email. _all for selecting all widgets.
  • include_report_link: Include link to report / dashboard. The default is false.
  • dont_send_when_empty: Just send the email when data is available. The default is false.

Schedule

  • repeat: Crontab expression. Example: 0 4 * * * will start a job at 4:00 AM everyday. You can check your crontab expression by using https://crontab.guru

    We recommend that you should change the repeat time to prevent anti-spam filter from your email provider. For example 0 4 * * * (job start at 4:00 AM) could be 1 4 * * * (job start at 4:01 AM) or 5 4 * * *(job start at 4:05 AM) . This will also help your database does not get overload when there are many email schedules that execute at the same time.

  • paused: Pause execution temporarily. The default is false.

Source

  • id: current source ID. You could find the Source ID via our docs.
  • type: current source type. You could find the Source Type via our docs.
  • title: source title, this will be used for field title if it is blank

Filter Values

This object is mainly matched by the filter variable that you created inside Holistics. The default value will be used if you do not input any specific data to it.

Suggest Edits

Update Email Schedule

Update an existing email schedule.

 

The updated email schedule detail would be responded if the payload is valid or an error if invalid or missing required fields. Cannot replace source in an existing email schedule.

Request

PATCH /api/v1/email_schedules/<email_schedule_id>

Payload examples

  • Update recipients
{
  "recipients": ["abc@holistics.io", "test@holistics.io"]
}
  • Change the title
{
  "title": "Company Overview - v3"
}
  • Change the attachment format
{
  "format": "csv"
}
  • Re-schedule sending email to 6:00 AM (UTC) every day
{
  "schedule": {
    "repeat": "0 6 * * *" 
  }
}

Fields

General

  • id: Numeric. Unique email schedule identifier
  • recipients: Array. List of email addresses.
  • bcc: Array. List of BCC addresses.
  • title: Text. Email subject. In default, source title will be used if title is not available.
  • body_text: Text. Text content inside email body.
  • format: Attachment format. Available values are txt, csv, excel, pdf or none. Default is excel.
  • options: Object. Optional fields. Please see the Options for detail.
  • schedule: Object. Please see the Schedule for detail.
  • filter_values: Object. Filter values applied in this report/dashboard. Please see the Filter Values reference for detail.

Options

  • preview: Include preview inside email content. Default value true
  • include_pdf: Include a PDF of a report or the whole dashboard. The default value is false.
  • include_header: Include column headers in an attached file. The default is false. Not available if the format is none
  • selected_widgets: List of widget IDs are selected to include in the email. _all for selecting all widgets.
  • include_report_link: Include link to report / dashboard. The default is false.
  • dont_send_when_empty: Just send the email when data is available. The default is false.

Schedule

  • repeat: Crontab expression. Example: 0 4 * * * will start a job at 4:00 AM everyday. You can check your crontab expression by using https://crontab.guru

    We recommend that you should change the repeat time to prevent anti-spam filter from your email provider. For example 0 4 * * * (job start at 4:00 AM) could be 1 4 * * * (job start at 4:01 AM) or 5 4 * * *(job start at 4:05 AM) . This will also help your database does not get overload when there are many email schedules that execute at the same time.

  • paused: Pause execution temporarily. The default is false.

Filter Values

This object is mainly matched by the filter variable that you created inside Holistics. The default value will be used if you do not input any specific data to it.

For example, we need to update an email schedule that have time_period filter with the default value is week, category filter with the default value is all and date_range filter default is 2 days ago. The filter_values in the payload would be:

"filter_values": {
    "time_period": "week",
    "category": "_all",
    "date_range": "2 days ago"
  }
Suggest Edits

Delete Email Schedule

Delete an existing email schedule.

 

An acknowledged message as below would be returned if success or error if invalid email schedule ID.

Request

DELETE /api/v1/email_schedules/<email_schedule_id>

Response

{
  "status": "OK"
}