# Connect to Databricks

> Learn how to connect Databricks to Holistics as a data source

## Introduction
This documentation provides a comprehensive guide on connecting Databricks to Holistics as a data source. In the following sections, we will walk you through the necessary steps:

- Creating a personal access token and granting permissions for Holistics to interact with Databricks.
- Establishing the connection between Databricks and Holistics as a data source.

## Create Databricks personal access token for Holistics

1. Create a personal access token for a Databricks user following the [Databricks Personal Access Tokens Documentation](https://docs.databricks.com/en/dev-tools/auth.html#databricks-personal-access-token-authentication)

2. Assign specific privileges to your `DATABRICKS_USERNAME` as outlined below:

- Grant [USE CATALOG](https://docs.databricks.com/en/data-governance/unity-catalog/index.html#catalogs) privilege on your catalog. This permits the use of a catalog. Refer to the [Catalogs Guide](https://docs.databricks.com/en/data-governance/unity-catalog/index.html#catalogs) for more information.

- Grant [USE SCHEMA](https://docs.databricks.com/en/data-governance/unity-catalog/index.html#schemas) privilege on your schema. This permits the use of a schema. Find additional details in the [Schemas Guide](https://docs.databricks.com/en/data-governance/unity-catalog/index.html#schemas).

- Grant [SELECT](https://docs.databricks.com/en/data-governance/unity-catalog/index.html#tables) privilege on the tables, it is required for users to query a table. Refer to the [Tables Guide](https://docs.databricks.com/en/data-governance/unity-catalog/index.html#tables) for more information.

For further insights, we recommend reading about [Databricks permission](https://docs.databricks.com/en/data-governance/unity-catalog/index.html#what-is-unity-catalog).

## Obtain Databricks Connection Details
Obtain these details by following [Databricks Doc](https://docs.databricks.com/en/integrations/compute-details.html).  
You will need these details when creating a Databricks data source [below](#connect-databricks-to-holistics-as-a-data-source).

## Connect Databricks to Holistics as a data source

From the Holistics app's header bar, open **Organization Settings** menu -> **Data Sources**.

Click **New Data Source** button and fill in the details:

- **Database Type**: Select **Databricks**.
- **Display Name**: Provide a name for your connection.
- **Host**: Databricks Server hostname (from [Databricks Connection Details](#obtain-databricks-connection-details))
- **Port**: Databricks Port (from [Databricks Connection Details](#obtain-databricks-connection-details))
- **Username**: Databricks username of the dedicated Holistics user (created [above](#create-databricks-personal-access-token-for-holistics)).
  * **NOTE**: If you are using a Personal Access Token, input the text `token` in this field.
- **Password**: Databricks Personal Access Token or password of the dedicated Holistics user (created [above](#create-databricks-personal-access-token-for-holistics)).
- **HTTP Path**: Databricks HTTP path (from [Databricks Connection Details](#obtain-databricks-connection-details))
- **Catalog**: The Databricks Catalog that you want to connect Holistics to.

Click on **Test connection** to ensure the connection works, and then **Save** to complete the process.

Now you can start using your Databricks data source within Holistics.

## FAQs

### Does Holistics support legacy Hive metastore?
    
While Holistics can execute queries on a data warehouse in a legacy Hive metastore, it does not provide support for fetching the database schema from it.
    
Suggested Solutions/Workarounds:
    
- Databricks recommends migrating tables managed by the Hive metastore to the Unity Catalog metastore. Guidelines for this process can be found [here](https://docs.databricks.com/en/data-governance/unity-catalog/migrate.html).
- Although fetching the database schema is not possible, users can construct the modeling by creating [Query Models](/reference/aml/query-model). However, creating [Table Models](/reference/aml/table-model) will not be feasible without the database schema.

### How to query tables from multiple catalogs in Databricks?

Databricks supports querying across multiple catalogs using a [three-level namespace](https://docs.databricks.com/en/data-governance/unity-catalog/queries.html#three-level-namespace-notation) that includes the catalog, schema, and table. However, Holistics Modeling primarily supports the schema and table levels, making direct querying of tables from multiple databases currently impossible.

**Workaround**: 

Assuming your current catalog (in DataSource connection form) is catalog1, and you want to join with data in catalog2, create a Query Model that refers to tables in Catalog2. Create a [Query Model](/docs/query-models) instead of a 'table model' to link to your database table:

```aml
Model catalog2 {
  type: 'query'
  data_source_name: 'databricks'

  dimension field1 {
    type: 'number'
  }
  ...

  query: @sql select * from catalog2.public.users ;;
}
    
```
    
This model fetches data from `catalog2.public.users` (Catalog2) rather than the current Catalog1. You can treat this model like a standard table model and create relationships to connect it with other models in Catalog1.

Note: Ensure you have [USE CATALOG](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-use-catalog.html) permission on all catalogs.
### Why does my query/report occasionally take a very long time to respond?
    
There are some common reasons:
    
- Databricks has a feature to [terminate the database/cluster after it goes idle](https://docs.databricks.com/en/clusters/clusters-manage.html#automatic-termination). After termination, a new query may take a few **minutes** to [restart](https://docs.databricks.com/en/clusters/clusters-manage) the database before execution.html#cluster-autostart-for-jobs-and-jdbcodbc-queries) the database first before actually getting executed.
- You can also check out **[Performance Troubleshooting](/docs/performance/troubleshooting)** documentation.