Databricks
Steps to prepare and connect to Databricks
Lightup account setup
Gather the following information:
- A Lightup personal access token
- Connection details for the compute cluster
- A Databricks personal access token
- The Server Hostname and HTTP Path
- If you want to connect to a specific Databricks Unity catalog, the catalog name
Step 1 - Get a personal access token
Lightup connects to a Databricks All Purpose Compute Cluster using a personal access token. If you don't already have one, you can generate one in Databricks on the Access Tokens tab of the User Settings page.
Step 2 - Get connection details for the compute cluster
-
Click
Compute in the sidebar.
-
Choose a cluster to connect to.
-
Navigate to Advanced Options.
-
Click on the JDBC/ODBC tab.
-
Copy the Server Hostname and HTTP Path field values.
- A Databricks cluster or a Databricks SQL endpoint.
- A Databricks personal access token.
- If you’re using a SQL endpoint, the SQL endpoint connection details. Specifically, you will need the Server Hostname and HTTP Path field values.
- If you’re using a cluster, the cluster connection details. Specifically, you will need the Server Hostname and HTTP Path field values.
Step 3: Get a Databricks personal access token
- Open your Databricks workspace.
- Click
Settings in the lower left corner of your Databricks workspace.
- Click User Settings.
- Go to the Access Tokens tab.
- Click the Generate New Token button.
- Optionally enter a description (comment) and lifetime.
- Click the Generate button.
- Copy the generated token and store in a secure location.
Step 4 - Get your Server Hostname and HTTP Path
Do one of the following:
-
If you're using a SQL endpoint:
- Click SQL Endpoints in the sidebar.
- Choose an endpoint to connect to.
- Navigate to the Connection Details tab.
- Copy the Server Hostname and the HTTP Path.\
-
If you're using a compute cluster:
- Click
Compute in the sidebar.
- Choose a cluster to connect to.
- Navigate to Advanced Options.
- Click on the JDBC/ODBC tab.
- Copy the Server Hostname and the HTTP Path.
- Click
Connector Setting
- Workspace URL - The Server Hostname of the compute cluster
- HTTP Path - The HTTP path for the compute cluster or the SQL Warehouse
- Token - The personal access token
- Catalog (Optional) - If you want to specify a Databricks Unity catalog, enter the catalog name here. You may leave this blank, in which case Lightup will connect to the default Databricks catalog, hive_metastore
Advanced
- Schema scan frequency - Set how often scans run for the datasource: Hourly, Daily, or Weekly.
Query governance
Databricks datasources support the Query history, Scheduling, and Enable data storage settings. For steps, see Set query governance settings for a datasource.
Updated 13 days ago