Databricks
Steps to prepare and connect to Databricks
Prepare in Databricks
To create a Databricks connection, you'll need an access token and the server hostname and path.
Step 1: Get a Databricks personal access token
- Generate a personal access token in Databricks.
- Copy the generated token and store in a secure location.
Step 2 - Get your Server Hostname and HTTP Path
Do one of the following:
-
If you're using a SQL endpoint:
- Click SQL Warehouses in the left nav.
- Choose an endpoint to connect to.
- Navigate to the Connection Details tab.
- Copy the Server Hostname and the HTTP Path.\
-
If you're using a compute cluster:
- Click Compute in the left nav.
- Choose a cluster to connect to.
- Navigate to Advanced Options.
- Click on the JDBC/ODBC tab.
- Copy the Server Hostname and the HTTP Path.
Configure connector
- Workspace URL - The Server Hostname of the compute cluster
- HTTP Path - The HTTP path for the compute cluster or the SQL Warehouse
- Token - The Databricks personal access token
- Catalog (Optional) - If you want to specify a Databricks Unity catalog, enter the catalog name here. You may leave this blank, in which case Lightup will connect to the default Databricks catalog, hive_metastore
Advanced
- Schema scan frequency - Set how often scans run for the datasource: Hourly, Daily, or Weekly.
Query Governance
Databricks datasources support the Query History, Scheduling, Enable data storage, and Maximum backfill duration, and Maximum distinct values settings. For steps, see Set query governance settings for a datasource.
Date/time data types
These Databricks date/time data types are supported:
Object types
These Databricks object types are supported:
- Tables
- Views
Partitions
Databricks datasources support partitions, multi-partitions, and partition time zones.
Updated 10 days ago