Manage datasources

Set up accounts and connections

After you have a workspace ready, you'll want at least one datasource. A datasource is a connection from inside a workspace to a supported database. The data assets in a datasource (schemas, tables, columns) are the subject of all data quality management actions and options for the source database, within the workspace.

❗️

  • The procedures on this page require a Workspace Admin role to complete.
  • A datasource is only available in the workspace where it was created.

To monitor your data quality, you connect to each database you want to monitor. To make that possible, you'll need a Lightup user account for each of them. Some datasources use existing database connection APIs (such as ODBC/JDBC), and others have proprietary APIs. The steps to connect to a database vary by platform.

📘

A datasource typically requires an account for Lightup in the target data asset (such as a database). Lightup supports numerous data platforms. Find steps on the corresponding subpage to prepare the lightup_user account before you connect to the datasource.

Connect a datasource

  1. In the left pane, open a workspace menu and select Datasources.
  2. In the main page select Create Datasource +.
1616
  1. Enter a Datasource Name, then select a Connector Type.
  2. Provide connection information for the connector you chose. For details, see the Connector setting section of the corresponding subpage.
  3. After entering the required settings and any optional settings that apply, below the Connector setting section select Test Connection.
  4. After a successful connection test, select Save.
  5. Your new datasource appears in the list of available datasources. By default, these are listed in alphabetical order, so you might have to scroll or change the sort order to see your new datasource.

Connector subpages

Set query governance settings for a datasource

📘

Not all query governance features are available for every data source connector. You might not see the feature, or it might appear but not be active. We're enabling query governance for more connectors. Look to the connector subpages for current information about query governance.

1563

🚧

If you use Schedules to control when a datasource's metrics can run, be sure to allow enough time for the metric queries to complete. Otherwise, if there are metric queries that haven't run when a datasource's schedule ends, the pending queries will not run.

  1. On your workspace menu, select Query Governance.
  2. In the main page, under Datasources choose a datasource to govern.
  3. Set the query governance settings:
    • Scheduling - If Scheduling is on, the datasource can be made available and unavailable to metrics by using user-created schedules. Some datasources support always allowing interactive queries (i.e., ignoring schedules when previewing a metric or retrieving sample data)— if so, you'll see a toggle, Always allow interactive queries.
    • Query timeout - If Query timeout is on, any Lightup query (the basis of metrics) that hasn't returned a value within the interval specified by Terminate after (number with time unit) stops running (fails).
    • Query date range limit- If Query date range limit is on, any Lightup query (the basis of metrics) that attempts to summarize data over a period that's longer that what's specified by Limit range to (number with time unit) is terminated (canceled).
    • Enforce sort-keyed timestamps - If enabled, you can't configure metrics unless the data asset's timestamp column is indexed.
    • Enable data storage - If enabled, record data that produced incidents can be stored.

Review Query History

The Query History view shows totals for: queries Completed, Failed, Cancelled, and Processing. This information helps you decide when to turn on/off query governors on active datasources. For example, if a lot of Lightup queries are failing, Query timeout might be set too low.​

740

Review Query Details

When you review Query History, the name column is a link that opens the Query Details panel.

1053
  1. Metadata appears at the top.
  2. The SQL that ran appears below the metadata.

Query Details metadata

FieldDescription
StatusThe query run's state
Start TimeWhen the query began
DurationHow long the query ran
Row countHow many rows the query returned
TypeReflects the process that initiated the query
Query IDThe UUID of the object that initiated the query

Types of queries

The Type field on the Query Details panel displays the process that initiated the query. There are six possible values, though one value (profilerMetric) is being deprecated.

TypeQueried by
trainer The monitor trainer or the incident validator
profilerMetric The profiler (soon to be deprecated)
metricThe metric collector (will soon include current profilerMetric queries)
previewThe backend, when someone previews an existing metric
fullCompareMetricThe full compare collector
unknownThe backend, when someone previews a metric that has not yet been saved

Query Details SQL

Below the metadata, the exact SQL statement that ran is displayed. This SQL statement can help you in a variety of ways, such as reproducing the issue in the source system or serving as a starting point for creating a custom SQL metric.

Scan a datasource

Lightup regularly scans datasources for changes, but you can also initiate a scan manually.

🚧

Lightup won't scan schemas that aren't Active.

  1. With the datasource selected in Explorer, select Actions > Manage Metrics.
  2. Near the top-right of the modal, select Run Manual Scan.
  3. The scan is queued to run as soon as possible. To stop a running scan, select Cancel Scan.
988