After you have a workspace ready, you'll want at least one datasource. A datasource is a connection from inside a workspace to a supported database. The data assets in a datasource (schemas, tables, columns) are the subject of all data quality management actions and options for the source database, within the workspace.
- The procedures on this page require a Workspace Admin role to complete.
- A datasource is only available in the workspace where it was created.
To monitor your data quality, you connect to each database you want to monitor. To make that possible, you'll need a Lightup user account for each of them. Some datasources use existing database connection APIs (such as ODBC/JDBC), and others have proprietary APIs. The steps to connect to a database vary by platform.
A datasource typically requires an account for Lightup in the target data asset (such as a database). Lightup supports numerous data platforms. Find steps on the corresponding subpage to prepare the lightup_user account before you connect to the datasource.
- In the left pane, open a workspace menu and select Datasources.
- In the main page select Create Datasource +.
- Enter a Datasource Name, then select a Connector Type.
- Provide connection information for the connector you chose. For details, see the Connector setting section of the corresponding subpage.
- After entering the required settings and any optional settings that apply, below the Connector setting section select Test Connection.
- After a successful connection test, select Save.
- Your new datasource appears in the list of available datasources. By default, these are listed in alphabetical order, so you might have to scroll or change the sort order to see your new datasource.
- Microsoft SQL
Not all query governance features are available for every data source connector. You might not see the feature, or it might appear but not be active. We're enabling query governance for more connectors. Look to the connector subpages for current information about query governance.
If you use Schedules to control when a datasource's metrics can run, be sure to allow enough time for the metric queries to complete. Otherwise, if there are metric queries that haven't run when a datasource's schedule ends, the pending queries will not run.
- On your workspace menu, select Query Governance.
- In the main page, under Datasources choose a datasource to govern.
- Set the query governance settings:
- Scheduling - If Scheduling is on, the datasource can be made available and unavailable to metrics by using user-created schedules. Some datasources support always allowing interactive queries (i.e., ignoring schedules when previewing a metric or retrieving sample data)— if so, you'll see a toggle, Always allow interactive queries.
- Query timeout - If Query timeout is on, any Lightup query (the basis of metrics) that hasn't returned a value within the interval specified by Terminate after (number with time unit) stops running (fails).
- Query date range limit- If Query date range limit is on, any Lightup query (the basis of metrics) that attempts to summarize data over a period that's longer that what's specified by Limit range to (number with time unit) is terminated (canceled).
- Enforce sort-keyed timestamps - If enabled, you can't configure metrics unless the data asset's timestamp column is indexed.
- Enable data storage - If enabled, record data that produced incidents can be stored.
The Query History view shows totals for: queries Completed, Failed, Cancelled, and Processing. This information helps you decide when to turn on/off query governors on active datasources. For example, if a lot of Lightup queries are failing, Query timeout might be set too low.
When you review Query History, the name column is a link that opens the Query Details panel.
- Metadata appears at the top.
- The SQL that ran appears below the metadata.
|Status||The query run's state|
|Start Time||When the query began|
|Duration||How long the query ran|
|Row count||How many rows the query returned|
|Type||Reflects the process that initiated the query|
|Query ID||The UUID of the object that initiated the query|
The Type field on the Query Details panel displays the process that initiated the query. There are six possible values, though one value (profilerMetric) is being deprecated.
|trainer||The monitor trainer or the incident validator|
|profilerMetric||The profiler (soon to be deprecated)|
|metric||The metric collector (will soon include current profilerMetric queries)|
|preview||The backend, when someone previews an existing metric|
|fullCompareMetric||The full compare collector|
|unknown||The backend, when someone previews a metric that has not yet been saved|
Below the metadata, the exact SQL statement that ran is displayed. This SQL statement can help you in a variety of ways, such as reproducing the issue in the source system or serving as a starting point for creating a custom SQL metric.
Lightup regularly scans datasources for changes, but you can also initiate a scan manually.
Lightup won't scan schemas that aren't Active.
- With the datasource selected in Explorer, select Actions > Manage Metrics.
- Near the top-right of the modal, select Run Manual Scan.
- The scan is queued to run as soon as possible. To stop a running scan, select Cancel Scan.
Updated 6 days ago