Connect to datasources

After you have a workspace ready, you'll want at least one datasource. A datasource is a connection from inside a workspace to a supported database. The data assets in a datasource (schemas, tables, columns) are the subject of all data quality management actions and options for the source database, within the workspace.

  • A DBA must set up a Lightup account in any source database to enable Lightup datasource connections. Lightup users will need some information about the source database and the Lightup account in order to create a datasource connection.
  • A datasource exists within a workspace— it points to a source database. You can add a datasource connection in each workspace where you need it. This allows people working in different workspaces to independently analyze and monitor the same source databases.
  • You must have the Workspace Admin role to create or edit a datasource.
  • You'll need to connect to your datasource with credentials. Each datasource has its own steps to connect. See the connector subpage for details.

Connect a datasource

  1. In the left pane, open a workspace menu and select Datasources.
  2. In the main page select Create Datasource +.
  1. Enter a Datasource Name, then select a Connector Type.
  2. Provide connection information for the connector you chose. For details, see the Configure connector section of the connector subpage. You will need some connection information from the DBA for the source database.
  3. After entering the required settings and any optional settings that apply, below the Configure connector section select Test Connection.
  4. After a successful connection test, select Save.
  5. Your new datasource appears in the list of available datasources. By default, these are listed in alphabetical order, so you might have to scroll or change the sort order to see your new datasource.

Connector subpages

Set query governance settings for a datasource

  1. On your workspace menu, select Query Governance.

  2. In the main page, under Datasources choose a datasource to govern.

  3. Change any query governance settings, then select Save.

    • Scheduling - Enables using schedules to control when the datasource is available to metrics.
      Some datasources support always allowing interactive queries (i.e., ignoring schedules when previewing a metric or retrieving sample data)— if so, you'll see a toggle, Always allow interactive queries.
      If you use Schedules to control when a datasource's metrics can run, be sure to allow enough time for the metric queries to complete. Otherwise, if there are metric queries that haven't run when a datasource's schedule ends, the pending queries will not run.
    • Query timeout - Enables automatic canceling of any Lightup query (the basis of metrics) that hasn't returned a value within the specified Terminate after interval.
    • Query date range limit- Enables automatic canceling of any Lightup query (the basis of metrics) that attempts to summarize data over a period that's longer than the specified Limit range to interval.
    • Enforce sort-keyed timestamps - Prevents configuring a metric if the data asset's timestamp column isn't indexed.
    • Enable data storage - Enables storage of records (data) that produced incidents.
    • Maximum backfill duration - Use to set a maximum backfill duration that's shorter than the system limit of 11,000 data points. For more information about the effect of this setting, see Backfill duration.
    • Maximum distinct values - Prevents collection of Categorical distribution metrics if the number of distinct values exceeds this maximum. Once the number of distinct values rises above the max, the metric will pause, and a tooltip appears indicating that the maximum has been exceeded.

Scan a datasource for changes

Lightup regularly scans datasources for changes, but you can also initiate a scan manually.

  1. With the datasource selected in Explorer, select Actions > Manage Schemas.
  2. Near the top-right of the modal, select Run Manual Scan.
  3. The scan is queued to run as soon as possible. To stop a running scan, select Cancel Scan.

Enable data catalog integration

You can enable integration with an Alation or Atlan data catalog for datasources that are in both Lightup and the data catalog. Note that an App Admin must first add the data catalog integration to your Lightup instance.

  1. On the workspace menu, select Datasources.

  2. In the main page, select the datasource's name to open its Datasource menu.

  3. Select Edit. The datasource's configuration details open in the main page.

  4. Scroll down below the Connection Settings. If the datasource is in your data catalog, an Alation integration and/or Atlan integration section appears:

    • Move either toggle to the right to enable the data catalog integration for the datasource.
    • Under Choose a datasource, select the source you want to include in regular Health updates to the data catalog.
  5. Select Save to complete the process.