Prepare data assets

Activate data assets and manage inheritable settings

Activate data assets

Before you can collect metrics on a data asset, it must be Active. You can only activate a data asset if its parent is already Active.

  1. To make a schema active, select its datasource in the Explorer tree, then choose Manage Metrics on the Actions menu on the top right panel.

A data asset in the Explorer tree, with its Actions menu open and the Manage Metrics command selected.A data asset in the Explorer tree, with its Actions menu open and the Manage Metrics command selected.

  1. In the Manage Metrics modal, in the schema's row move the Active toggle to the right to activate that schema.
    • It takes a few minutes for the schema to activate. During this time, the schema's data will be unavailable.
    • Leave the Manage Metrics modal open until the schema name changes to a link.
    • To activate all the tables, select the box at the top left corner of the list on the Manage Metrics modal before you move the Active toggle.
  2. In the Manage Metrics modal, select the link to the schema. The list of data assets in the modal refreshes, then displays tables for the selected schema.
    • To activate a table, move its Active toggle to the right. The Manage Table Settings modal immediately opens so you can provide values for inheritable settings.
    • When you close the Manage Table Settings modal, the Manage Metrics modal remains open. The table is now Active and its name is a link.
    • To activate all the columns, select the checkbox at the top left corner of the list on the Manage Metrics modal before you move the Active toggle.
  3. In the Manage Metrics modal, select the link to the table. The list of data assets in the modal refreshes, then displays columns in the selected table. To activate a column, move its Active toggle to the right.

The Manage metrics modal, where you can activate data assets and enable autometrics.The Manage metrics modal, where you can activate data assets and enable autometrics.

📘

Enable autometrics

You enable autometrics using the same modal that you use for activating data assets. To enable one, activate the data asset, and then when the autometric's toggle appears, move it to the right.

Manage Table Settings

When you activate a table, its Manage Table Settings modal opens so you can configure the table's inheritable settings, which are inherited by metrics based on the table (but you can override them when you create or edit a metric).

20802080

Data Collection Schedule setting

Traditionally, Lightup metrics run on regular schedules. But now there's a second way: triggered metric data collection. Triggered collection lets you specify that metrics based on a given table won't run on regular schedules, and instead will run only after being triggered, by setting the table's Data Collection Schedule setting to Triggered (the default setting is Scheduled).

Because Data Collection Schedule is an inheritable setting, you can change this setting for a metric to make it behave the opposite of its table (so the table could have triggered data collection while the metric runs on a regular schedule, or vice-versa).

After you specify that data collection is Triggered, you use an API call to trigger collection for the table's metrics. You use the same API to trigger data collection for specific metrics.

🚧

  • One data collection is triggered per API call (triggering whatever UUIDs you included, once).
  • When you trigger collection, the triggered metrics run at the start of the next collection cycle.
  • If you're viewing a metric chart when the metric is triggered, you'll need to refresh the page after collection occurs to see the new value.

❗️

API use requires authentication

To use most of the API calls in our API Reference, including Trigger metric collection, you need to get an auth token from Lightup. To prepare, follow the steps on Get access token.

Query Scope

One inheritable setting can't be changed when a table is Active: Query Scope, which controls how many rows of data get collected during metric collection (i.e., how much of the table is read). There are two options, Incremental and Full Table, and it should be the first inheritable setting that you specify: when you choose an option, some of the remaining inheritable settings change accordingly— they're dependent on Query Scope. This beta feature was designed to support DQI for dimension tables (those with no time-based fields).

Query ScopeWhat's collectedDependent settingsRecommended use case
IncrementalOnly those rows with a timestamp inside the most recent aggregation interval.Agg. Interval, Agg. Time Zone, Eval. Delay, Timestamp, Time ZoneFact tables (tables with a timestamp column)
Full TableAll rows, every time the metric query runs.Polling Interval, Polling Time Zone, Polling DelayDimension tables (tables with no timestamp column)

❗️

  • Full Table query scope is a beta feature.
  • Incremental query scope is the default and is fully supported.
  • Metrics where inherited settings are overridden to create a Full Table scope metric on a table with Incremental scope can consume a lot of compute resources, and may fail if too much data is returned by the metric query.

Incremental scope settings

  1. If needed, set Data Collection Schedule to Triggered. This will cause Lightup to stop scheduled data collection for metrics on the table, instead only collecting data when triggered by an API call (except for any metrics where this setting is overridden). When triggered for a table, data is collected for metrics on the table at the beginning of the next data collection cycle. When triggered for a metric, only that metric is triggered.
  2. Specify an Aggregation Interval to set the timeframe over which the metric's value is aggregated. For Scheduled metrics, this (combined with Evaluation Delay) also determines when data collection occurs: daily metric values are calculated over the 24-hour period starting at 12AM; hourly metric values are calculated at the top of each hour; and weekly metrics are calculated starting at 12AM on Monday. (For Triggered metrics collection occurs only after triggering.)
  3. Select the Aggregation Time Zone. This specifies the time zone that is used for metric aggregation. For example, if Aggregation Interval is daily, the data will be aggregated for the 24-hour period starting at 12AM in the Aggregation Time Zone.
  4. Set Evaluation Delay to a value which represents the time period required for your data to be guaranteed stable. Evaluation Delay is a blackout period during which data is considered not stable or not ready for running data quality checks. For example, if a metric has an Evaluation Delay of two hours, data with a timestamp in the past two hours is ignored. So, if the Aggregation Interval is daily and Evaluation Delay is 2 hours, each metric value aggregates 24 hours worth of data, starting at 2AM in the Aggregation Time Zone.
  5. Choose the timestamp column for your table. The aggregation period for a metric is based on the value in the timestamp column, translated into the time zone specified by the Aggregation Time Zone. As described above, only data with timestamps prior to [now] - Evaluation Delay are considered.

📘

Virtual timestamps

You can also use a virtual timestamp if no suitable column is ready to use and the datasource supports virtual timestamps.

  1. Some timestamp columns don't include time zone information, so you might need to specify the Time Zone where the data was written. If the time zone is part of the timestamp, this setting says Derived and can't be changed.
  2. Under Partitions, if your table has time-based partitions, you can specify the column and format of the partition so that you can use it to improve metric's query performance. Format should be specified using Python format codes. If your table doesn't have any partitions, the Partitions section doesn't appear in the Manage Table Settings dialog.
  3. Click Confirm and Save to save your settings. Now that you've activated the table, you can create metrics for it.

Full Table scope settings (BETA)

20802080
  1. If needed, set Data Collection Schedule to Triggered. This will cause Lightup to stop scheduled data collection for metrics on the table (except metrics where this setting is overridden/set to Scheduled), instead only collecting data after being triggered by an API call. When triggered, data is collected for all metrics on the table at the beginning of the next data collection cycle.
  2. Specify a Polling Interval to control how often the metric query is scheduled to run.
  3. Set Polling Time Zone to the time zone where the metric query runs.
  4. Use Polling Delay to introduce a delay at the beginning of the metric query. This delay may be exceeded depending on system load, but will always at least be met.
  5. Click Confirm and Save to save your settings. Now that you've activated the table, you can create metrics for it.