Activate data assets

Before you can collect metrics on a data asset, it must be Active. You can only activate a data asset if its parent is already Active.

1. To make a schema active, select its datasource in the Explorer tree, then choose Manage Metrics on the Actions menu on the top right panel.

2. In the Manage Metrics modal, in the schema's row move the Active toggle to the right to activate that schema.
   • It takes a few minutes for the schema to activate. During this time, the schema's data will be unavailable.
   • Leave the Manage Metrics modal open until the schema name changes to a link.
   • To activate all the tables, select the box at the top left corner of the list on the Manage Metrics modal before you move the Active toggle.
   • Once a schema is activated, you can enable its Table Activity metric by moving the Activity toggle to the right. When you return to Explorer, the Table Activity autometric will be enabled, which displays changes to tables in the schema.
3. In the Manage Metrics modal, select the link to the schema. The list of data assets in the modal refreshes, then displays tables for the selected schema.
   • To activate a table, move its Active toggle to the right. The Manage Table Settings modal immediately opens so you can provide values for inheritable settings.
   • When you close the Manage Table Settings modal, the Manage Metrics modal remains open. The table is now Active and its name is a link.
   • To activate all tables, select the checkbox at the top left corner of the list on the Manage Metrics modal before you move the Active toggle.
   • Once a table is activated, you can enable its Column Activity (shown as Activity), Data Delay, and Data Volume metrics by moving the appropriate toggles. When you return to Explorer the autometrics that you enabled will display in the table's Explorer pane.
4. In the Manage Metrics modal, select the link to a table. The list of data assets in the modal refreshes, then displays columns in the selected table.
   • To activate a column, move its Active toggle to the right.
   • Once a column is activated, you can enable its Category Activity (shown as Activity), Distribution, and Null Percent metrics by moving the appropriate toggles. After enabling autometrics, when you return to Explorer, the autometrics that you enable will display in the Explorer pane for the column
   • Note that Category Activity is only available for attribute columns. and that Distribution will enable a categorical distribution for attribute columns and a numerical distribution for numerical distribution.
   • You can change a columns type from numeric to attribute by clicking on the # or A in the Type column and selecting the alternate type.

🚧

Timestamp columns

When you activate columns in a table, the number of columns you see may be one less than the count of columns displayed in the Explorer summary for the table. If so, the table has a timestamp column, which is used in many metric calculations and cannot itself be the target data asset for metrics.

📘

Enable autometrics

You enable autometrics using the same modal that you use for activating data assets. To enable one, activate the data asset, and then when the autometric's toggle appears, move it to the right.

Manage Table Settings

When you activate a table, its Manage Table Settings modal opens so you can configure the table's inheritable settings, which are inherited by metrics based on the table (but you can override them when you create or edit a metric).

🚧

More interval options in metrics

If you override table settings when you create or edit a metric, several longer periods are available options for Aggregation Interval/Polling Interval: Monthly, Quarterly, and Yearly. The options are not available as table settings— you can only choose them in a metric when you override inherited table settings.

Data Collection Schedule

Traditionally, Lightup metrics run on regular schedules. But now there's a second way: triggered metric data collection. Triggered collection lets you specify that metrics based on a given table won't run on regular schedules, and instead will run only after being triggered, by setting the table's Data Collection Schedule setting to Triggered (the default setting is Scheduled).

Because Data Collection Schedule is an inheritable setting, you can change this setting for a metric to make it behave the opposite of its table (so the table could have triggered data collection while the metric runs on a regular schedule, or vice-versa).

After you specify that data collection is Triggered, you use an API call to trigger collection for the table's metrics. You use the same API to trigger data collection for specific metrics.

🚧

• One data collection is triggered per API call (triggering whatever UUIDs you included, once).
• Data collection is only triggered for live metrics. Paused metrics ignore triggers.
• When you trigger collection, the triggered metrics run at the start of the next collection cycle.
• If you're viewing a metric chart when the metric is triggered, you'll need to refresh the page after collection occurs to see the new value.

❗️

API use requires authentication

To use most of the API calls in our API Reference, including Trigger metric collection, you need to get an auth token from Lightup. To prepare, follow the steps on Get access token.

Query Scope

One inheritable setting can't be changed when a table is Active: Query Scope, which controls how many rows of data get collected during metric collection (i.e., how much of the table is read). There are two options, Incremental and Full Table, and it should be the first inheritable setting that you specify: when you choose an option, some of the remaining inheritable settings change accordingly— they're dependent on Query Scope. This feature was designed to support DQI for dimension tables (those with no time-based fields).

❗️

Full Table metrics generate full table queries which may result in compute intensive queries on large tables

Incremental scope settings

1. If needed, set Data Collection Schedule to Triggered. This will cause Lightup to stop scheduled data collection for metrics on the table, instead only collecting data when triggered by an API call (except for any metrics where this setting is overridden). When triggered for a table, data is collected for metrics on the table at the beginning of the next data collection cycle. When triggered for a metric, only that metric is triggered.

2. Specify an Aggregation Interval to set the timeframe over which the metric's value is aggregated. For Scheduled metrics, this (combined with Evaluation Delay) also determines when data collection occurs: daily metric values are calculated over the 24-hour period starting at 12AM; hourly metric values are calculated at the top of each hour; and weekly metrics are calculated starting at 12AM on Monday. (For Triggered metrics collection occurs only after triggering.)

3. Select the Aggregation Time Zone. This specifies the time zone that is used for metric aggregation. For example, if Aggregation Interval is daily, the data will be aggregated for the 24-hour period starting at 12AM in the Aggregation Time Zone.

4. Set Evaluation Delay to a value which represents the time period required for your data to be guaranteed stable. Evaluation Delay is a blackout period during which data is considered not stable or not ready for running data quality checks. For example, if a metric has an Evaluation Delay of two hours, data with a timestamp in the past two hours is ignored. So, if the Aggregation Interval is daily and Evaluation Delay is 2 hours, each metric value aggregates 24 hours worth of data, starting at 2AM in the Aggregation Time Zone.

5. Choose the timestamp column for your table. The aggregation period for a metric is based on the value in the timestamp column, translated into the time zone specified by the Aggregation Time Zone. As described above, only data with timestamps prior to [now] - Evaluation Delay are considered.

📘

Virtual timestamps

You can also use a virtual timestamp if no suitable column is ready to use and the datasource supports virtual timestamps.

6. Some timestamp columns don't include time zone information, so you might need to specify the Time Zone where the data was written. If the time zone is part of the timestamp, this setting says Derived and can't be changed.
7. Under Partitions, if your table has time-based partitions, you can specify the column and format of the partition so that you can use it to improve metric's query performance.
   • Format should be specified using Python format codes.
   • If your table doesn't have any partitions, the Partitions section doesn't appear in the Manage Table Settings dialog.
   • Because there might be more than partition column, the drop-down lists all active columns and appends the word partition for each column that has partitions in the source data. Make sure you pick a partition column.
8. Click Confirm and Save to save your settings. Now that you've activated the table, you can create metrics for it.

Full Table scope settings

1. If needed, set Data Collection Schedule to Triggered. This will cause Lightup to stop scheduled data collection for metrics on the table (except metrics where this setting is overridden/set to Scheduled), instead only collecting data after being triggered by an API call. When triggered, data is collected for all metrics on the table at the beginning of the next data collection cycle.
2. Specify a Polling Interval to control how often the metric query is scheduled to run.
3. Set Polling Time Zone to the time zone where the metric query runs.
4. Use Polling Delay to introduce a delay at the beginning of the metric query. This delay may be exceeded depending on system load, but will always at least be met.
5. Click Confirm and Save to save your settings. Now that you've activated the table, you can create metrics for it.