Metrics overview

Metrics are how you measure the quality of your data. Some metrics require no configuration— you can use them out of the box to gain fundamental insights about your data health. For the team that wants more customized data insights, Lightup supports a full spectrum of powerful metrics, from low-code template-driven metrics that answer common questions, to custom SQL metrics that answer questions specific to your business requirements.

Metric types

Lightup offers a full range of metrics, of four main types:

  • Column metrics answer questions about specific columns, and have the widest variety of configuration options.
  • Comparison metrics let you compare two tables (directly or through comparable metrics), helping you ensure ETL pipeline data quality.
  • Table metrics answer basic questions about your table's health, including low-cost metadata checks.
  • SQL metrics let you answer arbitrary questions about your data by writing SQL.

See Create and edit metrics for details on how to create each type.

Column metrics

Aggregation

Calculate an aggregate function on a column's values for a set of rows

MetricDescriptionColumn
AvgReturns the sum of values divided by number of rows.Numeric
CountReturns the count of values.Any (not recommended for ID columns)
Count UniqueReturns the count of distinct values.Any (not recommended for ID columns)
MaxReturns the largest value.Numeric
MedianReturns a number for which half of the values are smaller and half are larger.Numeric
MinReturns the smallest value.Numeric
PercentileReturns the value at or below which the specified fraction of values in its frequency distribution falls.Numeric
St.DevReturns the standard deviation.Numeric
SumReturns the sum of values.Numeric

Conformity check

Calculate the percent of rows that conform to a condition

MetricDescriptionColumn
Contains/Does not contain stringCompares selected column's values with input string.String
Equal/Not equalCompares selected column's values with input.Any
Greater thanCompares selected column's values with input number and matches if the column's value is greater.Numeric
Greater than or equal toCompares selected column's values with input number and matches if the column's value is greater.Numeric
In/Not inCompared selected column's values with an input set of values.Any
In column/Not in columnCompares selected column's values with a second column's values.Any
Is decreasingChecks that column values are always decreasing. Returns 100% when current value is less than the previous value, and 0% when it is greater than or equal to the previous value.Numeric
Is increasingChecks that column values are always increasing. Returns 100% when current value is greater than the previous value, and 0% when it is less than or equal to the previous value.Numeric
Is uniqueChecks for unique values.Any
Length equal toCompares selected column's values with input number and matches if the length is equal.String
Length greater thanCompares selected column's values with input number and matches if the length of the column value is greater.String
Length greater than or equal toCompares selected column's values with input number and matches if the length of the column value is equal or greater.String
Length less thanCompares selected column's values with input number and matches if the length of the column value is smaller.String
Length less than or equal toCompares selected column's values with input number and matches if the length of the column value is equal or smaller.String
Less than or equal toCompares selected column's values with input number and matches if the column's value is greater.Numeric
Match pattern/Does not match patternCompares selected column's values with input pattern.String
Match regex/Does not match regexCompares selected column's values with input regular expression.Any
Null/Not nullChecks selected column's values for nulls.Any

Distribution metrics

Calculate the distribution of a column's values

MetricDescription
DistributionMeasures the distribution of distinct values.

Null Percent metrics

MetricDescription
Null PercentMeasures the percentage of null values.

Comparison metrics

Calculate how much two similar objects match

MetricDescription
Compare aggregateMeasures the degree to which two aggregate metrics match.
Row by RowMeasures the percentage of rows in two tables where the key columns match, and the values of attribute columns in those matching rows.

Table metrics

Measure characteristics of a table or its behavior

MetricDescriptionSupported datasources
Data delayMeasures the time elapsed since the most recent data arrived.All
Data volumeMeasures the number of rows loaded.All
Column activityCounts changes to columns (columns added, altered or dropped).All
Update DelayMeasures the time since the last data write.BigQuery, Databricks, Incorta, MS-SQL, Redshift, Snowflake
Byte CountCounts the total number of bytes.Databricks
Row CountCounts the total number of rows.BigQuery, Incorta, MS-SQL, Oracle, Postgres, Redshift, Snowflake

Note that Column activity, Update Delay, Byte Count and Row Count are metadata metrics.

SQL metrics

SQL metrics support any valid SQL.