What is Lightup?

Lightup is a data quality observability and monitoring tool.

What is Lightup's definition of data quality problems?

We consider data quality broken when data in a table or a view or a stream does not match expectation. This could mean issues with data availability, data conformity, or data validity. Read more at https://blog.lightup.ai.

Who is the intended user of Lightup?

Lightup is designed for everyone who works on data quality— from business analysts and data stewards looking for no-code solutions to data engineers writing custom SQL to developers using Python for bulk operations— people who build, operate, and rely on ETL and ELT pipelines. Data quality is a shared responsibility of data producers and data consumers within an organization. Lightup supports them all by facilitating collaborative workflows within the platform.

What data sources does Lightup support?

Lightup can run data quality checks against SQL-speaking data stores including popular data warehouses -- Redshift, Snowflake, Databricks, Spark SQL, BigQuery, Athena, Postgres, MySQL, and more -- using built-in connectors. Lightup can also be integrated with streaming data sources including Kafka, Segment, and Rudder.

How scalable is Lightup?

Lightup leverages the scalability of the data warehouse by issuing aggregation queries to run its data quality checks. Raw data is never pulled out from the data source.

Does Lightup impose heavy query load on the warehouse?

No. Lightup only runs time-window aggregation queries against the data source, such as querying for number of new rows inserted in the last hour or last day. If your data is already indexed on time, Lightup does not cause full table scans.

Does Lightup cause the Snowflake warehouse to stay on all the time?

No. Lightup data quality checks can be configured to operate on a schedule that matches your duty cycles. For example, you could configure a check to run once every hour instead of every second or every minute. That way, the warehouse needs to wake up only once every hour. You could also synchronize Lightup checks with your data production jobs so that data quality checks are only executed when new data is produced and the warehouse is already awake.

Does Lightup store data?

Lightup does not pull out or store raw data out from warehouse. Aggregate metrics pulled out from the warehouse are cached by Lightup for a configurable retention period to make the system efficient and avoid repeated queries to the warehouse. Lightup is not intended to be a system of record for customer's data, raw or aggregate. Your own data warehouse is considered the source of truth.

Can I trust Lightup to access my data?

Yes. Lightup takes security and privacy very seriously. We have invested heavily in our security program and following best practices. We successfully completed the SOC2 Type 2 certification in July 2021. Documentation on our security posture is available upon request.

Can I deploy Lightup in my own environment?

Yes. Lightup supports both SaaS and on-premises deployment models.

The on-premises deployment can be configured to be entirely managed by you, the customer, or managed by Lightup. We have invested heavily in making the on-prem deployment easy enough to get started with that you can try it on your own in a couple of hours.

Can I request support for a new data source?

Absolutely! There is no one-size-fits-all data warehouse, data lake, or data streaming platform. Chances are you have a set of data sources in your stack. Reach out to us to discuss possibilities for adding support for additional data sources.

Can I try Lightup for free?

Yes! We support a free tier for both SaaS and on-premises deployment models. You can try out the entire feature set for free and keep using the system within the free usage capacity as long as you need to.

How do I get started with Lightup?

Find setup instructions specific to the deployment model in the left navigation bar.

How do I get in touch with Lightup?

You can write to us at [email protected] for issues or [email protected] for general information.

Do you have logo files I could download?

Light version:

Dark version: