Review DQ in Atlan

Atlan is a data catalog that leverages asset data and metadata to help teams discover, trust, and understand data they care about. Data assets in your Atlan catalog support a range of basic metadata, and Atlan supports extending the knowledge available about those assets with custom metadata.

You can add your Atlan data catalog to Lightup, expanding the health metadata that your teams can use to evaluate trustworthiness of data (including its lineage), and even generating announcements for your Atlan catalog assets when Lightup detects incidents.

Before you begin, consider creating an Atlan persona to grant Lightup's API token access to metadata and data for the connections you want to integrate. You can also create the persona when you create the API token, but if you create it ahead of time you can just add it to the token instead.

For example, the policies in this persona enable Lightup to send health information about three datasources:

Note that before Atlan can be enabled for a datasource, you must first grant the API token the necessary metadata and data access to the Atlan connection that corresponds to the Lightup datasource, using policies in a persona. This is required for each datasource.

Using Lightup with Atlan

After Atlan is enabled for a datasource, Atlan and Lightup perform regular scans of each other to keep your catalog information in sync. Any incident that occurs in Lightup on a data asset in the datasource generates (or updates) an Atlan announcement.

  • Your Atlan home page displays incident announcements for assets in the Recent Announcements feed.
  • Lightup provides data quality information to Atlan so you can review an asset's health in the Atlan catalog. You can also use Lightup filters on your Atlan data catalog, so you can focus on assets that have Lightup data quality information.
  • Lightup also adds data quality information to your Atlan Lineage diagram.

Review Recent Announcements to keep aware of data quality issues

Each asset in Atlan can have an announcement associated with it. When Lightup detects an incident, it generates an announcement for the related Atlan asset. These announcements appear on the right side of your Atlan home page for any assets you can see in Atlan.

Check data health of assets in your Atlan catalog

The Assets tab of your Atlan catalog helps you find and evaluate suitable data assets for specific information needs. Lightup provides filters to help you find assets in Atlan that have Lightup data health information, and keeps that information updated for those assets to help you evaluate their health.

Filter using Lightup information

When you select the Assets tab, by default Atlan lists all assets you can see. You can use one of the Lightup filters to narrow down the list so you only see assets that have Lightup data quality information.

For example, here's a list of assets that have at least one monitor on the corresponding table in Lightup, filtered using Table monitored:

  1. The Lightup filter, Table monitored, is set to Yes, limiting the list of assets.
  2. On each asset, data health is indicated by a flag showing the asset's announcement, which reflects whether an incident occurred and (if so) its status.
  3. Lightup custom metadata appears on the asset sidebar.

View Lightup information in an asset's Lineage

When you select the Lineage tab of an asset profile in Atlan, any Lightup health information appears as flags in the lineage of the asset, so you can see the source and impact of detected incidents. Details about the health of the asset appear in the asset sidebar.