Request data exports

The process described on this page requires a Lightup deployment that uses S3 or Azure Blob storage, as detailed below. 
Upon request, Lightup will begin regular exports of your data quality metrics to your cloud-based storage (currently, S3 or Azure Blob). You can then use the data exports for custom analysis, custom dashboard generation, and archiving purposes. If you use S3 for cloud storage, you can create a Redshift database and import the data there.  
What data we export
When exporting is turned on, each day we export a a summary of all metric data points collected that day along with indicators for whether or not an incident was detected for each data point. The export goes in a new CSV file in your cloud storage (S3 bucket or Azure Blob Container), using the following columns with one row per incident logged. You can then use these exports as needed. 
Items listed below have the format columnName - value source: value data type.
eventTs - datapoint timestamp: seconds from Unix Epoch
slice - datapoint slice: JSON object
value - datapoint value (what it represents depends on the type of metric): floating point number
metricUuid - datapoint metric uuid in Lightup db: string
metricName - datapoint metric name: string
metricType - datapoint metric type (whatever we have in config.aggregation.type): string
metricAggregationWindow - datapoint metric aggregation window (hourly, daily, etc.): string
metricTags - datapoint metric tags: Array[string]
sourceUuid - datapoint metric source uuid in Lightup db: string
sourceName - datapoint metric source name: string
schemaName - datapoint metric schema name: string
tableUuid - datapoint metric table uuid in Lightup db: string
tableName - datapoint metric table name in Lightup db: string
columnUuid - datapoint metric column uuid in Lightup db: string
columnName - datapoint metric column name: string
incidentCount - number of active incidents associated with the datapoint metric at eventTs time: integer
workspaceId - workspace id of the metric: uuid
monitorUuid - uuid of the monitor if datapoint was monitored: string
monitorName - name of the monitor if datapoint was monitored: string
monitorTags - monitor tags if datapoint was monitored: string
monitoredValue - processed value that is compared with monitor bounds: floating point number
monitorLowerBound - lower bound of the monitor if datapoint was monitored: floating point number
monitorUpperBound - upper bound of the monitor if datapoint was monitored: floating point number
dataExtractionComments - Description of any errors that were encountered during data extraction: string
Start getting data exports
​Email [email protected] and provide the following information, based on your cloud storage account. There are currently two options: S3 and Azure Blob.
​Start exports to S3  
​Start exports to Azure Blob​
S3 exports - parameters
Include these details in your message:
S3 Bucket name where you want the export files (Required)
AWS access key ID (Optional) 
AWS secret access key (Optional)
Region name (Optional)
AWS Blob exports - parameters
Include these details in your message:
Azure Blob Container name where you want the export files (Required)
Account name (Required)
Account key (Required) 
Import S3 CSV files into a Redshift database
If you use S3 cloud storage for Lightup you can create a Redshift database and import the data from your CSV files, unlocking numerous analytical options. 
The schema of the exports is subject to planned change. If you create a database per the following steps, include this factor in your maintenance plans.
1.Create a Redshift database. For syntax, see Amazon Redshift - CREATE DATABASE.
2.Open Redshift query editor in AWS console.
3.Make sure Redshift IAM role is allowed to read from the source S3 bucket. For help using IAM roles, see AWS Identity and Access Management - Using IAM Roles.
4.Create a table using following SQL:
CREATE TABLE datapointsexport (
eventTs timestamp,
slice super,
value float,
metricUuid varchar,
metricName varchar,
metricType varchar,
metricAggregationWindow varchar,
metricTags super,
sourceUuid varchar,
sourceName varchar,
schemaName varchar,
tableUuid varchar,
tableName varchar,
columnUuid varchar,
columnName varchar,
incidentCount int,
workspaceId varchar,
monitorUuid varchar,
monitorName varchar,
monitorTags varchar,
monitoredValue float,
monitorLowerBound float,
monitorUpperBound float,
incidentData varchar,
userDescription varchar,
dataExtractionComments varchar
);
​
5.To import a CSV file, run the following SQL, but replace "datapointsexport" with the name of the CSV file (excluding the .csv extension), "lightup-datapoints-dump" with your S3 bucket name, and the value of iam_role:

    COPY datapointsexport
    FROM 's3://lightup-datapoints-dump'
    iam_role 'arn:aws:iam::231612517276:role/RedshiftCopy'
    csv
    IGNOREHEADER 1
    DELIMITER ','
    EMPTYASNULL
    TIMEFORMAT 'epochsecs';
    
For information about automating imports from S3 files into Redshift, see A Zero-Administration Amazon Redshift Database Loader.  

Review data quality summaries

Add an integration

Last modified 1mo ago

Copy link

Outline

What data we export

Start getting data exports

Import S3 CSV files into a Redshift database