If you are interested in using this feature, please contact Lightup Support

ksqlDB is a specialized database for stream processing applications built on top of Apache Kafka.. It provides a SQL-like interface for interacting with streaming data, allowing users to treat Kafka topics as traditional database tables.

Topics in Kafka correspond to tables in Lightup. These tables appear under the "default" schema in explorer.

Terminology

Kafka broker cluster contains topics and each topic contains messages which are made up of a value and other optional fields like key, headers, partition etc.

Kafka cluster → Datasource
Topic → Table
Value schema → Table schema

Since Kafka doesn't have schemas, a schema is created in Lightup named default

Connect to an ksqlDB datasource

Connection parameters

1. In the left pane select Datasources*
2. In the main page select Create Datasource +.
3. Under Select a connector, enter a Datasource Name and select the Apache Kafka Connector type.
4. Under Configure connector, provide values for the following:

• KSQLDB host - The hostname for the database (check the browser address bar; use the string between // and /). Example: https://host.na.me:443/You can also use the IP address and port number of the host (a.b.c.d:8088)
• KSQLDB username - Enter username (whatever username you gave the Lightup user).
• KSQLDB password - Enter the password for the Lightup user.
• Kafka schema registry URL - The hostname for the database (check the browser address bar; use the string between // and /). Example: https://host.na.me:443/.You can also use the IP address and port number of the host (a.b.c.d:8081)
• Kafka schema registry username - Enter the kafka schema registry username
• Kafka schema registry password - Enter the schema registry password

5. After entering the required settings and any optional settings that apply, below the Configure connector section select Test Connection.
6. After a successful connection test, select Save.
7. Your new datasource appears in the list of available datasources. By default, these are listed in alphabetical order, so you might have to scroll or change the sort order to see your new datasource.

Advanced/Schema scan frequency

You can adjust how often scans run for a datasource.

• In section 3 - Advanced, select a value for Schema scan frequency: Hourly, Daily, or Weekly

Supported metric types

Schema or Table level auto metrics

1. Table activity (Schema) - Topic addition/ deletion
2. Data delay for a topic (Table)
3. Data volume for a topic (Table)

Column level auto metrics

1. Null percent
2. Category activity

Caveats

• Data profile is not supported on ksqlDB
•