If you are interested in using this feature, please contact Lightup Support

Apache Kafka is a distributed streaming platform that uses a broker cluster to manage topics. Each topic is a category or feed name to which messages are published. Messages in Kafka consist of a value (the actual data) and can include optional fields such as a key, headers, and partition information. This structure allows for efficient organization, storage, and retrieval of data in a scalable and fault-tolerant manner.

ksqlDB is a specialized database for stream processing applications built on top of Apache Kafka.. It provides a SQL-like interface for interacting with streaming data, allowing users to treat Kafka topics as traditional database tables.

Topics in Kafka correspond to tables in Lightup. These tables appear under the "default" schema in explorer.

Kafka cluster → Datasource in Lightup
Topic → Table in Lightup
Value schema → Table schema

Since Kafka doesn't have schemas, a schema named default is created in Lightup

Connect to an ksqlDB datasource

Connection parameters

In the left pane select Datasources*
In the main page select Create Datasource +.
Under Select a connector, enter a Datasource Name and select the Apache Kafka Connector type.
Under Configure connector, provide values for the following:

KSQLDB host - The hostname for the database (check the browser address bar; use the string between // and /). Example: https://host.na.me:443/You can also use the IP address and port number of the host (a.b.c.d:8088)
KSQLDB username - Enter username (whatever username you gave the Lightup user).
KSQLDB password - Enter the password for the Lightup user.
Kafka schema registry URL - The hostname for the database (check the browser address bar; use the string between // and /). Example: https://host.na.me:443/.You can also use the IP address and port number of the host (a.b.c.d:8081)
Kafka schema registry username - Enter the kafka schema registry username
Kafka schema registry password - Enter the schema registry password

After entering the required settings and any optional settings that apply, below the Configure connector section select Test Connection.
After a successful connection test, select Save.
Your new datasource appears in the list of available datasources. By default, these are listed in alphabetical order, so you might have to scroll or change the sort order to see your new datasource.

Advanced/Schema scan frequency

You can adjust how often scans run for a datasource.

In section 3 - Advanced, select a value for Schema scan frequency: Hourly, Daily, or Weekly

Supported metric types

Schema or Table level auto metrics

Table activity (Schema) - Topic addition/ deletion
Data delay for a topic (Table)
Data volume for a topic (Table)

Column level auto metrics

Null percent
Category activity

Caveats

Data profile is not supported on ksqlDB