Kafka stream processing via Lenses SQL, scale with Kafka

As mentioned in previous post,
Lenses SQL leverages Kafka Streams to process data and currently provides 3 execution modes
to run Lenses SQL processors. In this video we demonstrate how to scale out using CONNECT mode
as well as how to manage the Lenses SQL processors via the Lenses web interface or the CLI tool.

About Lenses SQL Processor modes

IN_PROC

is the default execution mode and the processors are executed locally within Lenses.
This can have scalability issues and poses a risk to the running application, and can affect stability.
IN_PROC is recommended only for testing.

CONNECT

is the execution mode that solves these limitations and provides availability guarantees and scalability.
Lenses can deploy your Lenses SQL processors in Kafka Connect. Kafka Connect provides a distributed,
fault tolerant and scalable framework as part of the core Apache Kafka distribution.

KUBERNETES

, which we’ll see in an upcoming blog post,
is an execution mode that provides scalability by deploying Lenses SQL runners into Kubernetes clusters.
Lenses can deploy and monitor SQL runner deployments created through Lenses or existing tools such as Helm or kubectl.