Learn streaming data with Lenses

Here is a glossary with all the basics in Kafka and data streaming engineering. Want to learn more? Join our Community Slack channel.

BFGIJNQUVWXYZ

A

  • Agent (Lenses Agent)

    A headless application deployed within a Kafka environment that connects back to Lenses HQ. It acts as a proxy for operations, monitoring, and data access.

  • Agentic Engineering

    A paradigm shift in software development where AI agents are orchestrated with engineering rigor to autonomously build, operate, and govern data systems under human oversight.

  • AI Agent

    An autonomous assistant, such as an SRE Agent, that uses tools and context (often via MCP) to perform complex tasks like troubleshooting or pipeline creation.

  • Apache Kafka

    A distributed event store and stream-processing platform used as a real-time data "central nervous system" for modern enterprises.

C

  • CIMD (Client ID Metadata Document)

    A web-based registration model for MCP clients where the identity is a URL hosting a JSON document, allowing for verified, domain-linked security.

  • Consumer Group

    A coordinated group of consumers that ingest data from topics, sharing the workload and tracking progress via offsets.

D

  • Data Lineage

    A visual representation of the data lifecycle, showing how events flow from producers through topics and processors to final sinks.

  • Data Masking

    A governance policy that obscures sensitive PII fields (like emails or phone numbers) in real-time before agents or unauthorized users see them.

  • DCR (Dynamic Client Registration)

    An OAuth flow that allows AI clients to programmatically register themselves with an authorization server at runtime.

E

  • Environment

    A logical grouping in Lenses representing an individual Kafka cluster and its supporting services (e.g., Schema Registry, Connect).

  • Exactly-Once Semantics (EOS)

    A guarantee that data is processed exactly once, even if failures occur, preventing duplicates and data loss.

H

  • HQ (Control Plane)

    The central Lenses portal where users and agents interact with all Kafka environments and manage governance globally.

K

  • K2K Replicator

    A Kubernetes-native, vendor-neutral tool for moving data, schemas, and offsets between any flavor of Kafka.

  • Kafka Cluster

    A set of one or more Kafka brokers working together to provide scalability and fault tolerance.

  • Kafka Connections

    Scoped access paths in Lenses 6.1+ that allow developers to deploy applications without handling or seeing cluster credentials directly.

  • Kafka Connector

    A pre-built component used to integrate Kafka with external systems (e.g., S3, Snowflake) to move data in (Source) or out (Sink).

  • Kafka Consumer

    An application or agent that reads and processes data from Kafka topics.

  • Kafka Partition

    A sub-division of a topic that enables Kafka to scale by distributing data across multiple brokers.

L

  • Lag

    The gap between the latest message produced to a topic and the last message a consumer has successfully read.

M

  • MCP (Model Context Protocol)

    An open-source standard that provides a secure, standardized interface between AI models and enterprise systems. Also known as the "USB-C for AI".

O

  • OAuth 2.1

    The modern authorization standard for MCP, mandating short-lived, scoped, and revocable tokens for AI agent access.

  • Offset

    A unique identifier for a record within a partition that acts as a "bookmark" for consumer progress.

P

  • Poison Pill

    A malformed record in a topic that causes processing applications to crash or fail repeatedly.

  • Producer

    An application or agent that sends (writes) events to Kafka topics.

R

  • Replication

    The process of copying data across multiple brokers to ensure high availability and durability.

S

  • Schema

    The "contract" defining the structure of a data stream (e.g., Avro, Protobuf). Lenses provides a Schema Registry to manage these globally.

  • Skills

    Structured context files (Markdown) that provide AI agents with domain-specific knowledge and remediation playbooks.

  • SQL Processor

    A streaming application in Lenses that uses SQL to continuously filter, transform, or join data streams in real-time.

  • Studio

    The centralized Lenses workspace (formerly SQL Studio) for exploring data and managing all Kafka resources like connectors and consumer groups.

T

  • Topic

    A logical category or "folder" in Kafka where events are stored and categorized.