How to create a Kafka topic (the safe way)

We live in a dynamic world. It is safe to say that companies aim to speed up time-to-market and out-innovate their competition with Kafka, but at the same time struggle with some limitations. These can range from compliance-related setbacks for regulations such as GDPR, CCPA and HIPAA, to self-service slip-ups that could see a whole Kafka cluster going down.

Even something as seemingly innocuous as configuring and creating a Kafka Topic can lead to operational U-turns, slowdowns and even downtime:

• Once a topic is created, changing settings and adding partitions is tough

• Lack of oversight on data retention times can see you losing messages, crashing applications or filling your disk by adding too many partitions

• Setting a replication factor too high can put a strain on a service; too low can see you falling short of an SLA

🙀

No one wants to get caught up in a Kafka Topic ticket storm.

This is why governance has become something of a daily consideration for data and even software engineers.

Luckily here at Lenses.io, we provide guardrails that allow developers to be more safe, self-sufficient and speedy with Topics! This blogpost goes over how compliance can be achieved easily with regards to Kafka Topic creation.

Let’s talk about Kafka Topic creation

Kafka Topic creation is one such activity where there is no rule of thumb, and hence plenty of opportunity to mishandle data. How you parameterize Kafka Topics may depend on:

• Cluster size

• Amount of storage

• Business context (you want Topics with names that mean something)

• The profile of your data and how you wish to key your messages

To meet these needs, Lenses is introducing more granular capabilities on how you can create and configure Topics by bringing you creation policies.

Users can specify defaults and constraints across Topic parameters (partitions, replication, retention, and naming). What could previously only be implemented very loosely (by using conventions, for example) can now be implemented strictly, through a dedicated UI and more. Let us now guide you through all the details you will need to properly configure your topics.

Where can you find Kafka Topic creation policies?

If you’re not using Lenses yet, just head over to the Lenses portal to spin up a free trial demo environment. Lenses users can access the admin Topic settings screen easily, by following a link in the admin navigation.

Once there, you are greeted by a read-only representation of whatever the Topic settings are for the time being. From there, simply click the "Edit" button, which offers an editable view.

Once you make it there, there are a couple of things to keep in mind:

• Although there are no permissions required for viewing Topic settings, you will need to manage permissions for Kafka settings to be able to edit them.

• Before any changes are made (or when optional values are not provided), Topic creation will default to whatever your broker’s default settings are.

Where do Topic settings apply?

Topic settings apply basically everywhere Topics are created/configured. That includes:

• Topic creation (when creating Topics via the Lenses-provided form)

• Topic configuration (some Topic configuration options, e.g. retention bytes or retention.ms are actually affected by Topic settings)

• SQL studio (as Topics can be created there too)

• Lenses CLI (additionally, Topic settings can be imported from / exported to a file; supported formats include JSON and YAML)

• Lenses API (rules are applied the same way as they would through the UI)

What can you configure when it comes to Kafka Topics?

Partitions, replication

For both partitions and replicas, you can configure minimum and maximum values. Other than the obvious constraints (e.g. min <= max), some infrastructure-related restrictions apply (see “Caveats”).

Retention time & size

For retention time and size, you can set both default and maximum values. Defaults will apply on Topic creation, where the appropriate fields get populated automatically (you can, of course, overwrite them). Both default and maximum values can be set to infinite (-1), yet the default <= max constraint still applies - setting default to infinite will effectively do the same for max

Naming strategy

Here, you can be creative. Several presets are provided, including:

• Only letters

• Letters and dots

• Letters and dashes

• Letters and numbers

• Letters, numbers, and dots

• Letters, numbers, and dashes

If the presets do not meet your needs out-of-the-box, you can provide a custom naming strategy via a regular expression along with a short hint to make the intent clearer. One thing to note is that the Topic name must match the provided regular expression entirely to be considered valid.

Caveats

Even though straightforward on the surface, there is more to Topic settings than meets the eye. To make full use of this feature, there are a couple of things you need to be aware of:

• Topic settings are top-level constraints.

This means you need to be granted permission to create/configure a Topic under a certain namespace first, then need to adhere to the Topic settings.

• There are scenarios where Topic settings are not applicable.

One such scenario is when setting the minimum number of replicas to N while the number of active brokers in the Kafka cluster for some reason drops to N-1. Lenses will handle this by locking all the API calls that could be affected by the redundant policy until this inconsistency is addressed.

• The constraints yielded by Topic settings are specific to Lenses and will only apply there.

None of them will take effect when creating/configuring Topics through other tooling.

Next steps

We are pretty excited by this new feature, and we’re sure you’re excited too. Now you’re empowered by Topic creation policies, go ahead and configure yours! Here are a couple of ways to get you up to speed:

• View Topic settings docs

• Set up a free Lenses demo environment

• Try out Topic settings in a local Kafka docker box