Catalogs for Apache Kafka – when a topic name is not enough

In any enterprise organization, whether it is from finance & insurance to energy & utilities, platform teams provide the building blocks used to build business use cases. These building blocks typically consist of but are not limited to relational or SQL databases, time series databases, message queues, monitoring systems, and deployment infrastructure. This is no different for organizations that are starting to adopt Apache Kafka for their streaming data use cases.

In the beginning, life is simple. A topic here, a topic there. With just a handful of topics existing on your cluster, the name of the topic itself tells you enough. You know who produces the data on a topic, you even know the person who worked on the app. You might even know the name of her pet. But, as word of mouth spreads, your Kafka platform grows in popularity. More and more developers find their way to your cluster and want you to support them with their use cases.

As mentioned, just by naming things, you can tell them apart quite well. Just as you keep files in the Documents folder of your laptop: by looking at the name or searching for a string, you can find the document you need.

But with 20, 30 or more topics, it is very easy to get lost. One way to make your life as a platform team and the life of your developers easier is by standardizing on the topic name. For instance, you can start using the department name in your topic, e.g. payments-transactionevent or insurance-policyupdate. Phew, you are good to go for a little while…  

When your cluster holds 50 or more topics, just standardizing on the topic name itself is not enough. It is unavoidable that you will get questions like

“Who can I contact to get more information about this topic”?

“How can I be sure I need this topic for my use case?”

“Which message encoding should I use for this topic?”

“What is the retention time used on this topic?

In short: you need to start administering topic metadata. Things like ownership, message type and a simple description of the data on the topic would help anyone in the organization immensely to get going quickly.

Sadly, open source distributions of Apache Kafka don’t offer you the possibility to add metadata to your topics, so you are kind of stuck. You need to build or start using a governance and management tool for Kafka, where you can not only store this information, but also make it available for and searchable by the people in your organization.

You need a Catalog for Kafka.

Now that you have enriched your Kafka catalog, you might as well use it to enable self-service on your clusters, so that topic creation and configuration can be done as well.

Having a Catalog for Kafka not only makes developers find their way in the forest of topics. It will help them to significantly reduce the time spent from idea to use case in production. All the time they can spend on their business use case is time well spent. Time spent on other things is waste, at best.

Secondly, waste of time and effort can better be eliminated because duplicate topics, even though the name is different, are less likely to exist on your Kafka cluster. Metadata to the rescue!

Lastly, a catalog can help developers and architects discover the streaming landscape in your organization. This helps to better understand information flows and bottlenecks, and can be used for dreaming up new exciting Kafka based use cases.

Start managing your topics with Axual

With Axual, organizations can scale their development teams around a central Kafka. Our platform comes with built-in self-service, data governance, and security functionalities to unlock the full potential of Kafka for your development teams. The graphical user interface makes it easy for teams to control their clusters, topics, applications, and schemas from one, central overview.

Start a free trial right here or get in touch to discuss your specific use case with our Kafka architects.

‍