Catalogs for Apache Kafka – when a topic name is not enough
As your organization scales the use of Apache Kafka, so does the amount of topics. Having a catalog for these topics enables you to sort through the chaos.
On this page
In any enterprise organization, whether it is from finance & insurance to energy & utilities, platform teams provide the building blocks used to build business use cases. These building blocks typically consist of but are not limited to relational or SQL databases, time series databases, message queues, monitoring systems, and deployment infrastructure. This is no different for organizations that are starting to adopt Apache Kafka for their streaming data use cases.
In the beginning, life is simple. A topic here, a topic there. With just a handful of topics existing on your cluster, the name of the topic itself tells you enough. You know who produces the data on a topic, you even know the person who worked on the app. You might even know the name of her pet. But, as word of mouth spreads, your Kafka platform grows in popularity. More and more developers find their way to your cluster and want you to support them with their use cases.
As mentioned, just by naming things, you can tell them apart quite well. Just as you keep files in the Documents folder of your laptop: by looking at the name or searching for a string, you can find the document you need.
But with 20, 30 or more topics, it is very easy to get lost. One way to make your life as a platform team and the life of your developers easier is by standardizing on the topic name. For instance, you can start using the department name in your topic, e.g. payments-transactionevent or insurance-policyupdate. Phew, you are good to go for a little while…
When your cluster holds 50 or more topics, just standardizing on the topic name itself is not enough. It is unavoidable that you will get questions like
“Who can I contact to get more information about this topic”?
“How can I be sure I need this topic for my use case?”
“Which message encoding should I use for this topic?”
“What is the retention time used on this topic?
In short: you need to start administering topic metadata. Things like ownership, message type and a simple description of the data on the topic would help anyone in the organization immensely to get going quickly.
Sadly, open source distributions of Apache Kafka don’t offer you the possibility to add metadata to your topics, so you are kind of stuck. You need to build or start using a governance and management tool for Kafka, where you can not only store this information, but also make it available for and searchable by the people in your organization.
You need a Catalog for Kafka.
Now that you have enriched your Kafka catalog, you might as well use it to enable self-service on your clusters, so that topic creation and configuration can be done as well.
Having a Catalog for Kafka not only makes developers find their way in the forest of topics. It will help them to significantly reduce the time spent from idea to use case in production. All the time they can spend on their business use case is time well spent. Time spent on other things is waste, at best.
Secondly, waste of time and effort can better be eliminated because duplicate topics, even though the name is different, are less likely to exist on your Kafka cluster. Metadata to the rescue!
Lastly, a catalog can help developers and architects discover the streaming landscape in your organization. This helps to better understand information flows and bottlenecks, and can be used for dreaming up new exciting Kafka based use cases.
Start managing your topics with Axual
With Axual, organizations can scale their development teams around a central Kafka. Our platform comes with built-in self-service, data governance, and security functionalities to unlock the full potential of Kafka for your development teams. The graphical user interface makes it easy for teams to control their clusters, topics, applications, and schemas from one, central overview.
Start a free trial right here or get in touch to discuss your specific use case with our Kafka architects.
Answers to your questions about Axual’s All-in-one Kafka Platform
Are you curious about our All-in-one Kafka platform? Dive into our FAQs
for all the details you need, and find the answers to your burning questions.
Related blogs
Apache Kafka has become a central component of modern data architectures, enabling real-time data streaming and integration across distributed systems. Within Kafka’s ecosystem, Kafka Connect plays a crucial role as a powerful framework designed for seamlessly moving data between Kafka and external systems. Kafka Connect provides a standardized, scalable approach to data integration, removing the need for complex custom scripts or applications. For architects, product owners, and senior engineers, Kafka Connect is essential to understand because it simplifies data pipelines and supports low-latency, fault-tolerant data flow across platforms. But what exactly is Kafka Connect, and how can it benefit your architecture?
Apache Kafka is a powerful platform for handling real-time data streaming, often used in systems that follow the Publish-Subscribe (Pub-Sub) model. In Pub-Sub, producers send messages (data) that consumers receive, enabling asynchronous communication between services. Kafka’s Pub-Sub model is designed for high throughput, reliability, and scalability, making it a preferred choice for applications needing to process massive volumes of data efficiently. Central to this functionality are topics and partitions—essential elements that organize and distribute messages across Kafka. But what exactly are topics and partitions, and why are they so important?
Strimzi Kafka offers an efficient solution for deploying and managing Apache Kafka on Kubernetes, making it easier to handle Kafka clusters within a Kubernetes environment. In this article, we'll guide you through opening a shell on a Kafka broker pod in Kubernetes and listing all the topics in your Kafka cluster using an SSL-based connection.