Governing Your Data: The Kafka Compliance Checklist

So you’ve architected a new streaming platform. It has quickly become the ‘central nervous system’ of your company. A couple of teams have already started working with Kafka and they’re continuously adding new real-time streaming use cases.

Your teams are excited to be streaming the data they need in real time. Everyone is happy. Then, one night, you wake up in a panic. You forgot something…

“How do I make sure it all stays compliant?”

We get it. Making sure Kafka remains compliant can be a real nightmare. Topics. Clusters. Schemas. Producers. Consumers. Authentication. Authorization. Sensitive information. The list goes on and on.

The good news is: You’re not alone in this challenge. And we can help you get a good night’s rest. We created the Kafka Compliance Checklist to give an overview of the things you need to think about and help streamline your internal processes.

1. KNOW YOUR TOPICS

Which topics exist on your Kafka clusters?

Identifying a topic by name alone can be hard, especially if you haven’t standardized topic naming. Including descriptive data on your topics, topic metadata, is extremely helpful. Imagine you’re building a new use case that requires a Kafka topic. That topic might already exist. Even better: If you know which contract or schema is used for the topic, you can be up-to-speed even quicker.

2. KNOW YOUR OWNERS

Which applications are producing to and consuming from your topics?

To get an overview of who is interacting with Kafka topics, it’s important to identify producers and consumers. Who maintains these applications? How can you contact them? If you have a major incident in one of your producing applications, getting in touch with the owner will help you assess the impact for consumers downstream.

3. SECURE YOUR PLATFORM

Are you allowing secure connections only?

Don’t trust any network — including your own. Your topics may hold sensitive information. You don’t want this ending up in the wrong hands. Configuring secure listeners on your Kafka platform helps you monitor and control who reads from and writes to your topics.

4. AUTHORIZE & AUTHENTICATE

How have you secured your Kafka platform?

There are a number of useful questions to ask here. Are you authenticating all client connections? Is every application authorized to access all topics by default? Or are you authorizing only as required? If so, what are your criteria for authorization? And who is responsible for approvals: your central ‘stream team’ or the owners of the data? How about creating topics? Do you allow producers to create new topics yet?

5. KNOW WHAT’S HAPPENING

Do you have an overview of changes on your platform?

If you zoom in on a Kafka platform, changes are made on a daily basis.Topics are created. Topics are configured. Authorizations are modified. These may seem like small changes, but they can have a big impact on compliance. Are you auditing changes as they happen, so you don’t have to play the detective later?

DATA GOVERNANCE AT SCALE WITH AXUAL

That’s it! We hope this Kafka Compliance Checklist helps you organize your approach to compliance.

What organizations need is a structure of data governance that allows for secure, controlled access for Kafka topic administration — without losing essential business agility. That’s where Axual comes in. If you would like to learn how Axual’s all-in-one Kafka platform helps to keep development teams productive while staying secure and compliant, feel free to request a free demo or 14-day trial.

And, of course, we’re here to help with any questions you have regarding Kafka and data governance. We would love to share our experiences. Feel free to get in touch anytime.

‍