Is it time to decentralize your data organization?

Centralized data administration can be great for smaller organizations. A single team of admins acts as gatekeeper on an event streaming platform, like Apache Kafka. It means data governance can be enforced from the top down to ensure full data compliance.

But as your business grows, you find more and more developers need you to create and configure topics for them and you are in need of stricter Kafka topic access control. Without continuously scaling your central team, they simply won’t be able to keep up with the sheer volume of requests. Your central organization can quickly go from gatekeeper to bottleneck.

In a centralized structure, your dev teams rely on the central team for Kafka topic ownership and troubleshooting. And they need it promptly to stay on track. Without it, timelines slip and features take too long to implement. If developers or product managers are complaining about delays, the writing is on the wall: Your days of central data governance are numbered.

4 STEPS TO EFFECTIVE SELF-SERVICE FOR KAFKA

When you are convinced and ready to move to self-service for Kafka data compliance, there are four steps to take.

1. Create an inventory

Start with an inventory. Which Kafka topics do you have? What data do they hold? Are (Avro) schemas used to describe the payload on the topics? You need to get their administration and metadata in order so you can start from a clean base.

2. Decide on ownership

In a best-case scenario, who should be responsible? Who needs to own data and topics? You may want to give the supplier (producer) control of Kafka compliance. Another option is to give the business side the responsibility. Before you implement anything, you’ll need a clear vision of what your ideal setup looks like.

3. Decentralize

Now you can set up automated processes for topic creation and configuration based on the ownership decisions you’ve made. Make sure the right teams have the right roles to guarantee Kafka security.

4. Communicate

To get self-service up and running smoothly, communication is key. It remains a critical central role. Communicate best practices clearly to quickly get everyone working on the same platform in the same way.

AXUAL SELF-SERVICE FOR KAFKA DATA GOVERNANCE

More and more organizations are using Axual to answer their compliance and security needs around Apache Kafka. Data and application ownership are core to the platform. No Kafka topic is created before an owning team and metadata have been defined. While the team starts with configuring the topic and defining schemas, it also has the responsibility to selectively allow produce/consume access to the topic. Any team that interacts with the streaming platform uses a one stop, easy to use Self-Service interface. This interface helps to maintain the big picture of “anything streaming” within the organization.

Giving control to your app developers means they’re free to try out different solutions — with guard rails in place to ensure security and compliance. And your IT department can focus on facilitating innovation. Their core function is what they do best: keeping the platform available, secure, and suitably scaled as you grow.

NEXT STEPS

Ready to decentralize your data organization? It is not a decision made lightly. Start by gathering the information you need to base your decision on:

• Take a hard look at your data and how you currently handle it. Is it slowing the pace of innovation?
• Talk to your development teams. Are they able to work as fast as they’d like? Do they have the tools they need to innovate efficiently?
• Talk to your central platform team. Are they happy with the current situation? Are they able to focus on their core responsibilities?

We’re here to help whenever you’re ready. Feel free to get in touch anytime.