Business 25 Nov 2021

Is it time to decentralize your data organization?

Centralized data administration can be great for smaller organizations. A single team of admins acts as gatekeeper on an event streaming platform, like Apache Kafka. It means data governance can be enforced from the top down to ensure full data compliance.  But as your business grows, you find more and more developers need you to […]

Centralized data administration can be great for smaller organizations. A single team of admins acts as gatekeeper on an event streaming platform, like Apache Kafka. It means data governance can be enforced from the top down to ensure full data compliance. 

But as your business grows, you find more and more developers need you to create and configure topics for them and you are in need of stricter Kafka topic access control. Without continuously scaling your central team, they simply won’t be able to keep up with the sheer volume of requests. Your central organization can quickly go from gatekeeper to bottleneck.

In a centralized structure, your dev teams rely on the central team for Kafka topic ownership and troubleshooting. And they need it promptly to stay on track. Without it, timelines slip and features take too long to implement. If developers or product managers are complaining about delays, the writing is on the wall: Your days of central data governance are numbered. 

4 STEPS TO EFFECTIVE SELF-SERVICE FOR KAFKA

When you are convinced and ready to move to self-service for Kafka data compliance, there are four steps to take. 

1. Create an inventory

Start with an inventory. Which Kafka topics do you have? What data do they hold? Are (Avro) schemas used to describe the payload on the topics? You need to get their administration and metadata in order so you can start from a clean base. 

2. Decide on ownership

In a best-case scenario, who should be responsible? Who needs to own data and topics? You may want to give the supplier (producer) control of Kafka compliance. Another option is to give the business side the responsibility. Before you implement anything, you’ll need a clear vision of what your ideal setup looks like.

3. Decentralize

Now you can set up automated processes for topic creation and configuration based on the ownership decisions you’ve made. Make sure the right teams have the right roles to guarantee Kafka security.

4. Communicate

To get self-service up and running smoothly, communication is key. It remains a critical central role. Communicate best practices clearly to quickly get everyone working on the same platform in the same way. 

AXUAL SELF-SERVICE FOR KAFKA DATA GOVERNANCE

More and more organizations are using Axual to answer their compliance and security needs around Apache Kafka. Data and application ownership are core to the platform. No Kafka topic is created before an owning team and metadata have been defined. While the team starts with configuring the topic and defining schemas, it also has the responsibility to selectively allow produce/consume access to the topic. Any team that interacts with the streaming platform uses a one stop, easy to use Self-Service interface. This interface helps to maintain the big picture of “anything streaming” within the organization. 

Giving control to your app developers means they’re free to try out different solutions — with guard rails in place to ensure security and compliance. And your IT department can focus on facilitating innovation. Their core function is what they do best: keeping the platform available, secure, and suitably scaled as you grow. 

NEXT STEPS

Ready to decentralize your data organization? It is not a decision made lightly. Start by gathering the information you need to base your decision on:

We’re here to help whenever you’re ready. Feel free to get in touch anytime.

 

Other blogs

3 hours ago

Axual named a representative vendor in the 2022 Gartner market guide for event stream processing

Joris Meijer
Product 1 month ago

Axual Release Update 2021.3

2021 is almost a wrap, but not until we have announced our last major release of this year: Axual Platform 2021.3. Read all about it in this blog.

Joris Meijer
Technology 2 months ago

Governing Your Data: The Kafka Compliance Checklist

So you’ve architected a new streaming platform. It has quickly become the ‘central nervous system’ of your company. A couple of teams have already started working with Kafka and they’re continuously adding new real-time streaming use cases. 

Joris Meijer

Apache Kafka is great, but what do you do
when great is not good enough?
See what Axual offers on top of Kafka.

Start your free trial
No credit card required