Standardizing Kafka for the enterprise – requirements and organizational challenges (part 1)

Standardizing Kafka across the enterprise comes with a lot of benefits.

It makes it easier to onboard new developers, it’s scalable and future-proof as you rely on best practices, it’s easier to collaborate and exchange data, and it saves money because maintenance is more streamlined.

But it’s not an easy task. It’s not something you achieve with the push of a button.

And it’s more than just technology.

But, no worries. We’re here to help.

In this blog post, I will explain some pragmatic steps you can think about and discuss together with your architecture group.

3-step approach to standardize Apache Kafka for the enterprise

Larger enterprises often operate in a highly regulated environment with operational pressure – meaning that downtime will not be tolerated. In addition to this, ensuring customer data is handled in a secure, compliant manner is a top priority for developers and operators.

To achieve this, you need a combination of enablement, support and technology. You need to train people who work with Kafka, you need to support and guide them with expert knowledge and you need to provide teams with Kafka as a platform.

Let’s dive into these three pillars.

Empowerment

Empowerment is all about empowering your development teams for success.

Training
Foster a learning culture by hosting monthly talks featuring internal and external experts. Develop an internal academy, where both business and IT colleagues can participate, ensuring a holistic understanding of Kafka and Event-Driven Architecture (EDA). Design the academy to include separate sessions for business users, encouraging them to articulate their business use cases and needs.

Schema registry and event catalog
A standardized schema registry and event catalog are foundational for consistent data management. Ensure that schemas and events are managed uniformly across the enterprise. This approach involves tracking schema versions and enforcing compatibility checks to prevent disruptive changes, thereby allowing schemas to evolve without breaking existing applications. Implement tools and governance mechanisms to maintain consistency in schema management.

Support

Determine streaming solutions for your architects
Not every architect is a streaming expert. Therefore it’s helpful to provide architects in your Enterprise Architecture Team with a standard approach for known streaming challenges. Think about default solutions for data (and database) synchronization, microservices, data retention, and more.

CCE in the Enterprise Architecture Team
Centralized Competence Centers of Excellence (CCE) with engineering skills provide dedicated support, ensuring that teams receive expert guidance in Kafka implementations. Integrate CCE within the Enterprise Architecture team, creating a hub of expertise. Foster collaboration between CCE and development teams to provide timely support and facilitate knowledge sharing.

Kafka as a platform

The benefit of having Kafka as a platform is that you’re able to split the responsibilities between functional (topic access, application registry and more) and technical maintenance (keeping the platform up and running). The ultimate goal is to reduce the number of bottlenecks and make sure that every team is as autonomous as possible. Let’s dive into the different aspects needed to achieve this.

Different needs for different personas
The personas associated with creating and using your EDA landscape don’t all have the same requirements for event management. Here’s an overview of different personas and their slice of event management:

• Platform Engineers – they are mostly focused on making sure the platform is stable and runs smoothly.
• Data Owners – they manage the quality, usage, and accessibility of the datasets that live on the platform. Not everyone should have access to sensitive data.
• Architects – they are interested in design principles, security, governance, and the life cycle of topics, applications, and schemas.
• Developers – they want to register their application and start streaming without the need for a Kafka team.

Data ownership and governance
What organizations need is a structure of data governance that allows for secure, controlled access for Kafka topic and application administration — without losing essential business agility. For large organizations, this will be federated. Access control is the management of permissions, determining precisely who is granted access to specific resources. This level of control is crucial for overseeing data and applications. For instance, it is essential to grant exclusive access to the fraud team for topics containing additional payment information, while restricting access for other teams is equally important.

Self-service for DevOps teams
Empower DevOps teams with self-service capabilities, reducing dependencies on the central platform team and accelerating development cycles. DevOps teams need to be able to manage their piece of the Kafka pie, without relying on a platform (or Kafka) team.

Centralized maintenance
Designate a central platform team responsible for ongoing, technical maintenance. This team ensures the platform remains robust, secure, and aligned with evolving organizational needs. We’ve seen enterprises only using 0.5FTE to maintain the event streaming platform while delivering Kafka-as-a-service across hundreds of DevOps teams. If you do it right, it doesn’t have to be a cumbersome process.

Monitoring and observability

For monitoring Kafka’s stability and performance, Axual probably recommends focusing on essential metrics such as broker resource usage, disk I/O, network throughput, topic and partition counts, consumer group lag, and message rates. Monitoring these key indicators helps in early identification of potential issues, ensuring Kafka’s reliability and efficiency.

Integrating with CI/CD pipelines

Integrating Axual’s Kafka standardization approach with CI/CD pipelines involves automating schema registration, topic creation, and access control configurations. By managing these elements as code, DevOps teams can automate the deployment of Kafka resources alongside application code, maintaining development agility while adhering to governance standards.

Remember, standardizing Kafka is a journey, not a destination. It requires vision, commitment, and a willingness to transform how your enterprise handles data. But with the right approach, the rewards—enhanced agility, improved decision-making, and robust compliance—are well worth the effort.

‍

‍