Standardizing Kafka for the enterprise – requirements and organizational challenges (part 1)
Standardizing Apache Kafka for the enterprise can be challenging, but it is possible with a three-pillar approach: enablement, support, and technology.
On this page
Standardizing Kafka across the enterprise comes with a lot of benefits.
It makes it easier to onboard new developers, it’s scalable and future-proof as you rely on best practices, it’s easier to collaborate and exchange data, and it saves money because maintenance is more streamlined.
But it’s not an easy task. It’s not something you achieve with the push of a button.
And it’s more than just technology.
But, no worries. We’re here to help.
In this blog post, I will explain some pragmatic steps you can think about and discuss together with your architecture group.
3-step approach to standardize Apache Kafka for the enterprise
Larger enterprises often operate in a highly regulated environment with operational pressure – meaning that downtime will not be tolerated. In addition to this, ensuring customer data is handled in a secure, compliant manner is a top priority for developers and operators.
To achieve this, you need a combination of enablement, support and technology. You need to train people who work with Kafka, you need to support and guide them with expert knowledge and you need to provide teams with Kafka as a platform.
Let’s dive into these three pillars.
Empowerment
Empowerment is all about empowering your development teams for success.
Training
Foster a learning culture by hosting monthly talks featuring internal and external experts. Develop an internal academy, where both business and IT colleagues can participate, ensuring a holistic understanding of Kafka and Event-Driven Architecture (EDA). Design the academy to include separate sessions for business users, encouraging them to articulate their business use cases and needs.
Schema registry and event catalog
A standardized schema registry and event catalog are foundational for consistent data management. Ensure that schemas and events are managed uniformly across the enterprise. This approach involves tracking schema versions and enforcing compatibility checks to prevent disruptive changes, thereby allowing schemas to evolve without breaking existing applications. Implement tools and governance mechanisms to maintain consistency in schema management.
Support
Determine streaming solutions for your architects
Not every architect is a streaming expert. Therefore it’s helpful to provide architects in your Enterprise Architecture Team with a standard approach for known streaming challenges. Think about default solutions for data (and database) synchronization, microservices, data retention, and more.
CCE in the Enterprise Architecture Team
Centralized Competence Centers of Excellence (CCE) with engineering skills provide dedicated support, ensuring that teams receive expert guidance in Kafka implementations. Integrate CCE within the Enterprise Architecture team, creating a hub of expertise. Foster collaboration between CCE and development teams to provide timely support and facilitate knowledge sharing.
Kafka as a platform
The benefit of having Kafka as a platform is that you’re able to split the responsibilities between functional (topic access, application registry and more) and technical maintenance (keeping the platform up and running). The ultimate goal is to reduce the number of bottlenecks and make sure that every team is as autonomous as possible. Let’s dive into the different aspects needed to achieve this.
Different needs for different personas
The personas associated with creating and using your EDA landscape don’t all have the same requirements for event management. Here’s an overview of different personas and their slice of event management:
- Platform Engineers – they are mostly focused on making sure the platform is stable and runs smoothly.
- Data Owners – they manage the quality, usage, and accessibility of the datasets that live on the platform. Not everyone should have access to sensitive data.
- Architects – they are interested in design principles, security, governance, and the life cycle of topics, applications, and schemas.
- Developers – they want to register their application and start streaming without the need for a Kafka team.
Data ownership and governance
What organizations need is a structure of data governance that allows for secure, controlled access for Kafka topic and application administration — without losing essential business agility. For large organizations, this will be federated. Access control is the management of permissions, determining precisely who is granted access to specific resources. This level of control is crucial for overseeing data and applications. For instance, it is essential to grant exclusive access to the fraud team for topics containing additional payment information, while restricting access for other teams is equally important.
Self-service for DevOps teams
Empower DevOps teams with self-service capabilities, reducing dependencies on the central platform team and accelerating development cycles. DevOps teams need to be able to manage their piece of the Kafka pie, without relying on a platform (or Kafka) team.
Centralized maintenance
Designate a central platform team responsible for ongoing, technical maintenance. This team ensures the platform remains robust, secure, and aligned with evolving organizational needs. We’ve seen enterprises only using 0.5FTE to maintain the event streaming platform while delivering Kafka-as-a-service across hundreds of DevOps teams. If you do it right, it doesn’t have to be a cumbersome process.
Monitoring and observability
For monitoring Kafka’s stability and performance, Axual probably recommends focusing on essential metrics such as broker resource usage, disk I/O, network throughput, topic and partition counts, consumer group lag, and message rates. Monitoring these key indicators helps in early identification of potential issues, ensuring Kafka’s reliability and efficiency.
Integrating with CI/CD pipelines
Integrating Axual’s Kafka standardization approach with CI/CD pipelines involves automating schema registration, topic creation, and access control configurations. By managing these elements as code, DevOps teams can automate the deployment of Kafka resources alongside application code, maintaining development agility while adhering to governance standards.
Remember, standardizing Kafka is a journey, not a destination. It requires vision, commitment, and a willingness to transform how your enterprise handles data. But with the right approach, the rewards—enhanced agility, improved decision-making, and robust compliance—are well worth the effort.
Answers to your questions about Axual’s All-in-one Kafka Platform
Are you curious about our All-in-one Kafka platform? Dive into our FAQs
for all the details you need, and find the answers to your burning questions.
This standardized integration mechanism streamlines data pipeline management, making it more reliable and easier to maintain. Kafka also serves as a centerpiece in microservices architectures, where it enables services to communicate asynchronously through events.
Kafka comprises brokers, controllers, partitions, consumers, producers, topics, schema registries, KRaft. Consumers and producers interact with topics, and Kafka Streams provides stream processing. Kafka Connect facilitates connecting Kafka to external data sources and sinks.
Organizations need to implement data quality checks, data cleansing, validation, and monitoring processes to ensure data quality in a Kafka architecture. These measures help identify and rectify data quality issues in real-time, ensuring that the data being streamed is reliable, accurate, and consistent.
Related blogs
Apache Kafka has become a central component of modern data architectures, enabling real-time data streaming and integration across distributed systems. Within Kafka’s ecosystem, Kafka Connect plays a crucial role as a powerful framework designed for seamlessly moving data between Kafka and external systems. Kafka Connect provides a standardized, scalable approach to data integration, removing the need for complex custom scripts or applications. For architects, product owners, and senior engineers, Kafka Connect is essential to understand because it simplifies data pipelines and supports low-latency, fault-tolerant data flow across platforms. But what exactly is Kafka Connect, and how can it benefit your architecture?
Apache Kafka is a powerful platform for handling real-time data streaming, often used in systems that follow the Publish-Subscribe (Pub-Sub) model. In Pub-Sub, producers send messages (data) that consumers receive, enabling asynchronous communication between services. Kafka’s Pub-Sub model is designed for high throughput, reliability, and scalability, making it a preferred choice for applications needing to process massive volumes of data efficiently. Central to this functionality are topics and partitions—essential elements that organize and distribute messages across Kafka. But what exactly are topics and partitions, and why are they so important?
Strimzi Kafka offers an efficient solution for deploying and managing Apache Kafka on Kubernetes, making it easier to handle Kafka clusters within a Kubernetes environment. In this article, we'll guide you through opening a shell on a Kafka broker pod in Kubernetes and listing all the topics in your Kafka cluster using an SSL-based connection.