Maturing your Kafka platform: From experiments to centralization

Successfully implementing organization-wide Kafka is complex and potentially high-risk for enterprise organizations. Yet the results are well worth pursuing: Leveraging Apache Kafka, the gold standard in data streaming, to optimize service provision with real-time data insights.  

Here, we’ll explore the typical evolution from dispersed event streaming to pervasive Kafka, reflecting Gartner’s Maturity Model for Event Driven Architecture.

Stage 1: Incidental usage — before Kafka

At the incidental stage, an organization has basic, dispersed procedures in place for event management. Usually, events are coming in from a handful of sources but applications handle them individually, with no central event management infrastructure in place.

Organizations began processing data a long time ago — far before events were a recognized concept or asset. Over recent decades, however, the volume of data they’re constantly being bombarded with has rapidly increased in volume and origin:

• Customer channels
• Communication with suppliers/business partners
• Internal data sources
• Public data sources

Typically, enterprises implement a 3-step process to ingest data and process this ever-increasing information:

Step 1: Collect, ingest, filter, route

Step 2: Interpret, associate, process, calculate

Step 3: Weigh, decide, plan, act

This process has accelerated over recent years, presenting increasing challenges for enterprises attempting to deal with the growing data influx. Intensifying source, volume, ingestion, and instant processing needs lead organizations to turn to event streaming solutions — at which point event brokers play a crucial role.

Stage 2: Event brokers introduced — local Kafka

As organizations realize the need to rely on a central broker to manage their information exchanges, they tend to start by looking for the streaming technology they’ll adopt. They enter a diverse landscape of potential streaming engines, including SaaS building blocks offered by large cloud vendors, open source options, and proprietary solutions.

Over the years, we’ve seen enterprises increasingly select Apache Kafka as their preferred event broker (in other words, their engine of choice for handling their organization-wide data streaming).

As a fault-tolerant, scalable, publish-subscribe messaging system enabling enterprises to develop distributed applications and run web-scale online businesses, Kafka has established itself as the gold standard for high-volume, real-time data streaming among enterprise organizations.

Yet it’s not without its challenges for organizations’ in-house teams. At the initial implementation stage, these are mostly technological:

• There’s a steep learning curve with Kafka, even for experienced IT architects
• Setting up a robust, secure Kafka platform is a complex process
• Maintaining and answering user questions about the platform creates an added burden
• There are many, high-complexity failure scenarios, which your internal team needs to be aware of to support and leverage Kafka to its full extent

For organizations that opt to deploy Kafka across various teams, it soon becomes clear that having each of these teams deploy and maintain Kafka independently isn’t the most efficient approach. Instead, centralized scenarios become an increasingly attractive option.

Stage 3: Event broker system — centralized Kafka

At this stage, organizations develop a centralized event and broker setup, through which their multiple internal teams can interconnect.  

With all your departments — as well as their connected applications — arranged around a centralized Kafka, you’ll unite a diverse range of business areas and skill sets to exchange business events on equal footing. This rapidly expands the number of people who are (in)directly involved with your centralized Kafka, all of whom will naturally have their own questions about and requirements of the platform.

Who’s using your centralized Kafka platform?

Data Owners

“I want to be in control of streaming data.”

IT Architects

“I want a ready-to-go and future-proof streaming platform with great security and a proven track record.”

Developers

“I want to develop robust streaming applications quickly and independently.”

Operators

“I want a predictable and maintainable platform.”

Procurement

“I want compliant and credible vendors that my organization can rely on.”

But, be aware – out-of-the-box Kafka isn’t enough to meet full-range requirements

With an ever-widening range of employee profiles required to connect to your centralized Kafka, the platform will need to cover all their typical requirements, in all their diversity. The issue is, out-of-the-box Kafka only supports a handful of them:

✓ Streaming technology: Purpose-built streaming technology core

✓ Vendor credibility: Multiple credible vendors to choose from

✓ Proven solution: Solid track record

However — if working without a specialist Kafka partner — your organization’s central Kafka team will face complex challenges during the shift to a centralized Kafka:

• Building your organization’s own watertight governance mechanism
• Configuring foolproof security from scratch
• Providing a Kafka self-service environment for your DevOps teams
• Supporting your organization’s entire Kafka stack — a significant (and increasing) challenge as the scope of your Kafka platform expands

When your Kafka team does overcome these hurdles, your organization will be primed to mature into the advanced stages of event-driven architecture.

Stages 4 and 5: Advanced Kafka presence — Kafka everywhere

At this stage, your organization’s myriad apps will be exchanging events as a standard approach to data exchange. You’ll have Kafka documentation across your internal processes, as it’s just another of your company’s regular integration methods.

The challenges your central Kafka team faces are now likely to be organizational, rather than technical:

• Defining cohesive organization-wide standards
• Governing your Kafka catalogs (topic, schema)
• Accelerating internal Kafka onboarding
• Handling your growing Kafka ecosystem as its scope steadily expands

Meet Axual – the ready-to-go streaming platform for enterprises

Recognizing that enterprises experience predictable maturation pains from early-stage to pervasive Kafka, the Axual platform is designed to solve the hurdles along the way.

Axual is a ready-to-go streaming platform for enterprises, trusted by sector leaders across finance, energy, utilities, and more. To provide a simplified contextual frame, Axual to Kafka is as Red Hat to Linux. Axual provides self-service, data governance and security on top of Kafka to make streaming simple at enterprise scale.

Essentially, Axual guides organizations towards a centralized Kafka and enterprise-wide data exchange. Within the Kafka maturity model we’ve explored, Axual best serves organizations approaching the centralized Kafka stage, then maturing into the advanced Kafka stage.

Axual is compiled to fill the gaps in out-of-the-box Kafka, meeting the typical persona requirements from Kafka stakeholders across enterprise organizations — from Data Owers to DevOps teams and from IT Architects to IT Operations.

Further insights on successful event streaming?
Watch our on-demand webinar “enterprise-wide data exchange – from Kafka experiments to a centralized Kafka” here.

‍