Enterprise Kafka-based event streaming at scale: build or buy?

“Kafka out-of-the-box isn’t suited for event streaming at enterprise scale. We spent about eight months with four FTEs to build governance, authentication and security layers on top of Kafka just to make secure, controlled event streaming possible across our teams and systems.”

That’s a recent conversation I had with a data engineering manager at a Fortune 500 company.

Enterprise event streaming management

For most enterprises, building an event streaming platform isn’t core to the organization. The saying goes that customers don’t care HOW you do something. Unfortunately, that doesn’t hold true for internal customers. Your DevOps teams have a vested interest in how the goal of event streaming is achieved. And they care very much how easy it is to get their applications streaming.

Kafka features vs. needs for your enterprise

Don’t get me wrong: Kafka is fantastic. It’s backed by a large community. All in all, it’s the best and most widely used streaming technology on the market.

But it’s not easy. There’s no user-friendly interface for configuration and monitoring. Developers need a CLI tool to set up Kafka, which takes expert knowledge of its own. And it’s hard to deploy because there are a LOT of parameters to work with.

Then come the questions of data governance and security. Kafka doesn’t ensure strict data governance (read: more work for your developers). Plus, security isn’t enabled by default. So, it’s possible to unintentionally roll out Kafka wholly unsecured. Tools to secure it are available but, again, that takes expert knowledge.

Core capabilities for enterprises

Over the years, we’ve talked to 100+ enterprise organizations to learn what they need from an event streaming platform. A few things stand out. Governance, support, availability, self-service capabilities, CI/CD support, multitenancy, alerting… In the end, it all boils down to four core capabilities:

• Data governance
• Security
• High availability (across on-prem, cloud or hybrid)
• Self-service for DevOps teams

Building and maintaining your own Kafka event streaming platform

Creating and building software takes a village. To garner longevity and adoption, resources are needed. Taking a product mindset for your event streaming platform requires product-level resources — from product managers, to different types of engineers who can write, run, and perform quality tasks, to the internal champions who can sell the solution.

Focusing on the costs of development, quality assurance and maintenance resources alone, you can see how costs quickly add up. And that doesn’t even take into account the opportunity cost.

To build and maintain such a platform yourself, you need between 8-10 FTE and it takes on average 6-12 months to go into production.

Buying a Kafka event streaming platform

For an enterprise or another big engineering organizations , buying a solution can feel like a blow to your ego. As a DevOps team, data engineer, IT Architect or data analytics leader, you’re perfectly capable of building what you need. And you wouldn’t be an engineer if you didn’t love building things and solving problems.

So why buy? It’s a matter of value and focus. Does the value you gain balance out the time, effort and resources you’ll need to dedicate to it? The decision becomes even harder if you’ve already made an investment and started building.

In the end, your teams just want to start streaming data and build awesome things, instead of maintaining a Kafka instance and building layers on top of it. We encountered a lot of enterprises that went down the road of building it themselves and after 1 or 2 years still bought a commercial solution.

Axual can solve this two-fold challenge for you. We can make sure that the platform continues to be inclusive of your technology (past, present, and future) and keeps up with ongoing demands, such as scalability and security. We learned from a wide-ranging set of customers and continuously improved the platform.

Enterprise data management

Apache Kafka serves as a robust solution for data management in enterprises, offering unparalleled capabilities in handling large volumes of real-time data streams. Its distributed architecture ensures high scalability and fault tolerance, making it ideal for mission-critical applications where reliability is paramount. Kafka’s ability to decouple data streams from applications enables seamless integration with various enterprise systems, facilitating efficient data processing and analytics. By leveraging Kafka, enterprises can achieve comprehensive data management strategies that enhance operational efficiency and support informed decision-making across the organization.

Streaming made simple

It’s our mission to make streaming simple. Axual’s all-in-one platform gets teams up and running with Kafka in no time. We provide a complete streaming platform, including self-service capabilities so teams can easily set up and manage data streams through an intuitive interface. It means you can run your Kafka instance reliably where you need it — on-premise, in the cloud or a hybrid.

All in all, our Axual platform reduces the need for deep Kafka expertise. Less dedicated resources (+- 1 FTE) are needed for maintenance, so you can be up and running in 2 to 4 weeks, not months. And our visual interface and self-service capabilities make it easy to onboard new DevOps teams, so they can start streaming quickly without relying on a central data (platform) team.

Build or buy? The choice is yours to make.

Customer case study – Rabobank

Rabobank, one of our customers in the financial services industry, chose Apache Kafka as the engine for their event streaming platform, but quickly discovered that Kafka was missing a number of features to scale across the enterprise while guarding governance and security. This insight sparked the collaboration between Rabobank and Axual.

Rabobank strategically decided to not develop and maintain their own Kafka platform, but to use a commercial-off-the-shelf solution. This allows them to focus on their own business while lowering the total costs of ownership at the same. Today, Axual is the digital nervous system at Rabobank facilitating 150+ DevOps teams and hundreds of millions of messages per day.

Would you like to start with event streaming or are you reaching the limits of your Kafka instance? Here are some great resources to get started:

• Free trial of Axual’s ready-to-go Kafka platform for enterprises
• Watch the video “what is an event streaming platform” on YouTube
• Talk to our industry experts if you have specific questions or use cases. Get in touch with us here.

‍