Release blog 2023.3 – the autumn release
In the autumn release of Axual Platform, most of the improvements have been done under the hood to be ready for the future of the platform. For Terraform Provider users we are now enabling the use of Avro schemas and data sources in their terraform declarations.
On this page
Terraform Provider: added AVRO Schema support + Data Sources
In the 2023.2 release we introduced topic approval request support through our Axual Terraform provider. In the latest release of the Axual Terraform provider (2.1.0) we are adding AVRO Schema support and we are supporting Data Sources. In this blog we will explain what that means for collaborating on a streaming platform in a Self-Service manner.
AVRO Schema support
AVRO schema support was one of the last desires expressed by users of the Terraform Provider for Axual, so we are proud to finally bring this to them.
Avro schemas can be made available in Axual Platform by specifying them in terraform files, using the axual_schema_version resource, which in turn points to a file on the file system.
Note: it is important to realize that for every schema, a version needs to be specified.
In the Terraform Registry you will find updated examples which show you how you can make use of Avro schemas for your topics.
Data Sources are now supported
Axual is all about collaborating on a streaming platform in a Self-Service manner. This concept is not limited to the Self-Service UI which is used by developers and engineers in the organization. When using Terraform to declare the desired state (in terms of topics, schemas), you need to refer to resources which are created (and owned) by other people.
For example: Team Awesome created a topic named “transactions” and the topic definition exists in a Git repository owned by the team. If another team, Team Bonanza, wants access to this topic for their consumer application “Alertproducer”, they need to add a reference to this topic in a terraform file in their own Git repo. Lastly, the platform team is responsible for any terraform definitions which should not be controlled by application or data owners, such as. “environments”.
As of release 2.1.0, we have added Data Sources support to the provider. This means you can very easily refer to other resources in the platform such as topics, applications, environments or schemas.
Check out the following documentation page if you want to learn more about using data sources in your terraform files.
Getting ready for the future: Axual Platform architecture 2.0
We speak to many organizations that want to use Apache Kafka strategically to achieve their real-time business objectives. Because of security and compliance limitations, public cloud is simply not an option for many of them. This is where Axual has always provided tools and documentation for on-premises / private cloud deployments.
We noticed however, that installations of the platform at our customers (and subsequent upgrades) became more and more complex. Part of this is caused by the microservice architecture we have adopted over the past couple of years, but there are also different reasons why we need to move to a new platform architecture.
As of release 2023.2 we have been slowly saying goodbye to the original platform architecture, step by step.
In short, we want to achieve the following goals:
- Reduced number of microservices: we are looking at our platform using a more DDD-like approach to see whether we can get rid of microservices by consolidating the responsibilities into one (existing) service. Every microservice we can responsibly drop, reduces the amount of operational overhead on our platform.
- Better component documentation: component documentation has been improved significantly over the last couple of releases to give platform operators the information they need to configure Axual Platform in their organization.
- Reduced conceptual complexity: in the past, we have introduced conceptual layers in our platform, such as “Cluster”, “Instance” and “Platform” (or Management). This causes a lot of confusion in the modern cloud era when people rather talk about control or data plane. By adopting common technology to describe our platform architecture, we are also reducing the conceptual complexity.
What about upgrading to this new architecture?
Axual Platform 2023.4 will be the last platform release in the current (old) architecture. As Kafka is more often than not powering business or mission critical applications, we realize that any upgrade to a newer version might make you a bit nervous as an operator. We are therefore ensuring a smooth transition to this new platform architecture by thoroughly testing and documenting how the upgrade is done by operators.
In addition we will share in more detail what changes we have made to the platform architecture in a future blog post.
Other updates
In our release notes you will find other (smaller) updates to our product, which we are continuously improving with your feedback. Check them out here.
Start your Kafka journey with Axual
Got excited from reading this post? Are you struggling with Kafka in an enterprise setting? We are here to help. Here are a couple of next steps you could take:
Answers to your questions about Axual’s All-in-one Kafka Platform
Are you curious about our All-in-one Kafka platform? Dive into our FAQs
for all the details you need, and find the answers to your burning questions.
Related blogs
Axual 2025.1 is here with exciting new features and updates. Whether you're strengthening security, improving observability, or bridging old legacy systems with modern event systems, like Kafka, Axual 2025.1 is built to keep you, your fellow developers, and engineers ahead of the game.
Consumer group offsets are essential components in Apache Kafka, a leading platform for handling real-time event streaming. By allowing organizations to scale efficiently, manage data consumption, and track progress in data processing, Kafka’s consumer groups and offsets ensure reliability and performance. In this blog post, we'll dive deep into these concepts, explain how consumer groups and offsets work, and answer key questions about their functionality. We'll also explore several practical use cases that show how Kafka’s consumer groups and offsets drive real business value, from real-time analytics to machine learning pipelines.
Apache Kafka is a powerful event-streaming platform, but does your enterprise need to go all in from day one? In this blog, we explore why starting small with Kafka is the best strategy. Learn how an incremental approach can help you reduce complexity, and scale efficiently as your needs grow. Whether you're new to Kafka or looking for a practical implementation strategy, this guide will set you on the right path.