Release blog 2023.3 – the autumn release
In the autumn release of Axual Platform, most of the improvements have been done under the hood to be ready for the future of the platform. For Terraform Provider users we are now enabling the use of Avro schemas and data sources in their terraform declarations.
On this page
Terraform Provider: added AVRO Schema support + Data Sources
In the 2023.2 release we introduced topic approval request support through our Axual Terraform provider. In the latest release of the Axual Terraform provider (2.1.0) we are adding AVRO Schema support and we are supporting Data Sources. In this blog we will explain what that means for collaborating on a streaming platform in a Self-Service manner.
AVRO Schema support
AVRO schema support was one of the last desires expressed by users of the Terraform Provider for Axual, so we are proud to finally bring this to them.
Avro schemas can be made available in Axual Platform by specifying them in terraform files, using the axual_schema_version resource, which in turn points to a file on the file system.
Note: it is important to realize that for every schema, a version needs to be specified.
In the Terraform Registry you will find updated examples which show you how you can make use of Avro schemas for your topics.
Data Sources are now supported
Axual is all about collaborating on a streaming platform in a Self-Service manner. This concept is not limited to the Self-Service UI which is used by developers and engineers in the organization. When using Terraform to declare the desired state (in terms of topics, schemas), you need to refer to resources which are created (and owned) by other people.
For example: Team Awesome created a topic named “transactions” and the topic definition exists in a Git repository owned by the team. If another team, Team Bonanza, wants access to this topic for their consumer application “Alertproducer”, they need to add a reference to this topic in a terraform file in their own Git repo. Lastly, the platform team is responsible for any terraform definitions which should not be controlled by application or data owners, such as. “environments”.
As of release 2.1.0, we have added Data Sources support to the provider. This means you can very easily refer to other resources in the platform such as topics, applications, environments or schemas.
Check out the following documentation page if you want to learn more about using data sources in your terraform files.
Getting ready for the future: Axual Platform architecture 2.0
We speak to many organizations that want to use Apache Kafka strategically to achieve their real-time business objectives. Because of security and compliance limitations, public cloud is simply not an option for many of them. This is where Axual has always provided tools and documentation for on-premises / private cloud deployments.
We noticed however, that installations of the platform at our customers (and subsequent upgrades) became more and more complex. Part of this is caused by the microservice architecture we have adopted over the past couple of years, but there are also different reasons why we need to move to a new platform architecture.
As of release 2023.2 we have been slowly saying goodbye to the original platform architecture, step by step.
In short, we want to achieve the following goals:
- Reduced number of microservices: we are looking at our platform using a more DDD-like approach to see whether we can get rid of microservices by consolidating the responsibilities into one (existing) service. Every microservice we can responsibly drop, reduces the amount of operational overhead on our platform.
- Better component documentation: component documentation has been improved significantly over the last couple of releases to give platform operators the information they need to configure Axual Platform in their organization.
- Reduced conceptual complexity: in the past, we have introduced conceptual layers in our platform, such as “Cluster”, “Instance” and “Platform” (or Management). This causes a lot of confusion in the modern cloud era when people rather talk about control or data plane. By adopting common technology to describe our platform architecture, we are also reducing the conceptual complexity.
What about upgrading to this new architecture?
Axual Platform 2023.4 will be the last platform release in the current (old) architecture. As Kafka is more often than not powering business or mission critical applications, we realize that any upgrade to a newer version might make you a bit nervous as an operator. We are therefore ensuring a smooth transition to this new platform architecture by thoroughly testing and documenting how the upgrade is done by operators.
In addition we will share in more detail what changes we have made to the platform architecture in a future blog post.
Other updates
In our release notes you will find other (smaller) updates to our product, which we are continuously improving with your feedback. Check them out here.
Start your Kafka journey with Axual
Got excited from reading this post? Are you struggling with Kafka in an enterprise setting? We are here to help. Here are a couple of next steps you could take:
Answers to your questions about Axual’s All-in-one Kafka Platform
Are you curious about our All-in-one Kafka platform? Dive into our FAQs
for all the details you need, and find the answers to your burning questions.
Related blogs
Apache Kafka has become a central component of modern data architectures, enabling real-time data streaming and integration across distributed systems. Within Kafka’s ecosystem, Kafka Connect plays a crucial role as a powerful framework designed for seamlessly moving data between Kafka and external systems. Kafka Connect provides a standardized, scalable approach to data integration, removing the need for complex custom scripts or applications. For architects, product owners, and senior engineers, Kafka Connect is essential to understand because it simplifies data pipelines and supports low-latency, fault-tolerant data flow across platforms. But what exactly is Kafka Connect, and how can it benefit your architecture?
Apache Kafka is a powerful platform for handling real-time data streaming, often used in systems that follow the Publish-Subscribe (Pub-Sub) model. In Pub-Sub, producers send messages (data) that consumers receive, enabling asynchronous communication between services. Kafka’s Pub-Sub model is designed for high throughput, reliability, and scalability, making it a preferred choice for applications needing to process massive volumes of data efficiently. Central to this functionality are topics and partitions—essential elements that organize and distribute messages across Kafka. But what exactly are topics and partitions, and why are they so important?
Strimzi Kafka offers an efficient solution for deploying and managing Apache Kafka on Kubernetes, making it easier to handle Kafka clusters within a Kubernetes environment. In this article, we'll guide you through opening a shell on a Kafka broker pod in Kubernetes and listing all the topics in your Kafka cluster using an SSL-based connection.