Release blog 2023.1 – the spring release
We are well into spring again and the flowers start popping up around us. What a wonderful moment to bring another platform release to the public. We are happy to announce Axual Platform 2023.1.
On this page
We are well into spring again and the flowers start popping up around us. What a wonderful moment to bring another platform release to the public. We are happy to announce Axual Platform 2023.1.
Viewing topics and applications in various scopes
If you are in an organization that is successful in leveraging Apache Kafka, dozens or perhaps hundreds of topics are created and an equally large number of applications are interacting with those topics. Keeping a good overview of what is happening on the platform is definitely a huge challenge. In a previous version of the platform we introduced a way to browse through environments to see which topics and applications are active there and how they are interacting with each other. In this release we are adding the possibility to select different scopes to display the topics and applications in.
You can select the following scopes:
- All teams: the entire organization
- My teams: all teams that the logged-in user is a member of
Consumer lag – how is my consumer doing?
As an application developer, especially in a DevOps setting, you need to make sure your application is running as it should. There are various metrics that you can look at and when it comes to applications interacting with Kafka, there is a specific one that is very extremely important.
Intermezzo: partitions and parallelism on Kafka
When you create a topic on Kafka, you define the number of partitions that you want to create for your topic. Those partitions are important to enable parallelism on Kafka as:
- Brokers are the leader for 1..n topic partitions
- All read-write operations from clients (producer/consumers) happen through the leader of the topic partitions.
For example, if you have a topic with 3 partitions and a 3 broker cluster, and the partitions are evenly distributed on the cluster, each broker takes up a third of the messages on the topic.
A core characteristic of Kafka is that messages on 1 topic partition can only be read by 1 instance of a consuming application at the same time. In a situation where you have a high input on the topic, e.g. 300 messages per second, and your consumer instances together cannot process more than 250 messages per second, you will start to lag. This is called consumer lag. In release 2022.3 we introduced the Consumer Lag metric to the metrics API. As part of release 2023.1 we are showing this metric in the user interface. If you have a consuming application, the consumer lag in records will be shown on the connection between the topic and the application.
Because you might consume from multiple topics, the lag is shown for each topic you are consuming from.
If you want to get the consume lag programmatically, please check the API documentation.
Huge improvement of the online documentation
In the last couple of months we have made significant improvements to our online documentation. We have restructured the content of the documentation and we have also made it possible to search through the entire collection of documentation repositories.
Please check it out by going to https://docs.axual.io and let us know what you think.
In other news: Launching Axual Governance
Only a week ago we released Axual Governance. With Axual Governance, we are bringing data governance and self-service capabilities to organizations that are already using Kafka. If you have an Aiven or Confluent Cloud Kafka cluster, you can start the onboarding right away. For Plain Kafka and Strimzi users, we are going to add support for onboarding soon.
Read more about Axual Governance in this blog or sign up for yourself!
Strimzi upgrade
Axual Platform is based on the Strimzi Operator. For the 2023.1 release we have upgraded to the latest Strimzi release, Strimzi 0.34.0 . It allows for an upgrade to the 3.4.0 release of Apache Kafka. The operator instructions contain upgrade steps related to Axual Operator 0.10.0 to upgrade to Kafka 3.4.0. You can find the operations documentation here.
Axual Operator 0.10.0 is effectively the last release of Axual Operator. As the Strimzi community introduced features to the operator which allows it to be configured and plug in functionality, like a PrincipalBuilder, there is no need anymore for us to fork Strimzi Operator.
On-premise installations of our platform can in the future use Strimzi operator with a reference to images prepared by Axual. We will send out an update about this in the coming months.
A more extensive list of updates done to Axual Platform can be found in the release notes.
Answers to your questions about Axual’s All-in-one Kafka Platform
Are you curious about our All-in-one Kafka platform? Dive into our FAQs
for all the details you need, and find the answers to your burning questions.
Related blogs
Apache Kafka has become a central component of modern data architectures, enabling real-time data streaming and integration across distributed systems. Within Kafka’s ecosystem, Kafka Connect plays a crucial role as a powerful framework designed for seamlessly moving data between Kafka and external systems. Kafka Connect provides a standardized, scalable approach to data integration, removing the need for complex custom scripts or applications. For architects, product owners, and senior engineers, Kafka Connect is essential to understand because it simplifies data pipelines and supports low-latency, fault-tolerant data flow across platforms. But what exactly is Kafka Connect, and how can it benefit your architecture?
Apache Kafka is a powerful platform for handling real-time data streaming, often used in systems that follow the Publish-Subscribe (Pub-Sub) model. In Pub-Sub, producers send messages (data) that consumers receive, enabling asynchronous communication between services. Kafka’s Pub-Sub model is designed for high throughput, reliability, and scalability, making it a preferred choice for applications needing to process massive volumes of data efficiently. Central to this functionality are topics and partitions—essential elements that organize and distribute messages across Kafka. But what exactly are topics and partitions, and why are they so important?
Strimzi Kafka offers an efficient solution for deploying and managing Apache Kafka on Kubernetes, making it easier to handle Kafka clusters within a Kubernetes environment. In this article, we'll guide you through opening a shell on a Kafka broker pod in Kubernetes and listing all the topics in your Kafka cluster using an SSL-based connection.