What is KSML

Kafka Streams without Java

Describe sources, joins, and aggregations in YAML, add Python where needed, and run on proven Kafka Streams without Java.

Download Axual Desktop
Read docs
How it works

KSML runs on proven Kafka Streams without Java

KSML turns Kafka Streams topologies into something you write, read, and change without touching Java.

Go to code on GitHub
KSML with AI

One low-code layer,
3 reasons teams adopt it

Low-Code Stream Processing
Easy, Readable Syntax
Detail
Write streaming applications using nothing more than YAML and Python snippets
The flow of messages is easy to read and modify
Import existing Python libraries directly into KSML definitions
No Java development or build pipelines necessary
Insert a new operation in under 10 seconds
Reuse a wealth of existing code in your streaming apps
Runs on proven Kafka Streams underneath
Your new app runs in under 20 seconds
No need to reinvent transformation logic from scratch

Value proposition tiles

Faster time-to-value

Skip the build pipeline. Go from an idea for a stream transformation to a running application in minute.

Governed by design

KSML applications register, deploy, and run directly on Axual inheriting the same ownership, environment, and access controls as the rest of your Kafka environment.

AI-assisted

Generate KSML applications with AI through Axual MCP, then register and run them on Axual

What you see is what you stream

The YAML topology is the documentation. Anyone on the team can read what a pipeline does.

Built for logic, not toy examples

Filter, join, aggregate, and route production data  KSML is used to power real-time notification logic processing 100M+ messages a day.

Rated with a 4.5 from more than
13 G2 reviews
Quick start

Get started with KSML

Download the KSML Repository

  • Go to the KSML repository
  • Clone the repository with GIT, or download a ZIP file using the Code button.
If you've downloaded a zip file, extract it to a working directory
  • Open the repository in your favorite editor.
We will be working in the root and the examples directory of the project.
    The other directories contain the source code of KSML. You can take a look if you're interested.

Start a local Kafka environment + data generator

  • Open the ‘docker-compose.yml’ file in the root directory.
This file start a Kafka environment with Schema Registry and a data generator for the examples.
  • Perform a ‘docker-compose up -d’ in the root directory to start the environment in the background

    Docker has recently introduced an alternative command ‘docker compose up -d’.
version: "3.7"
services:
# Services can be reached from the local machine and in the docker network
# From the host machine:
# * Localhost port 9092 connects to the broker
# * Localhost port 8081 connects to the schema registry
# From the Docker network:
# * broker port 9093 connects to the broker
# * schema_registry port 8081 connects to the broker
networks:
ksml:    
  name: ksml_example    
  driver: bridge

Verify that the data generator is running

Perform a 'docker-compose logs -d example-producer'. By looking at your console you should see the data generator in action, you should see produce messages appear.

The service can fail several times while waiting for the Kafka topics to be created.

Docker has recently introduced an alternative command 'docker compose logs -d example-producer'.
docker-compose logs -f example-producer
example-producer_1 | 2021-05-11T09:55:00,169Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer - Using Kafka backendexample-producer_1 | 2021-05-11T09:55:00,445Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer - Producing: 0example-producer_1 | 2021-05-11T09:55:01,161Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Produced message totopic ksml_sensordata_avro partition 0 offset 0example-producer_1 | 2021-05-11T09:55:01,661Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Producing: 1example-producer_1 | 2021-05-11T09:55:01,667Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Produced message totopic ksml_sensordata_avro partition 0 offset 1example-producer_1 | 2021-05-11T09:55:02,167Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Producing: 2example-producer_1 | 2021-05-11T09:55:02,173Z [system] [main]

Build your KSML definitions

The next step is to build a KSML definition that is going to process the data. Below you will find 3 simple examples that help you to understand the KSML syntax better and see what it is capable of.

Example 1
Inspecting topic data
Create a file ‘sensor-inspect.yaml’ in the folder ‘/examples’ with the following contents
Examples/sensor-inspect.yaml
streams:    
  ksml_sensordata_avro:        
    topic: ksml_sensordata_avro        
    keyType: string        
    valueType: avro: SensorData
functions:    
  print_message:        
  type: forEach        
  code: "print('key='+(key if isinstance(key,str) else          str(key))+', value='+(value if isinstance(value,str) else str(value)))"

pipelines:
main:

Launch the KSML Runner

It's time to start a KSML Runner with your KSML definition.

  • Open the file ‘ksml-runner.yml’ in the ‘examples’ directory.
  • replace the reference to ‘1-demo-inspect.yaml' with ‘sensor-inspect.yaml’
  • Execute the ‘run.sh’ script in the examples directory

If everything goes well, you should see the AVRO sensordata being written to STDOUT.

Examples/ksml-runner.yml
ksml:
  workingDirectory: /ksml
  definitions:
  - sensor-inspect.yaml
‍
backend:
  type: kafka
  config:
    bootstrapUrl: broker:9093
    applicationId: io.ksml.example.processor
    schemaRegistryUrl: http://schema_registry:8081
    streamsConfig:
    acks: all

Try out more examples

Now that your setup works fine and you have successfully launched the first KSML app, let's add some more.
Read docs
Dynamic topic routing
streams:
  ksml_sensordata_avro:
    topic: ksml_sensordata_avro
    keyType: string
    valueType: avro: SensorData
  ksml_sensordata_sensor0:
     topic: ksml_sensordata_sensoro
     keyType: string
     valueType: avro:SensorDa'
  ksml_sensordata_sensor1:
     topic: ksml_sensordata_:
     keyType: string
     valueType: avro: SensorD:
  ksml_sensordata_sensor2:
      topic: ksml_sensordata_:
Filtering topic data
streams:
  ksml_sensordata_avro:
    topic: ksml_sensordata_avro
    keyType: string
    valueType: avro: SensorData
  ksml_sensordata_sensor0:
     topic: ksml_sensordata_sensoro
     keyType: string
     valueType: avro:SensorDa'
  ksml_sensordata_sensor1:
     topic: ksml_sensordata_:
     keyType: string
     valueType: avro: SensorData
  ksml_sensordata_sensor2:
      topic: ksml_sensordata_sensor2
version: "3.7"
services:
# Services can be reached from the local machine and in the docker network
# From the host machine:
# * Localhost port 9092 connects to the broker
# * Localhost port 8081 connects to the schema registry
# From the Docker network:
# * broker port 9093 connects to the broker
# * schema_registry port 8081 connects to the broker
networks:
ksml:    
  name: ksml_example    
  driver: bridge
docker-compose logs -f example-producer
example-producer_1 | 2021-05-11T09:55:00,169Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer - Using Kafka backendexample-producer_1 | 2021-05-11T09:55:00,445Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer - Producing: 0example-producer_1 | 2021-05-11T09:55:01,161Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Produced message totopic ksml_sensordata_avro partition 0 offset 0example-producer_1 | 2021-05-11T09:55:01,661Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Producing: 1example-producer_1 | 2021-05-11T09:55:01,667Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Produced message totopic ksml_sensordata_avro partition 0 offset 1example-producer_1 | 2021-05-11T09:55:02,167Z [system] [main]INFO i.a.k.e.producer.KSMLExampleProducer Producing: 2example-producer_1 | 2021-05-11T09:55:02,173Z [system] [main]
Examples/sensor-inspect.yaml
streams:    
  ksml_sensordata_avro:        
    topic: ksml_sensordata_avro        
    keyType: string        
    valueType: avro: SensorData
functions:    
  print_message:        
  type: forEach        
  code: "print('key='+(key if isinstance(key,str) else          str(key))+', value='+(value if isinstance(value,str) else str(value)))"

pipelines:
main:
Examples/ksml-runner.yml
ksml:
  workingDirectory: /ksml
  definitions:
  - sensor-inspect.yaml
‍
backend:
  type: kafka
  config:
    bootstrapUrl: broker:9093
    applicationId: io.ksml.example.processor
    schemaRegistryUrl: http://schema_registry:8081
    streamsConfig:
    acks: all
Dynamic topic routing
streams:
  ksml_sensordata_avro:
    topic: ksml_sensordata_avro
    keyType: string
    valueType: avro: SensorData
  ksml_sensordata_sensor0:
     topic: ksml_sensordata_sensoro
     keyType: string
     valueType: avro:SensorDa'
  ksml_sensordata_sensor1:
     topic: ksml_sensordata_:
     keyType: string
     valueType: avro: SensorD:
  ksml_sensordata_sensor2:
      topic: ksml_sensordata_:
Filtering topic data
streams:
  ksml_sensordata_avro:
    topic: ksml_sensordata_avro
    keyType: string
    valueType: avro: SensorData
  ksml_sensordata_sensor0:
     topic: ksml_sensordata_sensoro
     keyType: string
     valueType: avro:SensorDa'
  ksml_sensordata_sensor1:
     topic: ksml_sensordata_:
     keyType: string
     valueType: avro: SensorData
  ksml_sensordata_sensor2:
      topic: ksml_sensordata_sensor2
Success stories

How our customers use Axual

PostNL: Logistics

"KSML (Kafka Streams Markup Language): a low-code Python-based framework for processing and transforming Kafka streams. KSML handles transformation logic for parcel status events and enables real-time notifications to be sent to customers and delivery personnel."

Results:

  • 100+ million messages processed daily, with no performance degradation
  • Seamless real-time notifications triggered by parcel status changes
  • Rapid implementation timeline enabling faster time-to-value
  • Confidence in the scalability and reliability of Axual Cloud and KSML
Read more

Enexis: Energy

"The governance framework ensures every data access, schema change, and application integration leaves an audit trail meeting regulatory requirements for critical infrastructure operators."

Results:

  • 70+ data streams processing smart meter events, grid operations, and system integrations
  • 60+ applications consuming real-time meter data for billing, analytics, fraud detection, and grid management
  • 150+ developers collaborating on the platform, provisioning new streams through self-service
Read more

Ready to use KSML?

Contact us
Get started