How to configure Kafka Authentication in Strimzi

At Axual, we deploy Kafka using Strimzi. On top we add a governance layer so customers can securely manage their own topics, schemas, and access. In the beginning, one Kafka cluster served a single tenant, so authentication was simple. That changed once we grew into a multi tenant cloud offering, where multiple customers share the same Kafka cluster. The challenge became: how do you configure authentication and authorization so that one tenant cannot access another tenant's resources? Today we run multiple multi-tenant clusters for customers in energy, banking, and government, many using their own PKI, each application connecting with its own certificate. The constant question is: who are you, and what are you actually able to do?

This post walks through Strimzi's authentication mechanisms, starting with the basics and mTLS, moving into our custom principal builder, covering the risks of leaning on Strimzi internals, and finishing with OAuth bearer authentication.

I also gave a talk on this at StrimziCon, covering the same subjects. In case you rather watch than read, watch  it here.

In this blog I will go over:

1. The foundation: authentication.type: tls
2. From single tenant to many CAs, and why the leaf DN alone isn't enough
3. The fix: SSL principal mappers give way to a custom KafkaPrincipalBuilder
4. Listener aware authentication: a configurable builder, and a listener per tenant
5. Risks and fragility: leaning on Strimzi internals
6. Switching tracks: the OAuth journey

1. The foundation: authentication.type: tls

This is a familiar setup to most people running Kafka on Strimzi. You configure mTLS by setting up a listener:

listeners:
  - name: tls
    port: 9092
    type: internal
    tls: true
    authentication:
      type: tls # client must present a cert

You see the mandatory fields here: port, type, TLS set to true so the broker also presents a server certificate, and a simple authentication.type: tls field. Strimzi takes care of the rest: it configures this listener to be mTLS, sets the keystore and truststore, and makes sure the correct principal builder is used.

What's good to know is that for anybody who authenticates with the Kafka cluster, the extracted principal is simply the distinguished name (DN) of their certificate. Meaning that this principal does not include any information about who issued it.

Kafka, through Strimzi, allows a couple of other listener types that are worth mentioning, next to mTLS.

SCRAM SHA 512 is a SASL mechanism, basically username and password but without sending the password over the line, using a challenge response mechanism instead. The user and its authentication are handled on the Kafka broker itself, and the principal is simply the username.

SASL OAUTHBEARER works with bearer tokens from an OIDC server. This used to be, and in the non v1 API versions of the Strimzi resources still is, an authentication type called oauth. That has now moved to type custom. This authentication type is quite useful because it lets you configure your listener however you like, as the name already suggests. What this gives you is authentication with tokens, with authorization based on one of the claims in that token, after all the mandatory checks like signatures have been done.

Strimzi also lets you define super users in your Kafka resource. These super users are added on top of the default set of super users that includes the brokers themselves, the cluster operator, and the entity operator (topic and user operator), among other Strimzi-managed components. These defaults are also distinguished names of certificates issued by Strimzi itself, based on the cluster CA, and still with no information on who issued them. Authentication works because Strimzi has an mTLS listener for these internal connections, on port 9090 for the control plane and 9091 for replication between brokers. They trust the cluster CA cert, so anything signed with it is authenticated and anything not signed with it is not. But once through the authentication layer, authorization happens again only on the distinguished name of the certificate.

2. From single tenant to many CAs, and why the leaf DN alone isn't enough

At some point we had to move from a single tenant to a multi tenant cluster. Since many tenants come with their own CA, we have to trust those CAs, defining an mTLS listener that trusts all of them at once.

We solved this by making use of the clients-ca-cert, used by Strimzi. The clients-ca-cert secret is usually generated by Strimzi if you don't provide extra configuration, but you can create it yourself, with a few labels and annotations Strimzi expects. That secret holds a ca.crt file, the base or main CA, PEM formatted. Next to that we added a customerA.crt and a customerB.crt, both CA certificates. This created a clients-ca-cert secret with all those CAs in it. Strimzi parses all of these into one truststore, so a single mTLS listener trusts all the certificates, and every tenant can issue their own certificate and be authenticated and authorized.

In any test setup, this is easy to trip over. Two customers who sign their own certificates can sign certificates with access to each other's resources. Worse, a customer could sign a certificate with the same distinguished name as, say, the cluster operator, and, signed by their own CA, get authenticated on the cluster and get superuser permissions. Which is not done.

Two different CAs, Customer A's and Customer B's, can each sign a leaf certificate with the same common name, for example CN=payments-app. Both CAs are trusted, so both leaves authenticate, and authorization only sees CN=payments-app. It cannot tell which tenant actually signed it, so Customer B could end up impersonating Customer A's ACLs.

3. The fix: SSL principal mappers give way to a custom KafkaPrincipalBuilder

Enter our custom principal builder. When the multiple CA situation arrived alongside our thinking about the principal builder, we built a builder that parses the context, gets the certificate chain, and encodes the whole chain as the principal. It starts from whoever issued it as the root, through one or more intermediates, ending with the leaf certificate and its distinguished name.

[0] CN=Axual Dummy Root 2018,
[1] CN=Axual Dummy Intermediate 2018 01,
[2] CN=app-one

These certificates carry more than just a common name field. They can include organization and organizational unit too, kept short here for clarity. Authorization is done on the full chain. This makes it easy to tell tenants apart, since they usually have their name in their root or intermediate certificate, and it prevents anybody from becoming superuser or reaching another tenant's resources, since colliding leaves now differ by their issuing chain.

This came with a small consequence. Since Strimzi issued certificates also have a certificate chain, our superuser list needed updating, replacing each default leaf entry with its full chain equivalent.

# Before, leaf DN only
super.users=User:CN=cluster-operator,O=io.strimzi

# After, the operator's full chain, or it stops matching
super.users=User:[0] CN=cluster-ca v0,O=io.strimzi, \
  [1] CN=cluster-operator,O=io.strimzi

This is custom, and slightly prone to issues, covered below.

How the builder ships. For the Strimzi images, we build from the Strimzi base image (from Quay), add our principal builder JAR files (the builder itself plus utilities), and put them directly in the Kafka libs folder, no classpath changes needed. You can use it immediately. But it has maintenance overhead: every Strimzi release means a new image for every Kafka version in that operator version, and whenever Strimzi retags an image, you have to rebuild.

There's another way. Kubernetes has image volume functionality, which makes this a lot easier.

pod:
  volumes:
    - image:
        reference: principal-builder:2
        pullPolicy: IfNotPresent
      name: principal-builder
kafkaContainer:
  volumeMounts:
    - name: principal-builder
      mountPath: /mnt/axual-principal-builder
  env:
    - name: CLASSPATH
      value: /mnt/axual-principal-builder/jars/*

You mount the principal builder image and use the JAR files inside it. Build it from scratch with just those JAR files, point the classpath at the mounted JARs, and the builder is now versioned and shipped independently of the underlying Kafka image.

4. Listener aware authentication: a configurable builder, and a listener per tenant

You can also configure a custom listener per tenant. In this setup, a truststore location is scoped to a single tenant. The important fields are the truststore type, truststore location, ssl.client.auth (to keep mTLS required), and the principal builder class.

- name: tenanta
  type: custom
  tls: true
  authentication:
    type: custom
    listenerConfig:
      ssl.truststore.location: /mnt/axual/tenanta.pem
      ssl.truststore.type: PEM
      ssl.client.auth: required
      principal.builder.class:
        io.axual.security.auth.SslPrincipalBuilder

This has several advantages. Only one tenant can authenticate on this listener, since everyone else fails an SSL check as untrusted. The principal builder now lives on the listener, so there's no need to touch the superuser list when certificate formats or chain lengths change for Strimzi's own certificates. And clients-ca-cert can revert to its default form, holding just our own Axual root CA, with tenant specific CAs defined per tenant elsewhere.

One thing to watch: a public TLS listener has no per-listener builder by default. If it should produce chain principals too, you must set the builder there explicitly, or it silently falls back to leaf DNs.

Making the builder itself configurable. The SSL principal builder up to this point is static. Like many Kafka classes, it can be made configurable, since any configuration fed to a listener is passed into the principal builder through Kafka's Configurable interface.

public class SslPrincipalBuilder
    implements KafkaPrincipalBuilder, Configurable {
  @Override
  public void configure(Map<String, ?> configs) {
    // every listenerConfig entry lands here
  }
}

You can specify listener behavior through extra listenerConfig, for example setting tenant: "tenant-a". Kafka namespaces the key by listener name, so it arrives in configure() as something like listener.name.tenanta-9095.tenant=tenant-a. From that, the builder can reshape the principal, for instance prefixing it with the tenant name, so there's never a mixture of authorizations, as long as tenant names and prefixes are unique.

This brings things full circle: one principal builder defined in the generic Kafka config, with behavior chosen by configuration rather than hard coded per listener. The one rule to honor is never remapping Strimzi's own internal listeners, control plane on 9090 and replication on 9091. The builder checks the listener name and falls back to Strimzi's default mapping for those, while every other listener gets the custom principal logic.

5. Risks and fragility: leaning on Strimzi internals

Two of these techniques lean on Strimzi internals, and when those internals move, things can break quietly.

Risk one: the truststore patch is a hack, not an API. We create the clients-ca-cert secret ourselves and add extra CAs next to the ca.crt entry, which isn't officially supported. It works only because Strimzi happens to parse every certificate it finds there and builds a trust bundle carrying all those entries through. The risk is that this behavior is incidental, not promised. A future release could stop honoring the extra entries at any time. How we live with it: pin the Strimzi version, re-verify the generated trust bundle on every upgrade, and watch it for drift before it reaches production.

Risk two: it already happened, in Strimzi 0.49.0. Support for PKCS12 keystore files was dropped in favor of PEM files, a good change, but as a side effect the broker certificates that used to be chained (root, intermediate, leaf) came through as just intermediate and leaf, dropping the root. In some test environments, after upgrading to 0.49.0, the shorter chain no longer matched the full chain string configured in super.users. The broker could still authenticate, since the intermediate was still trusted, but it couldn't authorize, because the superuser entry expected the longer chain. This was fixed in 0.50.x and 0.51.0, where broker certificates again appended the entire chain using PEM files. We jumped straight from before 0.49 to 0.51, which went smoothly. The broader point: a format change two layers down invalidated the broker's own identity. Override the principal builder, and you own that fragility on every upgrade.

6. Switching tracks: the OAuth journey

Moving from mTLS to OAuth bearer is another way of configuring authentication in Strimzi and in Kafka generally. We set up a Kafka cluster with Keycloak as the OIDC server, again using authentication.type: custom.

authentication:
  type: custom
  sasl: true
  listenerConfig:
    sasl.enabled.mechanisms: OAUTHBEARER
    oauthbearer.sasl.server.callback.handler.class: io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler
    oauthbearer.sasl.jaas.config: |
      org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
      oauth.valid.issuer.uri="http://keycloak:8080/realms/strimzioauth" \
      oauth.jwks.endpoint.uri="http://keycloak:8080/realms/strimzioauth/protocol/openid-connect/certs" \
      oauth.username.claim="client_id";

You need to configure the SASL mechanism, the callback handler, the JAAS config, which claim becomes the username, where to verify signatures, where the keys live, the valid issuer, and audience checks, everything needed to validate the token properly.

A sharp edge here: when a customer owns the OIDC server, they decide what the identity is. If a client's OIDC server is configured so the username claim contains a string matching a Kafka superuser, the token still validates fully, since signature, issuer, and audience all check out, and the principal built from that claim becomes a superuser. Authentication sits on the client's side. Authorization needs to stay on ours.

Keeping control. The Strimzi kafka-oauth-server library includes an option to prefix the username. The result is that whatever claim is used, the prefix is applied before the principal is built.

oauthbearer.sasl.jaas.config
  …OAuthBearerLoginModule required \
  oauth.username.claim="client_id" \
  oauth.username.prefix="tenant-a:";

client_id: "cluster-operator" → client_id: "tenant-a:cluster-operator"

This mirrors the custom mTLS listener pattern from earlier, where the principal is prefixed with the tenant. The result is a unique principal that, done correctly, will never be a superuser in the cluster.

mTLS as a gate, OAuth for identity. Some customers prefer mTLS, or are wary of SCRAM, or can't configure OAuth bearer easily, but their own certificate issuance isn't well matured and common names can be duplicated. For that case, mTLS can sit on top of an OAuth bearer listener using the custom authentication type.

type: custom
tls: true
authentication:
  type: custom
  listenerConfig:
    ssl.truststore.location: /mnt/axual/tenanta.pem
    ssl.truststore.type: PEM
    ssl.client.auth: required
    sasl.enabled.mechanisms: OAUTHBEARER

Anybody connecting has to present a client certificate, with the truststore holding that certificate's CA, and also send a token. Authentication happens on both mechanisms, but authorization happens only on the token. Once the mTLS gate is passed, the flow continues into the OAuth token exchange. This gives customers confidence that mTLS is in place, while duplicate common names are never a problem, since the certificate only confirms identity and the token is what actually gets authorized. The same layering works for SASL SCRAM listeners, adding an mTLS config on top of the SCRAM mechanism to gate those as well.

Where we landed

A quick recap of the ground covered here.

The mTLS journey: trusting every tenant's CA, moving to full chain principals, and landing on hard per tenant isolation through a listener per tenant.

The principal builder, up close: bundled into the image today, moving toward configurable and image volume shipped tomorrow.

Risks and fragility: knowing which parts ride on undocumented Strimzi behavior, and testing for it on every upgrade.

The OAuth journey: token identity, who really controls it, prefixes to keep authorization on our side, and mTLS layered with OAuth.

There will always be some risk. Responsibility sits on both sides, and it pays to be clear about who owns which part. Identity first, authorization second, and know which parts lean on Strimzi internals.

I hope this helped you learn something and that you can actually apply it in your own work processes. Also, I recently wrote a technical overview of the Strimzi 1.0.0 CRD migration path, if you want to check it out. Let me know if you have any questions or suggestions, you can email me at daniel.mulder@axual.com. I'd love to hear what other folks are doing.

‍

‍

‍