Communication security, authentication, authorization and message security. There are a lot of security settings you need to configure when working with Apache Kafka. In this blog post, Richard explains the four key security subjects.

Apache Kafka is a very powerful and flexible platform for event streaming. The use of event streaming often grows to every corner of the organization, and this means that sooner or later it will handle event data that is considered sensitive or confidential. This is where the security setup of our deployment becomes very important.
Kafka offers many configuration options regarding security, at the cluster and topic level.
But with all these options of Kafka, the question that we need to ask ourselves is:
did I miss something?
The generic nature of the question brings another fact into focus. Security itself is a generalization. It consists of very different subjects that handle different aspects of security. To make sure that I capture most requirements for securing a Kafka Cluster I usually split it into 4 different subjects.

Communication Security is the part of security I usually start with because it’s a fairly standard and straightforward subject with Kafka.
What do I mean with communication security?
Simply put, it’s the way how the communication between a client application and the Kafka cluster is set up and secured. If the event data is sensitive then we don’t want every system that’s on the same network as the cluster or client to be able to capture the data while it’s in transit. Kafka supports Transport Layer Security, or TLS, to make sure that the data that is transmitted is encrypted, and that the client applications can verify that nobody is impersonating the server.
There are several versions of TLS, and Kafka supports the latest version, 1.3, since Kafka 2.5.0 Kafka allows us to specify which TLS versions are supported, and which TLS version is used by default. We can also specify which cipher suites are supported, allowing us to use TLS settings that match the TLS capabilities of the client applications.
Authentication in Kafka is about determining the identity of connecting clients. Kafka has several ways of authenticating clients.
TLS Client Authentication, or Mutual TLS, can be activated as part of the TLS configuration. This requires connecting clients to provide a certificate signed by a certificate authority that is trusted by the Kafka cluster. Kafka can extract the distinguished name or parts of it from your certificate to function as the principal name for the authorization phase
Kafka can also be configured to use SASL, or Simple Authentication and Security Layer. This is a standard framework used for authentication for internet protocols. TLS can be activated on top of SASL to make sure that the data cannot be read by other systems on the network. The Kafka SASL implementation supports several mechanisms for authentication, which can be enabled in parallel:
The Kafka SASL implementation supports customising or externalisation of these mechanisms. The libraries with the new implementations need to be in the Kafka classpath before setting the callback handler configurations.
Authorization is the part of security about determining which operations a connecting client is allowed to perform and which operations are denied.
Kafka allows customisation of this logic by setting the authorizer class name in the configuration. The default implementation is provided by the AclAuthorizer class.
This class is based on Access Control Lists. An entry, or rule, consists of 7 parts:
The default ACL approach gives us a good way to get a fine grained access control. Don’t forget to disable the cluster configuration that allows everyone if no acls are found for a resource.
And don’t forget that the brokers need specific access entries as well, or be set to super user in the cluster configuration
The fourth and final subject I examine is message security.
I usually handle three parts of message security.
1. Access to the data as it’s stored on the brokers
2. Producer identity
3. Integrity check of record content
Kafka persists the record data it has received to disk. This means that anyone with access to the disk could read the data. Kafka itself doesn’t offer encrypted storage, but relies on the operating system to encrypt the storage drive if needed. Most of the operating systems or cloud providers have their own implementations to provide this functionality, and it is a good practice to investigate the advantages and disadvantages of each of the implementations.
The second and third part of message security, determining producer identity and integrity checking the content of a record can be important to some organisations. A common approach is by using message signatures, which use a producer specific key to generate a signature which is attached to the message. The consumer can verify that the contents of the record has not changed by using that information.
Kafka does not support this out of the box. If this is needed you need to develop or deploy third party implementations that can add the relevant information to the produced record and verify the message content and signature at the consumer side.
As a final note, Kafka supports multiple listeners.
A listener basically listens on a specific port, and each listener can have its own configuration regarding TLS and Authentication. This allows us to create a listener for external networks with more different TLS and Authentication options, while also having a separate listener for internal networks which can use internal authentication mechanisms. The authorisation model does not change per listener.
These are the security options of Kafka in a nutshell.
If you’re struggling with setting up security for Apache Kafka, feel free to reach out to us – we’re here to help.
