Running Kafka command-line tools

Event Streams Authentication and Authorization

Kafka provides a rich set of command-line tools to manage the topics and clusters. The default settings of the latest IBM Event Streams (Strimzi Operator based) gives some challenges to run these tools.

Broker Listeners

When a Kafka resource is deployed as an operator, a strong security configuration is applied normally. Take a look at the following excerpt of the cluster listener settings

listeners:
  external:
    authentication:
      type: scram-sha-512
    type: route
  tls:
    authentication:
      type: tls

The brokers will be listening on

1. Port 9094 for external connections with SRAM-SHA-512 authentication.
2. Port 9093 for internal communication, where mTLS are used. The m here (mutual) means the client will authenticate the brokers, and the brokers will authenticate the client also.

As there are only these listeners defined, the commonly used plain port 9092 for the Kafka command line will not work.

KafkaUser Resource — Client Authentication and Authorization

As IBM event streams turn on the authorization, once a user is authenticated, it needs to be authorized to perform any action against the topic, consumer group, and the Kafka cluster. In the operator model, this is achieved with the KafkaUser resource.

Let's create the following KafkaUser resource

apiVersion: eventstreams.ibm.com/v1beta1
kind: KafkaUser
metadata:
  name: super-user-tls
  labels:
    eventstreams.ibm.com/cluster: es
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: '*'
          patternType: literal
        operation: All
      - resource:
          type: cluster
          name: '*'
          patternType: literal
        operation: All
      - resource:
          type: group
          name: '*'
          patternType: literal
        operation: All

There are two folds, one is authentication and the other one is authorization.