Is there a way to set up Kafka ACL to allow using any consumer group without listing them

Question

I am trying to set up Kafka, where each user have several topics, but each topic may be consumed with any number of consumer group by the user the topic belongs to.

Kafka server version used: kafka_2.12-2.4.0 (Commit:77a89fcf8d7fa018)

Kafka client version used: confluent kafka 1.2.2

In Kafka ACL have successfully configured users, so they can only access only their own topic. I'm struggling to set up group permissions in such a way where each user can use any number of consumer groups just for their own topic without seeing what consumer groups others have.

The following enables every user to use any consumer group:

bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --allow-principal User:* --operation Read --group '*'

However, according to https://docs.confluent.io/current/kafka/authorization.html Read operation implicitly grants Describe operation. As Describe operation includes access to 'ListGroup' API, which I do not want my users to be able to do, I executed the following:

bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --deny-principal User:* --operation Describe --group '*'

The two commands above result in the following ACLs:

Current ACLs for resource `Group:LITERAL:*`:
        User:* has Deny permission for operations: Describe from hosts: *
        User:* has Allow permission for operations: Read from hosts: *

The problem with this is I'm getting the following exception:

Confluent.Kafka.ConsumeException: Broker: Group authorization failed

Which leads me to believe I'm either trying to achieve the impossible or trying it wrong.

TLDR: Is it possible to set up Kafka ACLs to allow using any consumer group without also granting ListGroups API permission at the same time?

Thanks for any answer.

Answer 1

For now decided to use prefix. Works well enough. For those wondering how to do this:

bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --allow-principal User:XYZ --operation Read --group 'ABC-' --resource-pattern-type prefixed 

This piece of code will allow user 'XYZ' to use any consumer group starting with 'ABC-', like 'ABC-123'