Kafka Consumer groups and Lag

When kafka consumers are subscribing to a topic, as and when messages are produced, they are consumed by the consumers. But due to high traffic or latency or high load issues, there would be a lag in consuming the messages.

With Kafka binaries it is possible to know the lag of every consumer group, and per topic even.

Display all consumer groups

/bin/kafka-consumer-groups --bootstrap-server localhost:9092  --list

Describe a consumer group

/bin/kafka-consumer-groups --bootstrap-server localhost:9092 --group consumergroup  --describe

This will display all the connected consumer clients, subscribing to each topic, and the current offset and lag. Sample output below.

GROUP           TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                            HOST            CLIENT-ID
 consumergroup         testtopic           0          376             376             0               consumer-consumergroup-1-e2e668bd-a2cf-4dca-b27c-68f6d89f6859 /10.195.50.316   consumer-consumergroup-1
 consumergroup         testtopic           1          364             374             10              consumer-consumergroup-1-e2e668bd-a2cf-4dca-b27c-68f6d89f6859 /10.195.50.316   consumer-consumergroup-2

Describe a consumer group with SSL

/bin/kafka-consumer-groups --bootstrap-server localhost:9093 --group consumergroup --describe --command-config ssl.properties