Kafka Consumer Lag Due to File Descriptor Exhaustion

intermediate

linuxkafka

18 min

100%

Kafka Cluster Service Topology

⚙️

Prometheus + Grafana

service

⚙️

Order Service (Producer)

service

📨

orders.placed

queue

⚙️

kafka-broker-1

service

⚙️

kafka-broker-2

service

⚙️

kafka-broker-3

service

⚙️

order-processing-cg (Consumer)

service

🗄️

orders-db

database

healthy

degraded

failed

unreachable

Kafka CLI

$

0/1 commands

context

PagerDuty pages the on-call at 14:32 UTC. Consumer lag on orders.placed has been above the 500,000-message threshold on every partition for 30 seconds. In Grafana, kafka_exporter shows order-processing-cg with 0 members, and the group's lag has climbed in a straight line since about 14:06.

explanation

order-processing-cg is on the checkout critical path: each message on orders.placed is an order line waiting for fulfillment, and nothing is consuming them. The group is 3.1 million messages behind, more than 522K on every partition: at ~2,000 msg/s, that's about 26 minutes of production. Checkout order processing has stopped, so the on-call declares SEV-1 at 14:33.

Engineer Thought

“Okay, this is bad. 3.1 million messages behind, more than 522K on every partition, and Kafka says the group has no active members. This isn't a slow consumer. Nothing has consumed for about 26 minutes, so something took out all three consumers at once. I'm declaring SEV-1.”

Concepts

Consumer Group State: Empty

A Kafka consumer group enters the 'Empty' state when all consumers have left the group. This means no consumer instances are actively polling for records, causing lag to accumulate indefinitely.

Instructor Notes (1)

mindset+45s

An Empty group with growing lag means nothing is consuming at all, which is a different problem from slow consumption. Before restarting anything, find out why every member left at once. Crashed or killed consumers, a deployment scaled to zero and members dropped for exceeding max.poll.interval.ms are all common causes. So is a dependency that every consumer shares.