What is Kafka Consumer Concurrency?
Ron Shepard
Last updated on
Your thoughts?
Share your thoughts
A short but loaded question :)
In general, concurrency is the ability to perform parallel processing with no affect on the end result.
In Kafka, the parallel consumption of messages is achieved through consumer groups where individual consumers read from a given topic/partitions in parallel. This works because each consumer in the instance is assigned a given number of partitions within the topic, allowing all messages to be read in parallel.
This is a key advantage of Kafka and makes it superior in terms of processing power on the consumer side.
Concurrency is the ability for processing to happen in parallel.
Consumer concurrency is the ability for a consumer group to process messages from a topic in parallel. This is achieved through partitions as each consumer in a consumer group is responsible for reading n number of partitions.
While each consumer in a group operates on a single thread, collectively the group of consumers utilizes multiple threads to consume messages from Kafka.
Concurrency can also apply to how records are processed at the individual consumer level. While each consumer is single threaded, the processing of records can use multiple threads. This presents a more advanced use case and requires adequate knowledge of thread management in Java.