I’m working on a Kafka consumer application using Akka Kafka connector. I would like the consumer to process messages parallelly. which consumer group should I choose Choosing a consumer? how can I configure the parallelism on the consumer side?
Akka streams Kafka consumer process parallel
863 Views Asked by vkt At
1
There are 1 best solutions below
Related Questions in APACHE-KAFKA
- Spark streaming + kafka throughput
- How to diagnose Kafka topics failing globally to be found
- kafka: what do 'soTimeout', 'bufferSize' and 'minBytes' mean for SimpleConsumer?
- Fail to create SparkContext
- Syntax error on tokens, delete these tokens - kafka spring integration demo application
- How could Kafka 0.8.2.1 with offsets.storage=kafka still require ZooKeeper?
- Message Queues: Per Message Guarantees
- How should a Kafka HLC figure out the # of partitions for a topic?
- Kafka multiple consumers for a partition
- Should Apache Kafka and Hadoop be installed seperatedly (on a diffrent cluster)?
- how does one combine kafka-node producer and node tail?
- How to fix NoClassDefFoundError with custom Kafka producer under Eclipse?
- Apache Samza's CheckpointTool won't give away partition offsets
- Offsets for Kafka Direct Approach in Spark 1.3.1
- Simulate kafka broker failures in multi node kafka cluster and what operations and tools to use to mitigate data loss issues
Related Questions in AKKA
- Akka actor invoked with a function delegate - is this bad practice?
- Akka supervisor on stop of subordinate
- Scala - TCP Packet frame using Akka
- Akka-http process requests with Stream
- Multiple dispatcher for spray
- Spray Dead Letter msg
- ActorNotFound Exception trying to run Spark 1.3.1 on windows 7
- What's the best way to get members of an Akka cluster?
- Is it possible to await for second response in Scala
- How to keep Akka running all the time in Play 2.3
- Suspending AKKA actor
- Spray Client (i.e. AKKA) within Tomcat, cause Threading issue
- Can I use Akka scheduler to schedule multiple things from a single actor context?
- akka: pattern for combining messages from multiple children
- Akka: multiple dispatcher vs multiple actor system
Related Questions in AKKA-STREAM
- Akka-http process requests with Stream
- Accessing the underlying ActorRef of an akka stream Source created by Source.actorRef
- Exceeded configured max-open-requests
- Akka Stream OnNext is not allowed
- How to respond with the result of an actor call?
- How to create a Source that can receive elements later via a method call?
- How to print dropped elements in a stream with OverflowStrategy?
- Can I implement my own OverflowStrategy?
- Live resources in Akka Stream flow description
- Collecting sink materialized values as source
- Akka Streams flow to handle paginated results doesn't complete
- How to make htttp request with akka stream for 10K request
- How do I create a Sink to read all bytes from the Source?
- Allow single connection with Akka stream as tcp server
- Elegant way of reusing akka-stream flows
Related Questions in ALPAKKA
- Save DTO in Alpakka Cassandra
- Play Framework 2.6 Alpakka S3 File Upload
- Why don't Akka Streams application terminate normally?
- can't get akka streams / alpakka S3 to work in simple case
- Why does my Alpakka SFTP connector never connect?
- How to log Kafka Producer Config on Startup
- Kafka does not reconsume messages after group-id has changed and offset.reset is set to earliest
- trying to convert java file to bytestring for using in stream
- Akka Streams buffer on SubFlows based on parent Flow
- Aplakka scala s3 connector hangs when trying to put data
- How is fetch size controlled in akka stream alpakka slick
- Slick result set size default limit
- Alpakka s3 `multipartUpload` doesn't upload files
- Exception in Alpakka JsonReader substream not being caught by supervisor
- alpakka cassandrasource read data from cassandra continuously
Related Questions in AKKA-KAFKA
- Resetting offset for partition after (Re-)joining group
- Is this Akka Kafka Stream configuration benefits from Back Pressure mechanism of the Akka Streams?
- avro4s can not deserialize AnyRef
- How Akka stream internal and explicit buffer interact with the underlying kafka client settings in alpakka Kafka?
- Akka Kafka stream supervison strategy not working
- Fast Processing Topic and Slow Processing Topic - Akka Kafka
- Reactive-Kafka Stream Consumer: Dead letters occured
- How to group messages so that they go to the same partition every time?
- Akka streams Kafka consumer process parallel
- Incompatible equality constraint while using Akka Kafka Streams
- What is a good pattern for committing Kafka consumer offset after processing message?
- How do you return a future containing a list of messages after all available messages have been consumed from a Kafka topic?
- Getting last message from kafka Topic using akka-stream-kafka when connecting with websocket
- Troubles with AVRO schema update
- Kafka producer with adjustable amount of messages per second
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
There are different levels of "parallelism" when you talk about Kafka and then akka-stream.
Kafka processing can be paralleled by introduction of multiple partitions in a topic and having a single Kafka consumer consuming from a singe partition all of the messages in a sequential order, ie messages in the order of their sequence id or offset. This allows for multiple consumers consuming from multiple partitions simultaneously.
Akka streams has a very sequential approach and it does processing in a one by one manner by default. But this of course depends on your interconnected flows. I strongly suggest reading this to understand how threading and concurrency are used in akka-stream.
The answer to your question is that parallelism depends on all of these aspects and not on choosing the correct akka stream "Consumer" (I would call it a Source in akka stream terms), which mostly defines access to offset that can be committed after processing.