Using both, request-reply and pub-sub for microservices communication

Question

We are planning to introduce both, pub-sub and request-reply communication models to our micriservices architecture. Both communication models are needed.

One of the solutions could be using RabbitMQ as it can provide both models and provide HA, clusterring ang other interesting features.

RabbitMQ request-reply model requires using queues, both for input and for output messages. Only one service can read from the input queue and this increases coupling.

Is there any other recommended solution for using both request-reply and pub-sub communication models in the same system? Does service mesh could be a better option?

It shall be suppoered by node.js, python and. Net CORE.

Thank you for your help

Answer 1

There multiple pub-sub and request-reply support HA communication models :

1. Kafka

Kafka relies heavily on the filesystem for storing and caching messages. All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means that it is transferred into the kernel’s pagecache.

Kafka is designed with failure in mind. At some point in time, web communications or storage resources fail. When a broker goes offline, one of the replicas becomes the new leader for the partition. When the broker comes back online, it has no leader partitions. Kafka keeps track of which machine is configured to be the leader. Once the original broker is back up and in a good state, Kafka restores the information it missed in the interim and makes it the partition leader once more.

See :

• https://kafka.apache.org/
• https://docs.cloudera.com/documentation/kafka/latest/topics/kafka_ha.html
• https://docs.confluent.io/4.1.2/installation/docker/docs/tutorials/clustered-deployment.html

2. Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

See :

• https://redis.io/
• https://redislabs.com/redis-enterprise/technology/highly-available-redis/
• https://redis.io/topics/sentinel

3. ZeroMQ

ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems.

See :

• https://zeromq.org/
• http://zguide.zeromq.org/pdf-c:chapter3
• http://zguide.zeromq.org/pdf-c:chapter4

4. RabbitMQ

RabbitMQ is lightweight and easy to deploy on premises and in the cloud. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

Answer 2

• My preference would be to have REST api for request-reply pattern. This is specially applicable for internal microservices where you are in control of communication mechanism. I don't understand your comment about why they are not scalable if you defined them as properly and you can scale out and down the number of instances for the services based on demand. Be it Kafka, RabbitMQ, or any other broker, I don't think they are developed for request-reply as primary use case. And don't forget that whatever broker you are using, if it is A->B->C in REST, it will be A->broker->B->broker->C->broker->A and broker need to do it house keeping.

• Then for pub-sub, I would use Kafka as it is unified model which can support pub-sub as well as point to point.

• But if you still wanted to use a broker for request-reply, I would check Kafka as it can scale massively via partitions and lot of near real streaming applications are built using that. So It could be near the minimal latency requirement of request-reply pattern. But then I would want a framework on top of that to associate request and replies. So I would consider using Spring Kafka to achieve that