How you scale infrastructure with distributed messaging ?

We use snapchat, whatsapp, sms, email to send text message to each other. Same as server can send and receive messages. Distributed messaging is servers communicate with each other via messages. Messages can be in any format like text, binary, xml, json etc. It is generally in format that other server can understand it. Servers generally follow some protocol(some specific format). Protocol is manual of messages which describes in which format and how servers will communicate.

  • Applications
    • Inter server communication
      • This is to communicate between more than one server. Examples RabbitMQ, ActiveMQ, ZeroMQ.
    • Lazy execution or Task workers
      • Server sends message to different server to execute some code base and later that server executes them. Different programming languages have different implementation. Examples are Python Celery, Python Gearman, Java quartz, Ruby sidekiq, Golang gocelery Clojure backtick etc.
    • Distributed deployments
    • Management and Monitoring

Distributed messaging can be done in two ways  one-to-one or one-to-many. It can be divided into two types of messaging

  1. Broker equipped
  2. Broker less

Broker is individual who helps us to buy or sell virtual or physical things and takes commission from it. Brokers are generally between two parties. If I’m selling something then they’ll understand my requirement and will redirect me to proper party and broker will be between parties. If dispute occurs then they help us to resolve it.

Broker equipped architecture works same as brokers. Broker is single server which sits between multiple servers. If some server wants to communicate with other server then that server will tell this message to broker and later broker distributes this message. Same applies for other servers. Advantage of broker equipped architecture is that everybody communicates via broker so, messaging is easy. Everything can be found on single server. Broker equipped architecture has drawback that if broker fail then whole infrastructure will suffer.


Broker less messaging has no broker. Server communicate with their choice of server. Each server acts like boss. Anybody can communicate with each other. Advantage of this kind of architecture is that if single server fail then infrastructure will not suffer. It has disadvantage that server communicate to other server without any control.

There are many open source implementation available to implement broker equipped architecture.

Messaging queue characteristics

  • Delivery
    • Garranties delivery of message.
  • Transactions
    • Messages can be grouped together such that either none or all of the messages in the transaction would be processed by the server in case of failure.
  • Messaging model
    • Models are simple distributed queue, publish subscribe and message will be deleted once received by client.
  • Durability
    • If broker dies then also message should be persisted into disk.
  • Synchronous & Asynchronous message
    • Synchronous messaging is once client submited message to broker then it will wait for response from broker and Asynchronous messaging is client will not wait after message published.
  • High Availability
    • If broker dies then there should be new broker who will be elected as new broker.
  • Load balance
    • Broker can be load balanced between multiple nodes to share load. Example: If we have 3 clusters A, B, C. Client is sending messages on cluster A. There are some consumers connected on client B and C. Broker can load balance messages between B and C.


RabbitMQ is popular for building broker equipped architecture. It is built on top of Erlang. Erlang has concurrency support built in. It implements concurrency by Actor Model. RabbitMQ has really matured broker implementation. It is available since 10 years. It supports multiple communication protocols. It has support for Java, .NET, Ruby, Python, PHP, Golang, Scala, Groovy and Grails, Clojure, Javascript, C/C++, Erlang, Haskell, OCaml, Swift, Perl. Even it can be used inside databases Oracle, SQL Server, Postgresql, Riak via extensions. It can be extended by extensions. For an example want to build websocket support on top of RabbitMQ can be done via plugins. RabbitMQ can handle large amount of data at a time. It is really light weight. Documentation is really good. It is really easy to get started with RabbitMQ. Go through docs. It supports AMQP protocol and other protocols can be supported via plugins.

RabbitMQ has single server and multiple clients. Clients communicates with Server via broker in multiple ways. Client sends message to RabbitMQ Broker and RabbitMQ broker later sends it to appropriate client. RabbitMQ has built in Queue support. If client sends message to server and if other client is not available then it stores that message in it’s disk storage. Once client is available then it delivers message to client and deletes message from its storage. RabbitMQ has broker in between so it comes under broker equipped. It supports queue, pub-sub, Message routing based on pattern, Topic publish subscribe, RPC.

All the queue data can be mirrored to other RabbitMQ servers. But, they are only for availability not for load balancing. All consumers will be connected to master node only. RabbitMQ can be extended by plugins. It should never be used between WAN.

As per characteristics it supports Delivery, Transactions at single message level, distributed queue, pub-sub, durability, synchronized and asynchronized messages, high availability. It does not support load balance.


ZeroMQ can be used to build both broker equipped and broker less architecture. It is implemented in C++. Under the hood it uses libzmq. It supports C, C++, C#, Common Lisp, D, Erlang, Go, Haskell, Java, Lua, Node.js, Perl, PHP, Python, Racket, Ruby, Swift, TCL. ZeroMQ uses BSD sockets. It’s documentation is good but whoever is using this library he/she needs to have good understand of distributed computing and should handle scenarios which can occur in distributed computing. It can be used for inter-process communications on single machine. It can be used to build variety of architectures. It can mix both broker and broker less architecture. It can be used to build all the things which I mentioned in RabbitMQ but, caveat is developer should know distributed computing.

Distributed programs can be written in a way which can support any characteristics.


ActiveMQ is built on top of Java. It supports .NET, C, C++, Erlang, Golang, Haskell, Haxe, Node.js, Perl, Python, Ruby, Java, Julia, Kotlin, Lisp, Lua, TCL, Scala, Rust. It supports STOMP, OpenWire and AMQP protocols so, any programming language which has this protocol implementation can be used with ActiveMQ. ActiveMQ is similar to RabbitMQ. ActiveMQ can have backup server which will backup live server data. In case of failure Backup server will be elected as live server and once crashed server comes back that server will be given high priority to become master in case of another failure. ActiveMQ uses libaio internally. If you are using Java then ActiveMQ has single JAR file dependency. It supports queue, publish subscribe, queue routing, message filtering, Last-Value queue, message groups, duplicate message detection. ActiveMQ supports throttling. This parameter can be tuned to limit number of messages will be transferred between client and server. Messages can be sent with TTL(Time To Live). Once TTL reached message will not be delivered. It can be used to send large messages or files. Messages can be scheduled to delivered at specified time. ActiveMQ has broker equipped architecture. ActiveMQ can be extended by write plugins. It has support for JMS libraries.

As per characteristics it supports Delivery, Transactions at single message level, distributed queue, pub-sub, durability, synchronized and asynchronized messages, high availability. It does not support load balance.

Apache Apollo

Apache Apollo built from the foundation of ActiveMQ. It uses different approach. It started with experiment to see what can be done on top of ActiveMQ to work better with machines with higher cores. ActiveMQ has complex threading implementation and that is why it is not able to work better on multi processor systems. Experiment resulted in more deterministic, stable and salable version of ActiveMQ. It is built on top of Scala, written from scratch and supports REST based management. It uses Reactor_pattern and message swapping. It supports new protocol MQTT. MQTT is standard protocol for IOT(Internet Of Things). RabbitMQ can be extended with available MQTT plugin. It supports Queue, Last published message, Message TTL, Topic Pub-Sub, Sequence numbers to messages, Message groups, Message routing, Message filter. For persistent storage it uses Google’s leveldb or Java  edition of Berkeley DB. It’s data can be replicated to other servers with Zookeeper and LevelDB.

As per characteristics it supports Delivery, Transactions at single message level, distributed queue, pub-sub, durability, synchronized and asynchronized messages, high availability. It does not support load balance.


HornetQ is written in Java. It has support for JMS libraries. HornetQ claims that their performance is at rates normally seen for non persistent messages.  It uses libaio internally. It built on basis of JBoss Messaging 2.0. It can be used as Enterprise Message Broker. It can be used independently of JBoss Application server. It can do load balance between multiple servers. It supports throttling, Schedule messages, Message routing, Message divert, TTL message, Last value queue, message priority, HA, data replication, message grouping. For storage it uses own implementation.

As per characteristics it supports Delivery, Transactions at single message level, distributed queue, pub-sub, durability, synchronized and asynchronized messages, high availability, load balance.


We can help you to scale your infrastructure.

Contact us

Leave a Reply

Your email address will not be published. Required fields are marked *


DBA, DevOps, Big data, Data Analyst