Is it a good idea to collect data from devices into a message queue?

Question

Let's say you have 1000's of devices all sending data in all the time, would a message queue be a good data collection tool for this data?

Answer 1

Obviously, it depends:

• Do you need to process every set of data, regardless of how old it is?
• Will data arive in a stead stream, or be bursty?
• Can a single application process all the data, or will it need to be load balanced?
• Do you have any need for the "messaging" features, such as topics?
• If your devices would be the clients, is there a client implementation that works on them?

If you have a stream of data that can be processed by a single application, and you are tolerant of occasional lost data, I would keep it simple and post the data via REST or equivalent. I would only look to messaging once you needed either scalability, durability, fault tolerance, or the ability to level out load of time.

Answer 2

You can't go wrong design-wise with employing queues, but as Chris (other responder) stated, it may not be worth your effort in terms of infrastructure as web servers are pretty good at handling reasonable load.

In the "real world" I have seen commercial instruments report status into a queue for processing, so it is certainly a valid solution.