Can I use RabbitMQ to distribute large files to multiple machines?

Question

I work for a digital signage company where we send lots of animations/videos to various machines on the network. We have used ftp in the past to send files from a server app (which is our content management system) to machines that play the content.

Is it possible to use RabbitMQ in such an environment where we are sending large files (some videos may be a gigabyte or more in size)? Most files will be really small but occasionally we will send large videos.

Also some of the player machines will be added later which means that the queuing has to be dynamic (i.e. player added so now we can send content to it). Can RabbitMQ be configured in this way? (from the demos I have seen you have to create the queues up front before the server and client applications launch).

Also can we secure the sending of files so that the sending app has to log in before sending contents to a machine (like login for FTP)?

Answer 1

In order to add dynamic consumers of your files You can use EasyNetQ on top of your RabbitMQ broker with a publish/subscribe behavior.

As mentioned by Russ Cam, in the google group people have sent big messages but this could break a RabbitMQ cluster.

The solution could be an hybrid between FTP (or any other file transfer protocol) and Messages:

• Every time your client want to transfer a file it publish a message to the bus with a reference to the file (e.g. the FTP url)
• The consumers of this message will then connect through FTP in order to download the file

EasyNetQ already provides you a default subscription mechanism wich helps to subscribe dinamically (see link above) new consumers without configuring RabbitMQ in advance.

Finally you can think about splitting your files in many small message chunks but then you should create a function wich will reassemble the file on the receiver or worst you could lose some messages if your consumers start later.