I have created an API in Ruby on Rails. Rather than sending the response body back with the API response, I am broadcasting it to an external service, which then takes care of the realtime distribution to all connected clients (i.e. WebSockets).
Today I have hit a snag where I realized that our 3rd party provider only allows data packets of up to 25kb. One of our responses has started giving problems as the data grew to be more than this limit and the 3rd party service has started blocking calls. As a side note, data packets will seldom grow to be greater than 25kb.
I did some research and was contemplating what the best idea would be. One idea I was thinking of, was to compress the response using ZLib and then to decompress it on the JS side. The article that led to this was this StackOverflow question.
I managed to get the deflation & Base64 encoding right, but could not decode on the JS side. I also tested the Base 64 string generated but services like this one, flags the base64 string as invalid.
My code looks like this:
In a Rails Controller
...
compressed_data = Zlib::Deflate.deflate(json_data.to_s)
encoded_data = Base64.encode64(compressed_data)
broadcast encoded_data
...
In JS that receives the broadcast:
import pako from 'pako';
import { decode } from 'js-base64';
...
const decoded = decode(payload.data);
const decompressed = pako.inflate(decoded);
...
When I execute the broadcast, I get the error: unknown compression method. I understand that this might have something to do with pako, but I have also tried other approaches with no success. Does anybody have any ideas or perhaps even a better way to approach this problem?
UPDATE:
The Base64 string generated in rails looks like this:
eJxlU2Fr2zAQ/SuHP25JkJW2abMPY6OllJWkNGFftmEU+RKLypKRTknD6H/f\nyQ5ru0EQutPdy3vvzr8LUxfz8nJ2NSrIkMViXkj4GjyGCNc+tBjgZxJCXsAq\nbbfePsF3NNa4XTEqaow6mI6Md9xWSgGPqm0RPkC+r8gjeW9yLDn+5vktRDPA\nYWrtJ4uwPBUIka9wr/qiCTze3N6t1o9f1nfLBTzcLK7vFref4cGiiggBdyYS\nU9scQRHkJEE5axjEO6AGoVZH2ODWB+xDlXRmOYGl0wgHhEbtM4xGs8cajj6F\nE2hQuXA0pGokZWyEg7GW4SCiIyDfQwa0uFccW4aI5PUTqF1+PzR+aNDekdKU\noXKT9m1nkQZCeyRiE6ELXmNkvWzniWRlvVYnT+/9gVUuQ4euVjyc16JIKlBV\nK+onJmQ5FuVYynU5nQvBv4kQ4qOQfHvTxCinFpesfc3TscswK2NZANdvTF0z\nuwqfCV0cqDj/JuSSriL1XIUeTXCcjjy3qgvYmtT2qRq3KlkmiZ2PhvqcTiGg\n00cGXKgF4+iADFFXigYdYlzKsTxbl2I+vZpPy4mUs786Ule/K+5Flyya32Uu\nvijL1+KIocrbPcv0gnK66cOzy1ER2fMsPMeDFSy5Mo7ZtGxBtVc2YSzmP0q2\ncSTzMc3HeT5yTvwaFU2/q9kG/oLOR0XLQzeaV7Hq0saa2FTkn9D9a7bl4VKq\n/xuC9W73/kHFNs+5H7HnFcCaZTlFKeTMATf5z/rvMO/VYEtuffkDW0lDVA==\n
Your data starts off with a zlib header, but the compressed data is corrupted for some reason.