I currently have a Sinatra app running in an FCGI handler. I want to write a handler that will sit within the rackup file (probably in front of the Sinatra app) and will stream big file uploads to another server via sockets (without buffering it on disk first) and do so in interlock with the request. So what I would like to do is some kind of stream-decode-send workflow without param preparsing. I've read somewhere that there is a problem with this because specifically due to the way the Rails team wants to see the middleware pipeline all uploads in Rack have been made rewindable which implies that the upload will be buffered, so not only I cannot provide an upload progress within Rack but I also have to buffer the file on disk and then send it downstream.
Is there some cross-backend solution that ties the request loop of the webserver to the Rack responder and does not force rewinding on the input (and does not force in-memory buffering of the upload which is an absolute stupid madness)? What are the current approaches to this kind of problem?
You are right: the Rack spec mandates rewindable input, which implies buffering. It seems Rack is not the tool for this job.
You may want to try FastCGI, which does indeed allow non-buffered streaming. Or maybe a Java Servlet. My 2¢: Do you really need it? If not, don't worry, disk space is really cheap. If so, do you really need to do it in Ruby?
edit: Mongrel::HTTPRequest does not support unbuffered large streaming inputs (without monkeypatching)