Play Enumerator hanging

Question

I'm using Play 2.1.2 and I'm having some trouble with an Enumerator and I'm seeking ideas on how to debug this.

I'm trying to stream some S3 data through my server. I can get an InputStream from the Amazon SDK for my S3 file (getObject(bucket, key).getObjectContent()). I then turn that InputStream into an Enumerator[Array[Byte]] using Enumerator.fromStream.

All this type checks and on my local development machine it all works perfectly. When I formulate my Result in Play, I just return Ok.stream(enum).

The problem comes when I deploy this to a production server. The very first time I request the file, it works just fine and I get the whole file. But subsequent times it frequently gets part way through (different amounts each time) and then gets "stuck". I wrapped the Enumerator as follows to be able to log whether the enumeration completed:

val wrapped = enum.onDoneEnumerating { println("Contents fully enumerated"); }
Ok.stream(wrapped);

As expected, on my development machine (and the first time through on the production machines), I get the message "Contents fully enumerated". But after that, the production machine will start the download of the file, but it doesn't finish (in both the HTTP sense and in the Enumerator sense).

I'm not sure how to debug this. Obviously, fromStream does some magic and I don't know how to figure out what is happening between chunks. I thought this might be a thread pool issue so I wrapped the whole response in a future { blocking { ... } } block, but it didn't appear to make any difference.

I'm trying to avoid the hassle of creating a local temporary file from S3 and then building my Enumerator from that. Using the fromStream to create an Enumerator seemed like the elegant way to do this...if it worked.

Suggestions?

Answer 1

OK, so I think I figured this out. It turns out that on the Play side, things appear to be working. I tried all kinds of variations (different ways of constructing the enumerator, creating temporary files, etc). It didn't really matter.

What did matter was the proxy I was using. I'm using node-http-proxy and if I make a request on the server behind the proxy, I get the correct response (directly from Play). If I make the request on the server outside the proxy, I get an incorrect (empty) response. So it looks like the proxy is "dropping" the response.

It appears the issue is that the response is chunked by the stream call and this is causing the problem with the proxy. If I reformulate my response to be:

SimpleResult( header = ResponseHeader(200), body = enum)

Then play uses the Enumerator to construct a complete response (not streamed) and things work again. Of course, it is stupid to have to form the complete response in this case, but it does work. Hopefully I can find a better solution than this in the long term, but this seems to work for now.