Long Response get cuts: clojure.lang.ExceptionInfo: Couldnt' write to stream

Question

I added the following code to a clean Luminus and met the following problem:

If I request an API which responses with 100 KB (maybe, or more) data, the long responses are cut off, so that I can only get part of that.

I am not sure whether this is because of the following code. I am not familiar with JAVA, can you give me some advices? I am using Luminus framework and the related codes are:

(def app-routes
  (routes
   (-> #'home-routes
       (wrap-routes middleware/wrap-csrf)
       (wrap-routes middleware/wrap-formats)
       )
   (->
    (route/not-found
     (:body
      (error-page {:status 404
                   :title "page not found"})))
    middleware/wrap-proxy)))

(defn app [] (middleware/wrap-base #'app-routes))

(defn create-proxy [handler fns]
  (let [identifier-fn (get fns :identifier-fn identity)
        host-fn (get fns :host-fn {})
        path-fn (get fns :path-fn identity)]
    (fn [request]
      (let [request-key (identifier-fn request)
            host (host-fn request-key)
            stripped-headers (dissoc (:headers request) "content-length" "host")
            path (path-fn (:uri request))
            ]
        (if host
          (->
           {:url              (build-url host (path-fn (:uri request)) (:query-string request))
            :method           (:request-method request)
            :body             (if-let [len (get-in request [:headers "content-length"])]
                                (slurp-binary
                                 (:body request)
                                 (Integer/parseInt len)))
            :headers          stripped-headers
            :follow-redirects true
            :throw-exceptions false
            :as               :stream
            :insecure?        true
            }
           client/request
           (update-in [:headers] dissoc "Transfer-Encoding")
           prepare-cookies
           )
          (handler request))))))


(defn wrap-proxy [handler]
  (-> handler
      (create-proxy
       {:identifier-fn :uri
        :host-fn (fn [^String uri]
                   (cond
                     ;; (.startsWith uri "/www") "http://localhost:6787/p/www/stockinfogate"
                     (.startsWith uri "/www") "https://m.joudou.com/p/www/stockinfogate"
                     (.startsWith uri "/beta") "https://m.joudou.com/p/beta/stockinfogate"
                     :else nil))
        :path-fn (fn [^String uri] (s/join "/" (cons "" (drop 2 (s/split uri #"/")))))})))

The url get response data cut: http://localhost:3000/www/stock/klinedata?stockid=300370.SZ&period=D

the original url: https://www.joudou.com/stockinfogate/stock/klinedata?stockid=300370.SZ&period=D

The error message are:

2017-05-20 00:19:02,287 [qtp1106269094-16] WARN  org.eclipse.jetty.server.HttpChannel - //localhost:3000/www/stock/klinedata?stockid=300370.SZ&period=D
clojure.lang.ExceptionInfo: Couldnt' write to stream
    at clojure.core$ex_info.invokeStatic(core.clj:4617)
    at clojure.core$ex_info.invoke(core.clj:4617)
    at qbits.jet.servlet$write_stream_BANG_.invokeStatic(servlet.clj:131)
    at qbits.jet.servlet$write_stream_BANG_.invoke(servlet.clj:126)
    at qbits.jet.servlet$eval37883$fn__37884.invoke(servlet.clj:153)
    at qbits.jet.servlet$eval37796$fn__37797$G__37787__37806.invoke(servlet.clj:88)
    at qbits.jet.servlet$set_response_body_BANG_.invokeStatic(servlet.clj:93)
    at qbits.jet.servlet$set_response_body_BANG_.invoke(servlet.clj:91)
    at qbits.jet.servlet$set_body_BANG_.invokeStatic(servlet.clj:229)
    at qbits.jet.servlet$set_body_BANG_.invoke(servlet.clj:220)
    at qbits.jet.servlet$eval38053$fn__38054.invoke(servlet.clj:263)
    at qbits.jet.servlet$eval38024$fn__38025$G__38015__38032.invoke(servlet.clj:236)
    at qbits.jet.servlet$update_response.invokeStatic(servlet.clj:241)
    at qbits.jet.servlet$update_response.invoke(servlet.clj:239)
    at qbits.jet.server$make_handler$fn__38414.invoke(server.clj:78)
    at qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:518)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
    at java.lang.Thread.run(Thread.java:745)

and I lost a slurp-binary, I've tried two versions of this function, one is

(defn slurp-binary
  "Reads len bytes from InputStream is and returns a byte array."
  [^java.io.InputStream is len]
  (with-open [rdr is]
    (let [buf (byte-array len)]
      (.read rdr buf)
      buf)))

the other is:

 (defn slurp-binary
    "Reads len bytes from InputStream is and returns a byte array."
    [^java.io.InputStream is len]
  (when (and is
             (> len 0))
    (with-open [rdr is]
      (let [buf (byte-array len)]
        (.read rdr buf)
        buf))))

I am not sure the function is where the error happens, but neither of the two version works.

Thanks.

---

I tried to get the raw req and client/request the req, then I got the full response in REPL. So I doubt that it is the wrap-base who cause the response to be cut off.

Answer 1

What is this slurp-binary? It seems to be doing all of the important work, but is not part of any commonly-used library I know of.

Googling around suggests you may be using https://crossclj.info/ns/com.guaranteedrate/ring-proxy/3.0.0/tailrecursion.ring-proxy.html#_slurp-binary, which is a function that is obviously incorrect at first glance. I could imagine it would cause errors like those you're seeing, but I'm not sure. Looking there I also see a wrap-proxy function that is very similar to the body of your function: did you write both of these, or are you copying them both, or what?

At any rate, slurp-binary is clearly an important part of the problem and we can't really answer your question without knowing what is going on.