Solr 8.4 Getting Async exception during distributed update: java.io.IOException: Broken pipe when trying to post a document

Question

Why I am seeing the error:

Async exception during distributed update: java.io.IOException: Broken pipe when trying to post a document to solr

• Solr version: 8.4.1
• Zookeeper: 3.4.14
• OpenJDK 11;
• 2 solr node + 1 zookeeper (hosted zookeeper in one of the solr node)

Using basic authentication on solr and also with TLS1.2.

I am not seeing any error when trying to post to solr node which is a leader and it's working consistently; it is the issue only when I post to the non-leader node which is throwing the below error.

java.io.IOException: java.io.IOException: Broken pipe
    at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
    at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
    at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
    at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
    at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
    at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:172)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:103)
    at org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
    at org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:339)
    at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:236)
    at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
    Suppressed: java.io.IOException: java.io.IOException: Broken pipe
        at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
        at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
        at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
        at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
        at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
        at org.apache.solr.common.util.JavaBinCodec.close(JavaBinCodec.java:1269)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:104)
        ... 10 more
    Caused by: java.io.IOException: Broken pipe
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:927)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)
        at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:247)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:755)
        at org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:734)
        at org.eclipse.jetty.http2.client.HTTP2ClientConnectionFactory$HTTP2ClientConnection.onOpen(HTTP2ClientConnectionFactory.java:130)
        at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:441)
        at org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:115)
        at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:85)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)

Answer 1

• Client disconnected the connection because of too much waiting time on the server. Higher timeouts may cause performance issues.

• Another reason might be the OS cache. Check the disk space and try to increase the disk space.

Answer 2

When we had this issue it was because we were running the wrong JDK. We had installed OpenJDK 16 and that was still being used by Solr and Zookeeper. There was some incompatibility that was causing the broken pipe error when the leader was communicating with followers.

To fix this we had to install Open JDK Java 1.8 and force Solr and Zookeeper to use that.