Getting "Remotely closed" error for HTTP POST Request In the MULE 4

Question

I am Getting below error while calling a Mulesoft system api from mulesoft process API via DLB in the cloudHub. The frequency of closing the remote connection error is not fixed. Sometimes this error comes after 2 minutes and sometimes after 5 minutes. Though, with second retry it works but still I want to avoid this error as its being happening very frequently.

 HTTP POST on resource 'https://internal-nonprod-dlb.lb.anypointdns.net:443/api/sys/aws/s3/databricks/object' failed: Remotely closed.

Mule Version : 4.4
HTTP connector version: 1.7.3
DLB Timeout : 7 minutes
Payload size : ~ 30 MB
System APIs Listener Idle timeout : 5 Minutes

request configuration in process api

<http:request method="POST" doc:name="POST GZIP / aws system api" doc:id="0b490747-5069-4546-9446-8b77130ae848" config-ref="Aws_Sys_API_HTTP_Request_configuration" path="${awsSysApi.databricksPath}" responseTimeout="600000">
                <reconnect />
                    <http:headers><![CDATA[#[output application/java
---
{
    "client_secret" : p('secure::awsSysApi.client_secret'),
    "Content-Type" : "application/gzip",
    "client_id" : p('secure::awsSysApi.client_id')
}]]]></http:headers>
            <http:query-params><![CDATA[#[output application/java
---
{
    "bucketName" : p('aws.bucket.datalakeRawDeBucket'),
    "key" : vars.key
}]]]></http:query-params>
            <http:response-validator>
                <http:success-status-code-validator values="200..499" />
            </http:response-validator>
        </http:request>

HTTP requests global configuration in process API

<http:request-config name="Aws_Sys_API_HTTP_Request_configuration" doc:name="HTTP Request configuration" doc:id="5a7eb30f-9850-4de5-8cca-a7d77b0c10d4" basePath="${awsSysApi.basepath}">
        <http:request-connection host="${awsSysApi.host}" port="${awsSysApi.port}" protocol="HTTPS" connectionIdleTimeout="${awsSysApi.idletTimeout}">
            <reconnection>
                <reconnect frequency="${retry.millisecondsBetweenRetries}" count="${retry.maxRetries}" />
            </reconnection>
            <tls:context>
                <tls:trust-store insecure="true" />
                <tls:key-store type="jks" path="${tls.keyStore.path}" keyPassword="${secure::tls.keyStore.keyPassword}" password="${secure::tls.keyStore.password}" />
            </tls:context>
        </http:request-connection>
    </http:request-config>

system APIs listener configuration:

        <http:listener-connection host="${http.host}" port="${http.private.port}" readTimeout="300000" connectionIdleTimeout="360000">
            <reconnection>
                <reconnect frequency="30000" count="2" />
            </reconnection>
        </http:listener-connection> ``` 


Please let me know if i have missed any information. 

Answer 1

Probably you system API is failing to process big payloads. It could be also related to concurrency, or both. Maybe insufficient resources.

You should try to reproduce locally with a similar memory configuration with different payloads to understand the issue.

The HTTP configurations alone are unlikely to be related to the issue. FYI the Connection idle timeout is not related to connection timeouts.

Answer 2

If this is not a consistent issue try using until-successful component.