I have 4 node microservices, of which 3 are deployed in AWS AppRunner and a main orchestrator service which is deployed in AWS Elastic Beanstalk (HTTPS Configured for all the services).
The flow normally works like the orchestrating node service from Elastic Beanstalk communicates with the node services on AppRunner in order to finish a process. The orchestrator works as expected for few processes at the beginning and at the final function, it communicates with one of the services on AppRunner. This process should take more than 100 seconds to complete, and then return a success response. But, at the halfway it throws an exception 503 : Service Unavailable : 'upstream connect error or disconnect/reset before headers. reset reason: connection termination' whereas the microservice on AppRunner continues the execution and completes it successfully but the main orchestrator service stops without further acceptance of the response back.
Earlier, the main orchestrator was also deployed in the AppRunner and was giving the same 503 issue. I tried moving it to Elastic Beanstalk, tried increasing the timeouts, setting up keep-alive connection header, reduce the data to process on order to not make the service run for more than 100 seconds (I was able to achieve 45 seconds) but the issue persist and the exception is caught. But only sending a successful response without any processing done inside makes the process complete successfully.
Attaching the logs of the orchestrator error below...
----------------------------------------
May 24 11:43:44 ip-172-31-10-7 web: statusText: 'Service Unavailable',
May 24 11:43:44 ip-172-31-10-7 web: headers: {
May 24 11:43:44 ip-172-31-10-7 web: 'content-length': '95',
May 24 11:43:44 ip-172-31-10-7 web: 'content-type': 'text/plain',
May 24 11:43:44 ip-172-31-10-7 web: date: 'Tue, 24 May 2022 11:43:43 GMT',
May 24 11:43:44 ip-172-31-10-7 web: server: 'envoy',
May 24 11:43:44 ip-172-31-10-7 web: connection: 'close'
May 24 11:43:44 ip-172-31-10-7 web: },
May 24 11:43:44 ip-172-31-10-7 web: config: {
May 24 11:43:44 ip-172-31-10-7 web: url: 'https://m2me6p2bmj.us-east-2.awsapprunner.com/adminpmreport',
May 24 11:43:44 ip-172-31-10-7 web: method: 'post',
May 24 11:43:44 ip-172-31-10-7 web: data: {<<HIDDEN FOR PRIVACY>>}
May 24 11:43:44 ip-172-31-10-7 web: headers: [Object],
May 24 11:43:44 ip-172-31-10-7 web: transformRequest: [Array],
May 24 11:43:44 ip-172-31-10-7 web: transformResponse: [Array],
May 24 11:43:44 ip-172-31-10-7 web: timeout: 0,
May 24 11:43:44 ip-172-31-10-7 web: adapter: [Function: httpAdapter],
May 24 11:43:44 ip-172-31-10-7 web: xsrfCookieName: 'XSRF-TOKEN',
May 24 11:43:44 ip-172-31-10-7 web: xsrfHeaderName: 'X-XSRF-TOKEN',
May 24 11:43:44 ip-172-31-10-7 web: maxContentLength: Infinity,
May 24 11:43:44 ip-172-31-10-7 web: maxBodyLength: Infinity,
May 24 11:43:44 ip-172-31-10-7 web: validateStatus: [Function: validateStatus],
May 24 11:43:44 ip-172-31-10-7 web: transitional: [Object]
May 24 11:43:44 ip-172-31-10-7 web: },
May 24 11:43:44 ip-172-31-10-7 web: request: <ref *1> ClientRequest {
May 24 11:43:44 ip-172-31-10-7 web: _events: [Object: null prototype],
May 24 11:43:44 ip-172-31-10-7 web: _eventsCount: 7,
May 24 11:43:44 ip-172-31-10-7 web: _maxListeners: undefined,
May 24 11:43:44 ip-172-31-10-7 web: outputData: [],
May 24 11:43:44 ip-172-31-10-7 web: outputSize: 0,
May 24 11:43:44 ip-172-31-10-7 web: writable: true,
May 24 11:43:44 ip-172-31-10-7 web: destroyed: false,
May 24 11:43:44 ip-172-31-10-7 web: _last: true,
May 24 11:43:44 ip-172-31-10-7 web: chunkedEncoding: false,
May 24 11:43:44 ip-172-31-10-7 web: shouldKeepAlive: false,
May 24 11:43:44 ip-172-31-10-7 web: _defaultKeepAlive: true,
May 24 11:43:44 ip-172-31-10-7 web: useChunkedEncodingByDefault: true,
May 24 11:43:44 ip-172-31-10-7 web: sendDate: false,
May 24 11:43:44 ip-172-31-10-7 web: _removedConnection: false,
May 24 11:43:44 ip-172-31-10-7 web: _removedContLen: false,
May 24 11:43:44 ip-172-31-10-7 web: _removedTE: false,
May 24 11:43:44 ip-172-31-10-7 web: _contentLength: null,
May 24 11:43:44 ip-172-31-10-7 web: _hasBody: true,
May 24 11:43:44 ip-172-31-10-7 web: _trailer: '',
May 24 11:43:44 ip-172-31-10-7 web: finished: true,
May 24 11:43:44 ip-172-31-10-7 web: _headerSent: true,
May 24 11:43:44 ip-172-31-10-7 web: socket: [TLSSocket],
May 24 11:43:44 ip-172-31-10-7 web: _header: 'POST /adminpmreport HTTP/1.1\r\n' +
May 24 11:43:44 ip-172-31-10-7 web: 'Accept: application/json, text/plain, */*\r\n' +
May 24 11:43:44 ip-172-31-10-7 web: 'Content-Type: application/json\r\n' +
May 24 11:43:44 ip-172-31-10-7 web: 'Connection: keep-alive\r\n' +
May 24 11:43:44 ip-172-31-10-7 web: 'User-Agent: axios/0.21.4\r\n' +
May 24 11:43:44 ip-172-31-10-7 web: 'Content-Length: 220176\r\n' +
May 24 11:43:44 ip-172-31-10-7 web: 'Host: m2me6p2bmj.us-east-2.awsapprunner.com\r\n' +
May 24 11:43:44 ip-172-31-10-7 web: '\r\n',
May 24 11:43:44 ip-172-31-10-7 web: _keepAliveTimeout: 0,
May 24 11:43:44 ip-172-31-10-7 web: _onPendingData: [Function: noopPendingOutput],
May 24 11:43:44 ip-172-31-10-7 web: agent: [Agent],
May 24 11:43:44 ip-172-31-10-7 web: socketPath: undefined,
May 24 11:43:44 ip-172-31-10-7 web: method: 'POST',
May 24 11:43:44 ip-172-31-10-7 web: maxHeaderSize: undefined,
May 24 11:43:44 ip-172-31-10-7 web: insecureHTTPParser: undefined,
May 24 11:43:44 ip-172-31-10-7 web: path: '/adminpmreport',
May 24 11:43:44 ip-172-31-10-7 web: _ended: true,
May 24 11:43:44 ip-172-31-10-7 web: res: [IncomingMessage],
May 24 11:43:44 ip-172-31-10-7 web: aborted: false,
May 24 11:43:44 ip-172-31-10-7 web: timeoutCb: null,
May 24 11:43:44 ip-172-31-10-7 web: upgradeOrConnect: false,
May 24 11:43:44 ip-172-31-10-7 web: parser: null,
May 24 11:43:44 ip-172-31-10-7 web: maxHeadersCount: null,
May 24 11:43:44 ip-172-31-10-7 web: reusedSocket: false,
May 24 11:43:44 ip-172-31-10-7 web: host: 'm2me6p2bmj.us-east-2.awsapprunner.com',
May 24 11:43:44 ip-172-31-10-7 web: protocol: 'https:',
May 24 11:43:44 ip-172-31-10-7 web: _redirectable: [Writable],
May 24 11:43:44 ip-172-31-10-7 web: [Symbol(kCapture)]: false,
May 24 11:43:44 ip-172-31-10-7 web: [Symbol(kNeedDrain)]: false,
May 24 11:43:44 ip-172-31-10-7 web: [Symbol(corked)]: 0,
May 24 11:43:44 ip-172-31-10-7 web: [Symbol(kOutHeaders)]: [Object: null prototype]
May 24 11:43:44 ip-172-31-10-7 web: },
May 24 11:43:44 ip-172-31-10-7 web: data: 'upstream connect error or disconnect/reset before headers. reset reason: connection termination'
May 24 11:43:44 ip-172-31-10-7 web: },
May 24 11:43:44 ip-172-31-10-7 web: isAxiosError: true,
May 24 11:43:44 ip-172-31-10-7 web: toJSON: [Function: toJSON]
May 24 11:43:44 ip-172-31-10-7 web: }
May 24 11:43:44 ip-172-31-10-7 web: !!! Error !!! : Request failed with status code 503
May 24 11:43:44 ip-172-31-10-7 web: << processAdminPMReportStartTime INITIAL CALL ENDED IN 120.069 SECONDS (function toISOString() { [native code] }) TIME >> Line 346
May 24 11:43:44 ip-172-31-10-7 web: !!! Error !!! : Couldn't Fetch Target URL
May 24 11:43:44 ip-172-31-10-7 web: !!! Error !!! : Couldn't Fetch extractionResult ..
May 24 11:43:44 ip-172-31-10-7 web: AUDIT::WIP Execution completed for : 2021 :: 2021-12-31 :: true
May 24 11:43:44 ip-172-31-10-7 web: << LOGOUT CALL STARTED AT 2022-05-24T11:43:44.094Z >> Line 374
May 24 11:43:44 ip-172-31-10-7 web: QBO Reset completed
May 24 11:43:44 ip-172-31-10-7 web: (node:7453) UnhandledPromiseRejectionWarning: Error: Response has an Error
May 24 11:43:44 ip-172-31-10-7 web: at /var/app/current/node_modules/intuit-oauth/src/OAuthClient.js:526:40
May 24 11:43:44 ip-172-31-10-7 web: at processTicksAndRejections (internal/process/task_queues.js:95:5)
May 24 11:43:44 ip-172-31-10-7 web: at async Object.logout (/var/app/current/services/auth-service.js:124:28)
May 24 11:43:44 ip-172-31-10-7 web: (node:7453) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 10)
Looking out for a solution!.. I'm being stuck in this issue for weeks.
Thanks in advance.