I have created a AWS lambda function (written in python) that reads a tar.gz file from one S3 bucket, unzips and untars it and writes the extracted files to another S3 bucket. Tar inside the GZ is of >1GB size so lambda takes more time to complete the task.
I invoke this lambda function from a java client. I am using AWS SDK V2 for Java (software.amazon.awssdk.*), and using Lambda sync client software.amazon.awssdk.services.lambda.LambdaClient.
Though the lambda invocation works (lambdaClient.invoke(invokeRequest)), but it fails with "Read timed out" error. In the background (in AWS) the lambda completes its execution after sometime.
Following is the lambda client bean creation code.
LambdaClient lambdaClient = LambdaClient.builder()
.credentialsProvider(awsCredentialsProvider)
.region(Region.US_EAST_1)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofMinutes(30))
.apiCallAttemptTimeout(Duration.ofMinutes(30))
.build()
)
.build();
Following is the lambda invocation code.
//This is a user defined pojo object that maps to lambda input json payload
UntarLambdaPayload untarLambdaPayload = UntarLambdaPayload.builder()
.sourceBucket(lambdaProps.getSourceBucket())
.destinationBucket(lambdaProps.getDestinationBucket())
.sourceKey("myTarFile.tar.gz")
.build();
ObjectMapper mapper = new ObjectMapper();
String jsonRequest = mapper.writeValueAsString(untarLambdaPayload);
SdkBytes payload = SdkBytes.fromUtf8String(jsonRequest);
InvokeRequest invokeRequest = InvokeRequest.builder()
.functionName(lambdaProps.getFunctionName())
.overrideConfiguration(AwsRequestOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofMinutes(30))
.apiCallAttemptTimeout(Duration.ofMinutes(30)).build())
.payload(payload)
.build();
InvokeResponse res = lambdaClient.invoke(invokeRequest);
And I am getting the below exception.
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read timed out
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:83)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:167)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.lambda.DefaultLambdaClient.invoke(DefaultLambdaClient.java:2355)
My code's further logic depends on the successful completion of the lambda function. If lambda is timed out then code cannot proceed with processing the untarred files in S3 bucket#2.
I tried overrideConfiguration with apiCallTimeout and apiCallAttemptTimeout in InvokeRequest (as well as in Lambda Client), but it did not work. I am going to do research on LambdaClient waiter functionality for which I haven't got any help so far on how to use it with Lambda.
How can I make the lambdaClient.invoke(invokeRequest) wait until the lambda running in AWS completes its execution?