I want to do a multipart upload from Kinesis into my Amazon S3 bucket. As Amazon S3 is a file system, for every entry it will create a file under the given bucket name.
My Amazon S3 feeds the AWS glue jobs as it triggers the Lambda function as soon as there is a new file in that particular folder in Amazon S3. With stream data there will be multiple files per second.
How can I control the size of the file on the Kinesis side so that Kinesis only pushes data on an Amazon S3 bucket after the certain threshold is reached? So that I trigger my job whenever I reach that size.
You can use AWS Glue Triggers instead of lambda.
For example, set a cron time for the Glue.