I have a Kinesis stream that I would like to archive with Firehose to S3. I use a transformator lambda to decompress data from Kinesis stream. Data is originally in compressed format to fit the payload size limitation of Kinesis. I enabled logging for both Firehose and lambda. If the Firehose Transform buffer size is too big, the records won't be processed and go to the processing-failed folder. Lowering the buffer size solves the problem, but I would like to set up a solution to see any errors in the future. In this case of error, records on S3 have the following error code and message:
"errorCode":"Lambda.FunctionError",
"errorMessage":"The Lambda function was successfully invoked but it returned an error result."
While I can't see any error either in Firehose logs, or lambda logs. Errors can be seen only as processing-failed records.
How can I log the specific error to any of the logs?
How can I define a CloudWatch alert to catch these types of errors?