Load from S3 into Redshift using Kinesis

Question

I receive my source files in S3, i need to load this data into redshift.

The source S3 files are populated via Kinesis Firehose.

Source :

                 via Kinesis
     Webstreams -------------- > S3

Requirement :

                 via Kinesis
     Webstreams -------------- > S3 ------> Redshift

I would like to know if i can use Kinesis to load into redshift from S3.

Thanks & Regards, Srivignesh KN

Answer 1

You can use theCOPY command to copy data from S3 to Redshift

More details here

Answer 2

Amazon Kinesis Firehose can send incoming streaming data to Amazon S3, Amazon Redshift and Amazon Elasticsearch Service.

Please note that a Firehose stream is slightly different to a normal Kinesis stream. It has a slightly different format and is fully automated. It will not accept your file from Amazon S3, but it will create its own files.

Answer 3

I have resolved this use case in the following manner.

1. Created an API Gateway endpoint.

Applications will curl the gateway endpoint and post the records.

2. Firehose will be the endpoint for API Gateway.

3. So, records are streamed into S3 intermediate buckets and loaded into redshift.

The entire flow is as below :

WebStreams -> API Gateway -> Kinesis Firehose -> S3 (Intermediate Bucket) -> Redshift.

Thanks & Regards,

Srivignesh KN