I have data coming into Kinesis streams from different producers. The data needs to be transformed into several different tables depending the the they type of that data. I'm trying to have a Lambda read from Kinesis Streams, transform the data into the different tables and then use Kinesis Firehose to batch and COPY into Redshift.
However since Firehose can only deliver to one Redhsift table at a time, I need to have multiple Firehose instances running equal to the number of tables I will have in my Redshift schema. I see that there is a default cap of 20 Firehose instances per region and I'm wondering if this is an ideal data pipeline for what I am trying to achieve.
I have designed almost similar data pipeline like yours. I get the data from the kinesis, transform the data after it has reached redshift. All the data from 1 stream goes to a one master table in redshift.Then, I run a cron job, which transforms the data into different tables.
If I’ve made a bad assumption please comment and I’ll refocus my answer.