AWS design pattern for moving a file delivered to s3 bucket to a directory within an (Fargate) EKS container?

24 Views Asked by At

We are migrating an on-prem solution where we currently receive files from vendors via ftp into a directory location on our unix server and then a program on that server moves that file to some other path performs ETL on it.

We have containerized the application and it still expects to read a file from a directory.

What we want to do generally is this:

  1. Vendor delivers file to S3 bucket
  2. Container sees the zipped file (constantly polling)
  3. Container moves and extracts the file to some persistent disk (EFS?)
  4. Container does ETL

A couple ideas have come to mind and wanted to get advice since this is new to us:

Options 1:

S3 -> Lambda function pushes file to EFS -> EFS is exposed as a PV which is mounted to a directory on the container Observation: We feel like there's excessive overhead of houskeeping the file in two places (S3 and EFS).

Options 2:

Look into datasync to see if it can replace the need for Lambda.

To reiterate constraints:

  1. File must be delivered from the vendor to the S3 bucket
  2. The container must be able to read it from some directory that appears local to it (mount points are OK)

Any ideas or suggestions on a simple/reusable design? This is a pattern we plan to use for a large number of ingestions from different sources.

0

There are 0 best solutions below