Accessing specified key from s3 bucket?

Question

I have a S3 bucket xxx. I wrote one lambda function to access data from s3 bucket and writing those details to a RDS PostgreSQL instance. I can do it with my code. I added one trigger to the lambda function for invoking the same when a file falls on s3.

But from my code I can only read file having name 'sampleData.csv'. consider my code given below

public class LambdaFunctionHandler implements RequestHandler<S3Event, String> {

private AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();

public LambdaFunctionHandler() {}

// Test purpose only.
LambdaFunctionHandler(AmazonS3 s3) {
    this.s3 = s3;
}

@Override
public String handleRequest(S3Event event, Context context) {
    context.getLogger().log("Received event: " + event);
    String bucket = "xxx";
    String key = "SampleData.csv";




     System.out.println(key);

     try {
         S3Object response = s3.getObject(new GetObjectRequest(bucket, key));
         String contentType = response.getObjectMetadata().getContentType();
         context.getLogger().log("CONTENT TYPE: " + contentType);
      // Read the source file as text
         AmazonS3 s3Client = new AmazonS3Client();
         String body = s3Client.getObjectAsString(bucket, key);
         System.out.println("Body: " + body);
         System.out.println();
         System.out.println("Reading as stream.....");
         System.out.println();

         BufferedReader br = new BufferedReader(new InputStreamReader(response.getObjectContent()));
  // just saving the excel sheet data to the DataBase       
         String csvOutput;
         try { 
            Class.forName("org.postgresql.Driver");
            Connection con = DriverManager.getConnection("jdbc:postgresql://ENDPOINT:5432/DBNAME","USER", "PASSWORD");
            System.out.println("Connected");
            // Checking EOF
         while ((csvOutput = br.readLine()) != null) {
            String[] str = csvOutput.split(",");
            String name = str[1];
            String query = "insert into schema.tablename(name) values('"+name+"')";
            Statement statement = con.createStatement();
            statement.executeUpdate(query);

         }
         System.out.println("Inserted Successfully!!!");
         }catch (Exception ase) {
            context.getLogger().log(String.format(
                     "Error getting object %s from bucket %s. Make sure they exist and"
                     + " your bucket is in the same region as this function.", key, bucket));
            // throw ase;
         }


         return contentType;
     } catch (Exception e) {
         e.printStackTrace();
         context.getLogger().log(String.format(
             "Error getting object %s from bucket %s. Make sure they exist and"
             + " your bucket is in the same region as this function.", key, bucket));
         throw e;
     }
}

From my code you can see that I mentioned key="SampleData.csv"; is there any way to get the key inside a bucket without specifying a specific file name?

Answer 1

These couple of links would be of help.

http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingJava.html

You can list objects using prefix and delimiter to find the key you are looking for without passing a specific filename.

Answer 2

If you need to get the event details on S3, you can actually enable the s3 event notifier to lambda function. Refer the link You can enable this by,

1. Click on 'Properties' inside your bucket
2. Click on 'Events '
3. Click 'Add notification'
4. Give a name and select the type of event (eg. Put, delete etc.)
5. Give prefix and suffix if necessary or else leave blank which consider all events
6. Then 'Sent to' Lambda function and provide the Lambda ARN.

Now the event details will be sent lambda function as a json format. You can fetch the details from that json. The input will be like this:

{"Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"ap-south-1","eventTime":"2017-11-23T09:25:54.845Z","eventName":"ObjectRemoved:Delete","userIdentity":{"principalId":"AWS:AIDAJASDFGZTLA6UZ7YAK"},"requestParameters":{"sourceIPAddress":"52.95.72.70"},"responseElements":{"x-amz-request-id":"A235BER45D4974E","x-amz-id-2":"glUK9ZyNDCjMQrgjFGH0t7Dz19eBrJeIbTCBNI+Pe9tQugeHk88zHOY90DEBcVgruB9BdU0vV8="},"s3":{"s3SchemaVersion":"1.0","configurationId":"sns","bucket":{"name":"example-bucket1","ownerIdentity":{"principalId":"AQFXV36adJU8"},"arn":"arn:aws:s3:::example-bucket1"},"object":{"key":"SampleData.csv","sequencer":"005A169422CA7CDF66"}}}]}

You can access the key as objectname = event['Records'][0]['s3']['object']['key'](Oops, this is for python) and then sent this info to RDS.