Different behavior for "aws s3 ls" and "aws s3api list-objects"

Question

I am seeing different behavior for aws s3 ls and aws s3api list-buckets

Here is the 1st one:

$ aws s3 ls s3://demo.for.customers

Bucket: demo.for.customers
Prefix: 

      LastWriteTime     Length Name
      -------------     ------ ----
                           PRE 5CE4D191-FD14-4C85-8146-9FB8C29B7A7B/
                           PRE FFBC4675-F864-40E9-8AB8-BDF7A0437010/

So, I am able to list the objects inside the bucket demo.for.customers

Now, When I run the same thing using s3api, I get access denied:

$ aws s3api list-objects --bucket demo.for.customers
A client error (AccessDenied) occurred: Access Denied

QUESTION: Why do I get access denied for listing objects via s3api.

The reason I am raising this issues is that I get the same problem if I use AWS S3 Ruby SDK.

However, things are fine when I use aws s3 ls.

So the AWS S3 Ruby SDK and the aws s3api are exhibiting the same behavior. So, I am pasting only the aws s3api CLI issue here.

BTW, here is the IAM policy that has been applied to the user who is running all above commands:

{
  "Statement": [
    {
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::"
      ]
    },
    {
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::demo.for.customers"
      ],
      "Condition": {
        "StringEquals": {
          "s3:prefix": [
            "",
            "FFBC4675-F864-40E9-8AB8-BDF7A0437010/"
          ],
          "s3:delimiter": [
            "/"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::demo.for.customers/FFBC4675-F864-40E9-8AB8-BDF7A0437010/"
      ]
    }
  ]
}

And here is the ruby code which produces exact same error;

#!/usr/bin/ruby

require 'aws-sdk'
require 'awesome_print'
AWS.config( :access_key_id      => 'whatever', 
            :secret_access_key  => 'again whatever',
            :region             => 'us-west-2')

s3 = AWS.s3
buckets = s3.client.list_objects(:bucket_name => "demo.for.customers")
ap buckets

and the output is:

# ruby s3policies.rb 
/var/lib/gems/1.9.1/gems/aws-sdk-1.14.1/lib/aws/core/client.rb:366:in `return_or_raise': Access Denied (AWS::S3::Errors::AccessDenied)

Answer 1

Based upon the Role you have defined, a call to list-objects requires both a Prefix and a Delimiter.

The following command will work for you:

aws s3api list-objects --bucket demo.for.customers --prefix "" --delimiter "/"

If you remove the Delimiter condition within your ListBucket policy, then this will work for you:

aws s3api list-objects --bucket demo.for.customers --prefix ""

And if you also remove the Prefix condition, then this will work for you:

aws s3api list-objects --bucket demo.for.customers

A good way to test the above is to duplicate your Role Policy and progressively remove Conditions until it functions as expected.

Answer 2

Make sure from the location (directory) where you are running this utility has the correct permissions set.

The directory should have the correct ownership (owner should be the user running this utility not root or any other user)and have sufficient permissions for creating directories.