AWS S3 C++ : List all the object when result is truncated

Question

I use following code to list all the keys in my S3 bucket.

I have around 15,000 objects in the bucket. However this code just loops through the first 1000 objects infinitly. Seems it doesnt respect the SetMarker() method.

Any tips on how to fix this ?

#include <aws/s3/S3Client.h>
#include <aws/s3/model/ListObjectsRequest.h>
#include <aws/s3/model/Object.h>

int main(int argc, const char* argv[])
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    {
        Aws::Client::ClientConfiguration config;
        config.region="ap-northeast-1";

        Aws::S3::S3Client s3_client(config);
        Aws::S3::Model::ListObjectsRequest objects_request;

        objects_request.WithBucket("MYBUCKETNAME").WithPrefix("some-prefox");

        bool isDone = false;
        bool isFailed= false;
        Aws::S3::Model::ListObjectsOutcome outcome;
        int c=0;

        while(!isDone) {
                 outcome=s3_client.ListObjects(objects_request);
                 if(!outcome.IsSuccess()) break;
                 //process
                 Aws::Vector<Aws::S3::Model::Object> object_list = outcome.GetResult().GetContents();

                 for (auto const &s3_object : object_list)
                {
                    std::cout << "* " << s3_object.GetKey() << std::endl;
                    c++;

                }
                std::cout<<"--------- Break"<<c<<"\n";
                 isDone=!outcome.GetResult().GetIsTruncated();
                 if(!isDone) {
                    objects_request.SetMarker(outcome.GetResult().GetNextMarker());

                }
            }
            std::cout << "Count "<<c<<"\n";
            //check isFailed
            if(!outcome.IsSuccess()) {
                std::cout << "ListObjects error: " <<
                outcome.GetError().GetExceptionName() << " " <<
                outcome.GetError().GetMessage() << std::endl;
            }

    }

    Aws::ShutdownAPI(options);
}

Answer 1

According to the documentation for NextMarker:

Note

This element is returned only if you specify a delimiter request parameter. If the response does not include the NextMarker and it is truncated, you can use the value of the last Key in the response as the marker in the subsequent request to get the next set of object keys.

Your code should therefore be:

if(!isDone) {
  objects_request.SetMarker(outcome.GetResult().GetContents().back().GetKey());
}

Also note that the V1 ListObjects method is deprecated, you should be using ListObjectsV2 which uses explicit continuation tokens which are a bit easier to use.