AWS Datasync S3 -> S3 cross account, confused about destination role/account

Question

I want to use Datasync to copy data from a single S3 bucket in one account to a single S3 bucket in another account. I'm following this official AWS Datasync blog: https://aws.amazon.com/blogs/storage/how-to-use-aws-datasync-to-migrate-data-between-amazon-s3-buckets/ in the second section "Copying objects across accounts".

I've set up the source and destination buckets, and

• done the initial steps to "Create a new IAM role and attach a new IAM policy for the source S3 bucket location" and
• "Add the following trust relationship to the IAM role" (you can see where I mean in the blog by searching for those strings in quotes) but
• I'm now confused about which account to use to "Open the source S3 bucket policy and apply the following policy to grant permissions for the IAM role to access the objects" and
• which account to use to run the AWS CLI command "aws sts get-caller-identity" and
• then the "aws datasync create-location-s3" command straight after that. Am I doing those on the source or destination accounts?

The blog is a bit confusing and unclear on those specific steps and I can't find a simpler guide anywhere.

Answer 1

The source S3 bucket policy is attached to the source S3 bucket, so you'll need to log into the source account to edit that.

The next steps have to be done from the CLI. The wording is a bit ambiguous but the key phrase is "ensure you’re using the same IAM identity you specified in the source S3 bucket policy created in the preceding step." The IAM identity referenced in the example S3 bucket policy is arn:aws:iam::DEST-ACCOUNT-ID:role/DEST-ACCOUNT-USER so you need to be authenticated to the destination account for the CLI steps. The aws sts get-caller-identity command just returns the identity used to execute the command, so it's there to confirm that you're using the expected identity rather than being strictly required for setting up the datasync location.

It's not explicitly mentioned in the tutorial but of course the user in the destination account needs appropriate IAM permissions to create the datasync locations and task.

It may help to think of it this way: you need to allow a role in the destination account to access the bucket in the source account, then you're setting up the Datasync locations and tasks in the destination account. So anything related to Datasync config needs to happen in the destination account.

Answer 2

For me this did the trick but then I got another error

An error occurred (InvalidRequestException) when calling the CreateLocationS3 operation: DataSync location access test failed: could not perform s3:GetObject in bucket some-bucket. Access denied. Ensure bucket access role has s3:GetObject permission.

Even though the permissions were correctly set on the bucket/role. The problem was that the bucket was encrypted as well with a KMS key which I easily solved with the solution be found here.
I wish I have seen my answer here as I would have spent at least 1 hour less debugging

Answer 3

I had the same issue and got a fix tho I was receiving the same error message

"An error occurred (InvalidRequestException) when calling the CreateLocationS3 operation: Please provide a bucket in the us-east-2 region where DataSync is currently used".

You need to pass the --region flag and and put the region in which the source bucket is located. So look at the source bucket view and get the region.

Answer 4

DOES NOT WORK! - https://aws.amazon.com/blogs/storage/how-to-use-aws-datasync-to-migrate-data-between-amazon-s3-buckets/

1) DESTINATION ROLE

• create the necessary IAM roles in the account where the destination S3 bucket is located
• Log in to the destination account and create an IAM role for the AWS DataSync

  DESTINATION-ROLE:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Action:
              - sts:AssumeRole
            Effect: Allow
            Principal:
              Service:
                - datasync.amazonaws.com
      Path: '/'
      RoleName: DESTINATION-ROLE
  DESTINATION-POLICY:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: DESTINATION-POLICY
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Action:
              - s3:GetBucketLocation
              - s3:ListBucket
              - s3:ListBucketMultipartUploads
            Effect: Allow
            Resource: arn:aws:s3:::SOURCEBUCKET
          - Action:
              - s3:AbortMultipartUpload
              - s3:DeleteObject
              - s3:GetObject
              - s3:ListMultipartUploadParts
              - s3:PutObjectTagging
              - s3:GetObjectTagging
              - s3:PutObject
            Effect: Allow
            Resource: arn:aws:s3:::SOURCEBUCKET/*
      Roles:
        - Ref: DESTINATION-ROLE

2) SOURCE S3 POLICY

• "Copy the ARN for the IAM role you created for the source S3 bucket location.
• Now, log in to the source account.
• Open the source S3 bucket policy and apply the following policy to grant permissions for the IAM role

Version: '2012-10-17'
Statement:
  - Sid: BucketPolicyForDataSync
    Effect: Allow
    Principal:
      AWS:
        - arn:aws:iam::DEST-ACCOUNT-ID:role/DESTINATION-ROLE
    Action:
      - s3:GetBucketLocation
      - s3:ListBucket
      - s3:ListBucketMultipartUploads
      - s3:AbortMultipartUpload
      - s3:DeleteObject
      - s3:GetObject
      - s3:ListMultipartUploadParts
      - s3:PutObject
      - s3:GetObjectTagging
      - s3:PutObjectTagging
    Resource:
      - arn:aws:s3:::SOURCEBUCKET
      - arn:aws:s3:::SOURCEBUCKET/*

3) CLI INVOKATION

• Now, launch the AWS CLI (from the source account)...

aws datasync create-location-s3                       \
    --s3-bucket-arn    'arn:aws:s3:::SOURCEBUCKET'    \
    --region           'us-east-1'                    \
    --s3-config '
        {
            "BucketAccessRoleArn":
                "arn:aws:iam::DEST-ACCOUNT-ID:role/DESTINATION-ROLE"
        }
    '

3.1) ERROR Source Account

An error occurred (AccessDeniedException) when calling the CreateLocationS3 operation: Cross-account pass role is not allowed.

3.2) ERROR Destination Account

An error occurred (InvalidRequestException) when calling the CreateLocationS3 operation: Please provide a bucket in the us-east-1 region where DataSync is currently used.

Answer 5

What I saw works was to:

1. Run the CLI command in the destination account
2. When you are running the CLI command, you should make sure that the ARN of the user/botocore session with which you are running the CLI command has access to the source bucket with, as said by @gph, at least s3:ListBucket. What I found tricky: in case you do not run the CLI command with a user, but using some roles, you need to make sure to grant permission to the botocore session after assuming a role in the source account. Of course, the permission is as short-lived as the botocore session duration, but at least it helps you create the location if you don't want to log in with an IAM user.

Answer 6

If you are running this via AWS CLI using your own AWS user credentials for the source account, make sure that your user is granted ListBucket permissions in the destination bucket permissions. (You will probably add this from the AWS console in the destination account):

{
  "Sid": "UserDataSyncCreateS3Location",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::112233445566:user/my-user-name"
  },
  "Action": "s3:ListBucket",
  "Resource": "arn:aws:s3:::this-destination-bucket"
}