docker push with AWS CodeBuild fails with exit status 1

Question

I created my CodeBuild project triggered by CodePipeline and the 'docker push' step always fails with the 'Reason: exit status 1' error message

Here are my build logs (replaced my org id by <MY_ORG_ID>):

[Container] 2021/06/12 14:39:47 Entering phase INSTALL
[Container] 2021/06/12 14:39:47 Phase complete: INSTALL State: SUCCEEDED
[Container] 2021/06/12 14:39:47 Phase context status code:  Message: 
[Container] 2021/06/12 14:39:47 Entering phase PRE_BUILD
[Container] 2021/06/12 14:39:47 Running command echo Logging in to Amazon ECR...
Logging in to Amazon ECR...

[Container] 2021/06/12 14:39:47 Running command aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin <MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[Container] 2021/06/12 14:39:51 Running command docker push <MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service:latest
The push refers to repository [<MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service]
An image does not exist locally with the tag: <MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service

[Container] 2021/06/12 14:39:51 Command did not exit successfully docker push <MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service:latest exit status 1
[Container] 2021/06/12 14:39:51 Phase complete: PRE_BUILD State: FAILED
[Container] 2021/06/12 14:39:51 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: docker push <MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service:latest. Reason: exit status 1

And here is my buildspec.yaml:

version: 0.2

env:
  git-credential-helper: yes
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin <MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com
  build:
    commands:
     - echo Pushing Docker image <MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service:latest
    - DOCKER_REPO=<MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com
    - IMAGE_TAG=${DOCKER_REPO}/reponame/core-service:${EKS_CLUSTER_NAME}-${CODEBUILD_RESOLVED_SOURCE_VERSION}-v${CODEBUILD_BUILD_NUMBER}
    - echo Set IMAGE TAG = $IMAGE_TAG
    - docker build --build-arg NODE_ENV=production --build-arg DOCKER_REPO=${DOCKER_REPO} -t $IMAGE_TAG core-service/.
  - docker push $IMAGE_TAG   

As many references point out, I have added this statement to the policy attached to the corresponding AWS CodeBuild service role but it still does not work.

{
  "Statement": [
    ### BEGIN ADDING STATEMENT HERE ###
    {
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:CompleteLayerUpload",
        "ecr:GetAuthorizationToken",
        "ecr:InitiateLayerUpload",
        "ecr:PutImage",
        "ecr:UploadLayerPart"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    ### END ADDING STATEMENT HERE ###
    ...
  ],
  "Version": "2012-10-17"
}

I can run these steps manually but it always gives me this error on CodeBuild.

Please, if you could help, there are similar threads out there but none could explain a solution for this one specifically. Thanks.

Answer 1

The issue could be the with policy which is not define access to ECR. that could help, but keep in mind it provides full access to ECR:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ecr:*",
            "Resource": "*"
        }
    ]
}

I am sure at least you need those:

"ecr:GetAuthorizationToken"
"ecr:BatchCheckLayerAvailability",
"ecr:BatchGetImage",
"ecr:CompleteLayerUpload",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart"

Some details might be found here

Some details about access can be found here as well

Answer 2

As you can see from the error message, trying to push the image with this tag was throwing an error:

<MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service:latest

That was happening because the image tag 'latest' already existed in my repo.

By running multiple times the same docker push command, I saw that sometimes CodeBuild would print the full error message and sometimes not. Pushing an unique image name tag solved the issue:

<MY_ORG_ID>.dkr.ecr.eu-west-2.amazonaws.com/reponame/core-service:${CODEBUILD_RESOLVED_SOURCE_VERSION}-v${CODEBUILD_BUILD_NUMBER}

Everything was fine with permissions and authentication.