aws cli is unable to locate credentials while running gitlab pipeline

Question

I have a gitlab pipeline configuration with a job configured to use a docker image with aws cli and terraform. The job fails as the aws cli command could not find credentials. However when I use another image which has latest version of aws cli it works. I tried to log relevant information for both failing and successful jobs including environment variables but could not find out any major difference in the config or setting on the image causing this issue.

Below is the pipeline job definition and output for both the images. Could someone please help.

Job definition:

aws-credentials:
  image:
    name: ${BASE_IMAGE_REGISTRY_PATH}
  stage: credentials-check
  script:
    - cat ~/.aws/credentials || true
    - cat ~/.aws/config || true
    - aws --version
    - unset AWS_ACCESS_KEY_ID
    - unset AWS_SECRET_ACCESS_KEY
    - unset AWS_SESSION_TOKEN
    - unset AWS_ACCESS_KEY
    - unset AWS_SECRET_KEY
    - aws sts get-caller-identity

Output for working image:

$ cat ~/.aws/credentials || true
cat: can't open '/root/.aws/credentials': No such file or directory
$ cat ~/.aws/config || true
cat: can't open '/root/.aws/config': No such file or directory
$ aws --version
aws-cli/1.25.32 Python/3.9.16 Linux/5.10.186-179.751.amzn2.x86_64 botocore/1.27.32
$ unset AWS_ACCESS_KEY_ID
$ unset AWS_SECRET_ACCESS_KEY
$ unset AWS_SESSION_TOKEN
$ unset AWS_ACCESS_KEY
$ unset AWS_SECRET_KEY
$ aws sts get-caller-identity
{
    "UserId": "XXXXXXXXXX:botocore-session-12345678",
    "Account": "YYYYYYYYYY",
    "Arn": "arn:aws:sts::YYYYYYYYYY:assumed-role/ZZZZZZZ-role/botocore-session-12345678"
}
Uploading artifacts for successful job

Output for failing image:

$ cat ~/.aws/credentials || true
cat: can't open '/root/.aws/credentials': No such file or directory
$ cat ~/.aws/config || true
cat: can't open '/root/.aws/config': No such file or directory
$ aws --version
aws-cli/1.16.181 Python/3.8.15 Linux/5.10.186-179.751.amzn2.x86_64 botocore/1.12.171
$ unset AWS_ACCESS_KEY_ID
$ unset AWS_SECRET_ACCESS_KEY
$ unset AWS_SESSION_TOKEN
$ unset AWS_ACCESS_KEY
$ unset AWS_SECRET_KEY
$ aws sts get-caller-identity
Unable to locate credentials. You can configure credentials by running "aws configure".
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1