Concourse Pipeline Not Retrieving Configuration Values from AWS Secrets Manager

Question

I'm experiencing a problem where my Concourse pipeline isn't picking up configuration values from AWS Secrets Manager. I have set up everything according to the documentation, and the necessary IAM roles and permissions are in place, but the values expected from Secrets Manager are not being fetched or recognized during the pipeline's job executions.

Here's an overview of my setup and what I've done so far:

1. I've verified that the IAM roles and policies attached to my Concourse components have the necessary permissions to access Secrets Manager.
2. The secret identifiers in my pipeline configuration match those stored in Secrets Manager. The specific behavior I'm observing is that the pipeline doesn't seem to recognize or fetch the values from Secrets Manager, leading to job failures or misconfigurations due to the absence of these essential values.

Here's a snippet of the test program I'm using:

jobs:
- name: test-secret-fetch
  plan:
  - task: use-secret
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: alpine}
      run:
        path: sh
        args:
        - -c
        - |
          echo "Using secret: $SECRET"
      params:
        SECRET: ((my-secret-key))

And I've set the following secret in AWS Secrets Manager:

aws secretsmanager create-secret --name "/concourse/sandbox/test-secret/my-secret-key" --secret-string "my-secret-value"

However, when I trigger the pipeline, the expected secret isn't being retrieved. Below is a redacted version of the Concourse value file I'm using:

imagePullSecrets: ["org-registry"]
persistence:
  enabled: true
  worker:
    size: "100Gi"
secrets:
  create: true
  localUsers: "ciuser:REDACTED"
  awsSecretsmanagerAccessKey: "REDACTED"
  awsSecretsmanagerSecretKey: "REDACTED"
  # ... other secrets ...
concourse:
  web:
    # ... configuration ...
    awsSecretsManager:
      enabled: true
      region: "eu-west-2"
      accessKey: REDACTED
      secretKey: REDACTED
      pipelineSecretTemplate: "/concourse/{{.Team}}/{{.Pipeline}}/{{.Secret}}"
    # ... additional configuration ...
    env:
      - name: AWS_SDK_LOAD_CONFIG
        value: "true"
      - name: CONCOURSE_AWS_SECRETSMANAGER_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: aws-credentials
            key: access-key-id
      - name: CONCOURSE_AWS_SECRETSMANAGER_SECRET_KEY
        valueFrom:
          secretKeyRef:
            name: aws-credentials
            key: secret-access-key

We're deploying using the following Helm command:

helm install cicd concourse/concourse -f concourse-values-final.yaml -n cicd --version 17.2.0 --debug

My questions are:

Are there any known issues with Concourse's integration with AWS Secrets Manager that might be causing this behavior? Is there a potential caching mechanism or a delay in secret updates from AWS Secrets Manager that I need to account for? What further diagnostic steps can I take to more precisely identify the cause of this issue? Any insights or advice on this matter would be greatly appreciated.