Debezium server docker cannot access AWS credentials

Question

My goal is to use Debezium for real-time CDC from MongoDB to AWS Kinesis. I am using the official Debezium server 1.8 docker image. I'm able to run the image locally and I've configured the settings via environment variables that I am passing into the docker container. Following the official documentation for debezium server.

It runs for a bit but then seems to fail when it tries to connect to AWS and can't find my credentials.

Error:

software.amazon.awssdk.core.exception.SdkClientException: Profile file contained no credentials for profile 'default': ProfileFile(profiles=[])
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.lambda$new$1(ProfileCredentialsProvider.java:76)
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.<init>(ProfileCredentialsProvider.java:73)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.<init>(ProfileCredentialsProvider.java:43)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider$BuilderImpl.build(ProfileCredentialsProvider.java:211)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.create(ProfileCredentialsProvider.java:113)
        at io.debezium.server.kinesis.KinesisChangeConsumer.connect(KinesisChangeConsumer.java:82)
        at io.debezium.server.kinesis.KinesisChangeConsumer_Bean.create(Unknown Source)
        at io.debezium.server.kinesis.KinesisChangeConsumer_Bean.create(Unknown Source)
        at io.debezium.server.DebeziumServer.start(DebeziumServer.java:119)
        at io.debezium.server.DebeziumServer_Bean.create(Unknown Source)
        at io.debezium.server.DebeziumServer_Bean.create(Unknown Source)
        at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:101)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:29)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:26)
        at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
        at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
        at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:26)
        at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:18)
        at io.debezium.server.DebeziumServer_ClientProxy.arc$delegate(Unknown Source)
        at io.debezium.server.DebeziumServer_ClientProxy.arc_contextualInstance(Unknown Source)
        at io.debezium.server.DebeziumServer_Observer_Synthetic_d70cd75bf32ab6598217b9a64a8473d65e248c05.notify(Unknown Source)
        at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:320)
        at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:302)
        at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:73)
        at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:128)
        at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:97)
        at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
        at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
        at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
        at io.quarkus.runtime.Application.start(Application.java:101)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:103)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:41)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:120)
        at io.debezium.server.Main.main(Main.java:15)

Here's what I've tried:

• Pass credentials as environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
• Mounting my local AWS credentials file to the docker container: docker run -it --name debezium -p 8080:8080 --env-file .env -v $PWD/conf:/debezium/conf -v $PWD/data:/debezium/data -v "C:\Users\jon\.aws\credentials":/root/.aws/credentials:ro debezium/server bash
• ^Related to above, tried setting the environment variable AWS_CREDENTIAL_PROFILES_FILE=/root/.aws/credentials
• Logged into the docker container once it's running and confined the environment variables are there and the credentials file was there.

My .env file:

DEBEZIUM_SINK_TYPE=kinesis
DEBEZIUM_SINK_KINESIS_REGION=us-east-1
DEBEZIUM_SINK_KINESIS_CREDENTIALS_PROFILE=default
DEBEZIUM_SINK_KINESIS_ENDPOINT=test

DEBEZIUM_SOURCE_CONNECTOR_CLASS=io.debezium.connector.mongodb.MongoDbConnector
DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME=data/offsets.dat
DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS=0

DEBEZIUM_SOURCE_MONGODB_NAME=test
DEBEZIUM_SOURCE_MONGODB_HOSTS=rs0/test-mongodb-shard-00-00.test.mongodb.net:27017
DEBEZIUM_SOURCE_MONGODB_USER=test
DEBEZIUM_SOURCE_MONGODB_PASSWORD=test
DEBEZIUM_SOURCE_MONGODB_SSL_ENABLED=true

DEBEZIUM_SOURCE_DATABASE_INCLUDE_LIST=test
DEBEZIUM_SOURCE_COLLECTION_INCLUDE_LIST=test.test

DEBEZIUM_SOURCE_CAPTURE_MODE=change_streams_update_full
DEBEZIUM_SOURCE_DATABASE_HISTORY=io.debezium.relational.history.FileDatabaseHistory
DEBEZIUM_SOURCE_DATABASE_HISTORY_FILE_FILENAME=history.dat

QUARKUS_LOG_CONSOLE_JSON=false

AWS_ACCESS_KEY_ID=test
AWS_SECRET_ACCESS_KEY=test
AWS_CREDENTIAL_PROFILES_FILE=/root/.aws/credentials

My AWS credentials file:

[default]
aws_access_key_id = test
aws_secret_access_key = test

According to the documentation for this AWS class, either of the above methods should work: https://www.javadoc.io/doc/software.amazon.awssdk/auth/2.13.54/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html

Answer 1

I had the same problem. I ended up solving it by opening a shell in the container (eg using docker exec -it {container id} /bin/bash) and realizing that the user was not root but jboss. When I mounted the credentials file to /home/jboss/.aws/credentials, it started working.