I have serveral dapr sidecars self-hosted with a docker compose that has mtls enabled. I am generating certificates for each sidecar, and the sidecar in the docker-compose has an .env file with the environmental variables needed. When i use Sentry for one sidecar everything works as expected.
However, when I'm trying to setup the sentry service for two sidecars I get some errors, because of the --issuer-credential path the sentry service are looking for the certificates at "wrong" place. Below is the sentry in the docker-compose, where inside my local directory ./Dapr/Certs there are two directories for my ServiceA and ServiceB with ca.crt, issuer.crt and issuer.key in each directory. I can confirm in my docker container that the directories are mounted inside dapr/certs/ServiceA & dapr/certs/ServiceB.
sentry:
image: "daprio/sentry"
container_name: sentry
command: ["./sentry",
"-issuer-credentials", "/dapr/certs",
"-trust-domain", "cluster.local",
"-config", "/dapr/config/config.yaml",
"-log-level", "debug",
]
volumes:
- "./Dapr/Certs/:/dapr/certs"
- "./Dapr/Config/:/dapr/config"
ports:
- "8082:8080"
servicea-dapr:
container_name: servicea-dapr
image: "daprio/daprd:latest"
command: ["./daprd",
"-app-id", "servicea",
"-app-port", "5000",
"-placement-host-address", "placement:50005",
"-components-path", "/dapr/components",
"-config", "/dapr/config/config.yaml",
"-log-level", "debug",
"--app-protocol", "https",
"-enable-mtls",
"-sentry-address", "sentry:50001",
]
volumes:
- "./Dapr/Components/:/dapr/components"
- "./Dapr/Config/:/dapr/config"
env_file:
- ./Dapr/Env/ServiceA.env
depends_on:
- servicea
network_mode: "service:servicea"
As i understood by looking at the Dapr Sentry source code as well it only looks in the issuer-credential path / ca.crt ... and is not capable of searching in directories inside the path you give. My question is then how can I set the sentry service to manage the certificates for the different sidecars? Or should there only be one set of certificates shared by all the sidecars!?
sentry logs:
2023-12-14 13:11:10 time="2023-12-14T12:11:10.582947762Z" level=info msg="Starting Dapr Sentry certificate authority -- version 1.12.2 -- commit f954254ed520409602024ea6d507d0405afa1f07" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.583022678Z" level=info msg="Log level set to: debug" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.583833845Z" level=info msg="metrics server started on :9090/" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.583913678Z" level=info msg="Healthz server is listening on :8080" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.583880678Z" level=info msg="Using local file system for trust bundle storage" instance=36e2b7fee46f scope=dapr.sentry.ca type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.583894637Z" level=info msg="Starting watch on filesystem directory: /dapr/certs" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.584060595Z" level=info msg="Root and issuer certs not found: generating self signed CA" instance=36e2b7fee46f scope=dapr.sentry.ca type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.584550762Z" level=info msg="Root and issuer certs generated" instance=36e2b7fee46f scope=dapr.sentry.ca type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.586061553Z" level=info msg="Self-signed certs generated and persisted successfully" instance=36e2b7fee46f scope=dapr.sentry.ca type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.586085178Z" level=info msg="CA certificate key pair ready" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.586111012Z" level=info msg="Adding validator 'insecure'" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.586140845Z" level=info msg="Using validator 'insecure'" instance=36e2b7fee46f scope=dapr.sentry type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.58641722Z" level=info msg="Security is initialized successfully" instance=36e2b7fee46f scope=dapr.runtime.security type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.58645597Z" level=info msg="Starting workload cert expiry watcher; current cert expires on: 2023-12-15 12:11:10 +0000 UTC, renewing at 2023-12-15 04:54:40 +0000 UTC" instance=36e2b7fee46f scope=dapr.runtime.security type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.586542928Z" level=info msg="Running gRPC server on port 50001" instance=36e2b7fee46f scope=dapr.sentry.server type=log ver=1.12.2
And the ServiceA logs, obviously as the Sentry service doesn't find the certificates the sidecar does not start..
2023-12-14 13:11:10 time="2023-12-14T12:11:10.68418922Z" level=info msg="Starting Dapr Runtime -- version 1.12.2 -- commit f954254ed520409602024ea6d507d0405afa1f07" app_id=webapi instance=be11d6897d1e scope=dapr.runtime type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.684223929Z" level=info msg="Log level set to: debug" app_id=webapi instance=be11d6897d1e scope=dapr.runtime type=log ver=1.12.2
2023-12-14 13:11:10 time="2023-12-14T12:11:10.684594179Z" level=info msg="Fetching initial identity certificate from sentry:50001" app_id=webapi instance=be11d6897d1e scope=dapr.runtime.security type=log ver=1.12.2
2023-12-14 13:29:05 time="2023-12-14T12:29:05.258906968Z" level=info msg="Received signal 'terminated'; beginning shutdown" app_id=webapi instance=be11d6897d1e scope=dapr.signals type=log ver=1.12.2
2023-12-14 13:29:05 time="2023-12-14T12:29:05.259549426Z" level=fatal msg="Fatal error from runtime: failed to retrieve the initial identity certificate: error establishing connection to sentry: context canceled: connection error: desc = \"transport: authentication handshake failed: x509svid: could not verify leaf certificate: x509: certificate signed by unknown authority\\nx509: certificate signed by unknown authority\"" app_id=webapi instance=be11d6897d1e scope=dapr.runtime type=log ver=1.12.2
Sidenote: I if use only one sidecar with the sentry service and set the --issuer-credential path directly where the certificates are loaded, everything works as expected...