We are looking into integrating datadog with our druid deployment. I have gone through the datadog document for druid integration and added the following to my druid configuration:
druid_extensions_loadList: '["druid-parquet-extensions", "druid-histogram", "druid-s3-extensions","druid-datasketches", "druid-lookups-cached-global", "druid-multi-stage-query","postgresql-metadata-storage", "statsd-emitter", "druid-basic-security", "prometheus-emitter"]'
druid_emitter: statsd
druid_emitter_statsd_hostname: "127.0.0.1"
druid_emitter_statsd_port : "8125"
druid_emitter_statsd_dogstatsd: "true"
druid_emitter_statsd_dogstatsdServiceAsTag: "true"
Along with this in datadog-agent/conf.d/druid.d/conf I have added my router url to the same. I have also added the required header credentials for our url. I am still not seeing any druid metrics displaying on datadog. When checking the cluster agent status, I do not see any Druid integration and when running node agent status I only see the same in logs. Is there any main datadog key I need to change to start druid integration? Is there anything wrong with my Druid configuration that Druid for Datadog is not able to start? Thanks in advance. FYI I installed datadog agent via helm. My values file is also as follows:
datadog:
apiKeyExistingSecret: <>
appKeyExistingSecret: <>
containerExcludeLogs: <>
site: datadoghq.com
collectEvents: true
logs:
## @param enabled - boolean - optional - default: false
## Enables this to activate Datadog Agent log collection.
#
enabled: true
## @param containerCollectAll - boolean - optional - default: false
## Enable this to allow log collection for all containers.
#
containerCollectAll: true
clusterName: cosmos
dogstatsd:
useHostPort: true
port: 8125
nonLocalTraffic: true
To fix the issue, I have looked at my agent's dogstatsd command where I am not seeing agent dogstatsd status return anything, but I am not seeing that return anything on another cluster where I have metrics showing. I have tried changing the druid IP to status.host which didn't work. None of the agents are displaying dogstatsd running. I have also ensure that statsd extension is loading properly.