Monitoring Azure AKS pod logs with SignalFX w/out Log Observer

Question

Is there any alternative way to monitor AKS Pod logs without Log Observer? I tried to implement this workaround using storage account/event hub, but neither does store application logs from pods. I'm able to get application logs easily from log analytics, using this query:

ContainerLog
| join kind = inner KubePodInventory on $left.ContainerID == $right.CointainerID
| where Namespace == "default" and LogEntry contains "error"
| project TimeGenerated, LogEntry, ContainerName

but not from logs stored in storage account or event hub. Is it actually possible to monitor pod logs in signalfx w/out log observer?

Answer 1

It's indeed possible to monitor AKS pod logs in signalfx without log observer. I've been sending them to Splunk Observabilty Cloud (SignalFX) using otel agent for months.

You can install the splunk-otel-collector helm chart and tune parameters to specify what kind of data you want to send (metrics, logs, traces). The values file can give you a rough image of the capabilities: see https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/values.yaml

And this is the chart's repo: https://github.com/signalfx/splunk-otel-collector-chart

splunkObservability:
[...]
  logsEnabled: true