Azure AKS in-container logs to Azure Logs/Azure Sentinel

Question

Is there an dynamic way to pull log data from inside my containers?

All of my searches are returning that Azure Logs/Azure Sentinel can read data about AKS relative to the containers as they exist in K8s (online, running, failed, etc.) but not the actual in-container logs. Examples of results asking for this:

• https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query
• https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-livedata-overview
• https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/azure-monitor/containers/container-insights-enable-new-cluster.md#enable-monitoring-of-a-new-azure-kubernetes-service-aks-cluster

...all of these provide documentation on monitoring containers (as they live in K8s) but not the app-level logs in the containers...

Is anyone aware of a technology or capability for Azure Logs/Azure Sentinel to consume in-container, on-disk container logs (e.g. inside the container: /var/log, /var/application/logs, etc.)?

Thanks!

Answer 1

Ok so I'd forgotten about this question but just to post an answer. I kinda used what @kedmardemootoo suggested, where we use the standard log output from a container to read the file.

however, the main difference is I used the K8s sidecar pattern to monitor log files in my container so that I could do a kubectl logs output exclusively to a particular log file.

The manifest changes looks like:

while we're using a BusyBox image, Alpine would also work fine. The idea is that it's a tiny OS image so you don't explode your worker node processor/memory utilization but echo's a particular log's output to STDOUT so it can be captured by Azure Log Analytics (AZA). Use a tiny linux OS that just tails a particular log file (with a container name that reflects the log file being tail'd) and it will show up under that container's log entries in AZA.

FYI -- BusyBox and Alpine are like 3-5MB sized container images.

Answer 2

Assuming you're referring to linux containers. You only need to have have the OMS agent enabled and pointing to the right workspace and this gets the logs streamed over easily.

The ContainerLog table which would show you the same thing as kubectl logs <pod>. Anything that's sent to stdout and stderr from your container should be available in the Log Analytics Workspace. So if these are not being sent to either, you could just write a small script as part of your container, that would send those logs to stdout.

Here's how I'm able to get SMTP logs from my container: