How can I get notifications for all stderr logs sent to Google Cloud Logging?

Question

I'd like to get notifications for all standard error logs sent to Google Cloud Logging. Preferably, I'd like to get the notifications through Google Cloud Error Reporting, so I can easily get notifications on my phone through the GCP mobile app.

I've deployed applications to Google Kubernetes Engine that are writing logs to standard error, and GKE is nicely forwarding all the stderr logs to Google Cloud Logging with logName: "projects/projectName/logs/stderr"

I see the logs show up in Google Cloud Logging, but Error Reporting does not pick up on them.

I've tried troubleshooting as described here: https://cloud.google.com/error-reporting/docs/troubleshooting. But the proposed solutions revolve around formatting the logs in a certain way. What if I've deployed applications for which I can't control the log messages?

A (totally ridiculous) option could be to create a "logs-based metric" based on any log sent to stderr, then get notified whenever that metric exceeds 1.

What's the recommended way to get notified for stderr logs?

Answer 1

• If Error Reporting is not recognizing the stderr logs from your container it means they are not displayed in the correct format for this API to be able to detect them.

• Take a look at this guide on how to setup error reporting for GKE API

• There are other ways to do this with third party products like gSlack where basically you will export the stackdriver logs to pub/sub and then send them into the Slack channel with Cloud Functions.

• You can also try to do it by using Cloud Build and try to integrate it with GKE container logs.

Still, I think the best and easiest option will be by using the Monitoring Alert.

Answer 2

You can force the error by setting the @type in the context as shown in the docs. For some reason, even if this is the Google library, and it has code to detect that an Exception is thrown, with its stack trace, it won't recognize it as an error worth reporting.

I also added the service array to be able to identify my service in the error reporting.