Optional attributes - DataDog monitor

Question

I'm programming a DataDog monitor, and I want to include the attributes of the event in the notification message. My query looks like this:

logs("service:<SERVICE NAME> @level:ERROR @logger_name:<LOGGER NAME>").index("main").rollup("count").by("@stack_trace,@logger_name,@organization").last("5m") > 0

Here's my problem: some of the errors that I want to monitor don't have the attributes listed in the query.

I'm looking to make all the attributes optional: stack_trace, logger_name & organization. I want my monitor to look for errors, then if they have the attributes, show them in the notification. If not, I want there still to be a notification but a question mark is printed instead of the attribute.

Is that possible?

Answer 1

Here's what I can up with. It's not an ideal solution, but it works...

To be able to use the log attributes in the notification message, you need to include them in the monitor's query. To make the attributes optional, I used a redundant bi-condition, i.e. @organization:* OR NOT @organization:*. This query returns logs that have the @organization attribute, and logs that don't.

My full query ended up looking like this:

logs("service:maas-ep-core maas_id:production @level:ERROR @logger_name:com.solace.* @stack_trace:(* OR -*) @logger_name:(* OR -*) @organization:(* OR -*)").index("main").rollup("count").last("5m") > 0

It's pretty messy, but that's the best solution I could come up with.

If the attributes are not included in the log, they are left as a blank string. Knowing this, you can check for a blank string, and replace it with a ❓ for easier formatting.

So in the notification message text box, I had the following code:

Host.numeric_project_id should be here: {{host.numeric_project_id}}

#is_match {{#is_match "some_random_log_attribute" "" }} This will be displayed if some_random_log_attribute is present {{/is_match}} 

^is_match {{^is_match "some_random_log_attribute" "" }} This will show if some_random_log_attribute is blank {{/is_match}}