Trigger alert when expected associated event is not present

Question

I have two events that should appear in the logs to represent the completion of a transaction. I need to set up an alert when a transaction starts but no completion log entry is found.

What I've done so far is created two user-defined log metrics:

• order_form_starts denotes the submission of an order.
• order_form_sents denotes the successful transmission of items of an order.
  • This entry should be present 1 or more times per ..._start.

I'm able to query the the events individually but not sure how I can express an alert that should be triggered for the following "BAD" scenarios:

Some examples of possibilities:

• GOOD: start(order_id=1), end(order_id=1)
• GOOD: start(order_id=1), end(order_id=1), end(order_id=1)
• BAD: start(order_id=1)
• BAD: start(order_id=1), end(order_id=2)

Start Event:

fetch global::logging.googleapis.com/user/order_form_starts
| group_by [resource.project_id], row_count()

End Event:

fetch global::logging.googleapis.com/user/order_form_sents
| group_by [resource.project_id], row_count()

Answer 1

I don't think that you can express it that simply. If I had to implement this, I would create a cloud function that is querying (recurring) the logs for the events and write the result as a metric (gauge) into cloud monitoring.

From there you can define a threshold for that metric and create an alert.

You also might want to create traces with Google Cloud Trace for your transactions where you can keep track of the transactions.