gce_instance monitoring and logging Disk Usage in bytes works for a few minutes then breaks

Question

I have followed the tutorials and successfully installed the monitoring and logging agents on my debian9 machine. All statuses ok.

In metrics explorer the gce_instance Disk Usage in bytes works for a few minutes then breaks. I get the following error on my machine:

Aug 04 15:43:23 master collectd[13129]: write_gcm: Unsuccessful HTTP request 400: {
                                              "error": {
                                                "code": 400,
                                                "message": "Field timeSeries[2].points[0].interval.s
    tart_time had an invalid value of \"2020-08-04T07:43:22.681979-07:00\": The start time must be before th
    e end time (2020-08-04T07:43:22.681979-07:00) for the non-gauge metric 'agent.googleapis.com/agent/api_r
    equest_count'.",
                                                "status": "INVALID_ARGUMENT"
                                              }
                                            }
Aug 04 15:43:23 master collectd[13129]: write_gcm: Error talking to the endpoint.
Aug 04 15:43:23 master collectd[13129]: write_gcm: wg_transmit_unique_segment failed.
Aug 04 15:43:23 master collectd[13129]: write_gcm: wg_transmit_unique_segments failed. Flushing.

EDITED Anyone experiencing these issues, it's a confirmed bug now. I issued a support ticket in google issue tracker

Answer 1

These error messages are harmless, you are not losing metrics so you can ignore them without any problem.

The root cause is a server-side config change and affects all agents. That change only affected the verbosity of the responses, not the processing of the requests. some of the incoming metrics were silently dropped before that change, and are now dropped noisily.

There is a issue tracker where you can see more details about the issue that are affecting you.