Unable to delete metrics with API from Prometheurs Pushgateway on Kubernetes Cluster

Question

Problem Description:

I am using a Prometheurs Pushgateway on a Kubernetes cluster, and I am experiencing an issue when trying to delete metrics. When the metrics on the Pushgateway increase, it appears to cause problems in the functionality of my service. To address this, I have created a Kubernetes cronjob to periodically delete all metrics related to the 'downloader' job.

Issue:

I'm using the following curl command to delete the metrics for the 'downloader' job:

curl -X DELETE 'http://localhost:9091/metrics/job/downloader'

However, although I receive a 202 response, the metrics do not get deleted from the Pushgateway.

I have also tested the request for fetching metrics, and it works without any issues. I have verified that there is no problem with the connection.

Answer 1

As per the Readme command line of Pushgateway , you need to use Delete Curl as below :

• Delete all metrics in the group identified {job="some_job",instance="some_instance"}: curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

• Delete all metrics in the group identified by {job="some_job"} (note that this does not include metrics in the {job="some_job",instance="some_instance"} group from the previous example, even if those metrics have the same job label):

curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job

• Delete all metrics in all groups (requires to enable the admin API via the command line flag --web.enable-admin-api):

curl -X PUT http://pushgateway.example.org:9091/api/v1/admin/wipe

Change your Delete command as per any of the above ones accordingly where Job and Instance need to be mentioned and have a try. This Delete method is used to delete metrics from the Pushgateway. The response code upon success is always 202. Sometimes a request is merely queued at that moment but it will make sure to delete the queued response first.