DropwizardMetricServices doesn't submit the gauge metric to JMX for second time (after removing the first time)

Question

DropwizardMetricServices#submit() I'm using doesn't submit the gauge metric for second time.

i.e. My use-case is to remove the gauge metric from JMX after reading it. And my application can send the same metric (with different value).

For the first time the gauge metric is submitted successfully (then my application removes it once it reads the metric). But, the same metric is not submitted the second time.

So, I'm a bit confused what would be the reason for DropwizardMetricServices#submit() not to work for the second time?

Below is the code:

Submit metric:

private void submitNonSparseMetric(final String metricName, final long value) {
    validateMetricName(metricName);
    metricService.submit(metricName, value); // metricService is the DropwizardMetricServices
    log(metricName, value);
    LOGGER.debug("Submitted the metric {} to JMX", metricName);
}

Code that reads and removes the metric:

protected void collectMetrics() {
    // Create the connection
    Long currTime = System.currentTimeMillis()/1000; // Graphite needs  
    Socket connection = createConnection();
    if (connection == null){
        return;
    }
    // Get the output stream
    DataOutputStream outputStream = getDataOutputStream(connection);
    if (outputStream == null){
        closeConnection();
        return;
    }
    // Get metrics from JMX
    Map<String, Gauge> g = metricRegistry.getGauges(); // metricRegistry is com.codahale.metrics.MetricRegistry
    for(Entry<String, Gauge> e : g.entrySet()){
        String key = e.getKey();
        if(p2cMetric(key)){
            String metricName = convertToMetricStandard(key);
            String metricValue = String.valueOf(e.getValue().getValue());

            String metricToSend = String.format("%s %s %s\n", metricName, metricValue, currTime);
            try {
                writeToStream(outputStream, metricToSend);
                // Remove the metric from JMX after successfully sending metric to graphite
                removeMetricFromJMX(key);
            } catch (IOException e1) {
                LOGGER.error("Unable to send metric to Graphite - {}", e1.getMessage());
            }
        }

    }

    closeOutputStream();
    closeConnection();
}

Answer 1

I think I found the issue.

As per the DropwizardMetricServices doc - https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/actuate/metrics/dropwizard/DropwizardMetricServices.html#submit-java.lang.String-double- , submit() method Set the specified gauge value. So, I think it's recommended to use DropwizardMetricServices#submit() method to only set the values of any existing gauge metric in JMX and not for adding any new metric to JMX.

So, once I replaced DropwizardMetricServices#submit() with MetricRegistry#register() (com.codahale.metrics.MetricRegistry) method to submit all my metrics it worked as expected and my metrics are readded to JMX (once they were removed by my application).

But, I'm just wondering what makes DropwizardMetricServices#submit() to only add new metrics to JMX and not any metric that's already been removed (from JMX). Does DropwizardMetricServices cache (in memory) all the metrics submitted to JMX? that makes DropwizardMetricServices#submit() method not to resubmit the metric?