Spring Boot app /prometheus endpoint taking a long time without a custom controller

Question

My application is using:

• spring-boot: 1.5.22.RELEASE
• spring-boot-starter-actuator: 1.5.22.RELEASE
• micrometer-spring-legacy: 1.3.5
• micrometer-registry-prometheus: 1.3.5

application.properties:

management.security.enabled=true
endpoints.health.sensitive=false
management.health.db.enabled=true
endpoints.prometheus.sensitive=false
endpoints.prometheus.enabled=true
endpoints.metrics.sensitive=false
endpoints.info.sensitive=false

Dockerfile (modified a bit to hide some info):

FROM tomcat:8-jdk8

RUN  apt-get update &&\
  apt-get -y --no-install-recommends install logrotate -o APT::Install-Suggests=0 -o APT::Install-Recommends=0 &&\
  rm -rf /var/lib/apt/lists/*

RUN rm -rf /usr/local/tomcat/webapps/ROOT
ADD my-service/target/my-service-*.war /usr/local/tomcat/webapps/ROOT.war

RUN echo "networkaddress.cache.ttl=60" | tee -a /opt/java/openjdk/jre/lib/security/java.security

RUN sed -i '/.*org.apache.catalina.valves.AccessLogValve[^\\/]*/a rotatable="false"' /usr/local/tomcat/conf/server.xml

ENV LOG_HOME "/log_home"
ENV UMASK 022

RUN chown -R myuser:myuser /usr/local/tomcat &&\
  mkdir /log_home && chown -R myuser:myuser /log_home &&\
  # useful for local execution
  mkdir -p /var/www/.aws && chown -R myuser:myuser /var/www/.aws

USER myuser
ENTRYPOINT ["catalina.sh", "run"]

My prometheus endpoint works fine takes around 20 ms on my local machine. When I deploy it on K8s and use Docker it takes around 12-18 seconds. Canary deployment also fails because the flagger advancement request takes more than 8 seconds whereas our pipelines threshold is 500 ms.

Things I tried but they didn't change the result:

1. upgraded micrometer-spring-legacy and micrometer-registry-prometheus to 1.3.20
2. set management.metrics.web.server.auto-time-requests=false in application.property
3. resolve any kind of dependency issue in pom.xml
4. removed/commented out any usage of io.micrometer.core.instrument.MeterRegistry

However, when I make a custom controller having a "/prometheus" endpoint, it passes the advancement request flagger checks and the canary deployment is successful.

import io.micrometer.prometheus.PrometheusMeterRegistry;
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.common.TextFormat;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.io.IOException;
import java.io.StringWriter;

@RestController
@RequestMapping("/prometheus")
public class CustomPrometheusController {

    private final PrometheusMeterRegistry prometheusRegistry;

    @Autowired
    public CustomPrometheusController(PrometheusMeterRegistry prometheusRegistry) {
        this.prometheusRegistry = prometheusRegistry;
    }

    @GetMapping
    public String getMetrics() throws IOException {
        CollectorRegistry cr = prometheusRegistry.getPrometheusRegistry();
        StringWriter writer = new StringWriter();
        TextFormat.write004(writer, cr.metricFamilySamples());
        String result = writer.toString();

        return result;
    }
}

Only the PrometheusController change has worked but I don't understand the reason. Why is it working with a manually defined endpoint? How can I resolve this issue?