Opentelemetry cant connect to Prometheus on Kubernetes

Question

I have a kubernetes cluster with apps in .net 7 installed . The cluster also has OpenTelemetry as a deployment and Prometheus server installed.

The opentelemetry has a values file as show below

mode: deployment presets:   # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines   kubernetesAttributes:
    enabled: true   # enables the kubeletstatsreceiver and adds it to the metrics pipelines   kubeletMetrics:
    enabled: true   # Enables the filelogreceiver and adds it to the logs pipelines   logsCollection:
    enabled: true

config:
     exporters:
    otlphttp/metrics:
      endpoint: http://prometheus-server.cmb.svc.cluster.local:9090
      tls:
         insecure: true
    prometheusremotewrite:
      endpoint: http://prometheus-server.cmb.svc.cluster.local:9091/api/v1/write
      tls:
         insecure: true   service:
    pipelines:      
      metrics:                
        exporters:
          - otlphttp/metrics

the Opentelemetry pods starts and runs just fine and is able to receive metrics from all apps on the cluster but it fails to write to the Prometheus cluster. The error generated is given below

2024-03-20T14:54:29.219Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/metrics", "error": "failed to make an HTTP request: Post "http://prometheus-server.cmb.svc.cluster.local:9090/v1/metrics": context deadline exceeded", "interval": "6.716454422s"}