Istio 1.1.6 - PassthroughCluster - Not Reaching Endpoint

811 Views Asked by At

I am using Istio 1.1.6, Prometheus 2.18.1, in conjunction with Seldon 1.1.0 (for Machine Learning); when Prometheus is attempting to scrape the endpoints, we have the following settings for the Istio ConfigMap:

outboundTrafficPolicy:
  mode: ALLOW_ANY

This should, as per the documentation, allow Prometheus to scrape any endpoint which does not have an associated Service associated with it (via VirtualServices). However, I consistently get the following alert blasting the access logs:

2020-05-27 12:56:43 
[2020-05-27T16:56:42.448Z] "- - -" 0 - "-" 247 0 0 - "-" "-" "-" "-" "10.51.105.4:6001" PassthroughCluster 10.51.105.4:35708 10.51.105.4:6001 10.51.106.2:51760 -
Show context
2020-05-27 12:56:43 
[2020-05-27T16:56:42.448Z] "- - -" 0 UO "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.51.105.4:6001 10.51.105.4:35708 -

Now, when I bash-exec into a container inside the pod, the destination, localhost:6001 is reachable, and is formatted for scraping by Prometheus.

In addition, the DestinationRule has no short-circuit limitations:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  generation: 1
  name: destination-pod
  ownerReferences:
  - apiVersion: machinelearning.seldon.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: SeldonDeployment
    name: destination
spec:
  host: destination-pod
  subsets:
  - labels:
      version: 41949a6
    name: pod

What might be the reason that Istio is appearing to block this?

0

There are 0 best solutions below