I am using Istio 1.1.6, Prometheus 2.18.1, in conjunction with Seldon 1.1.0 (for Machine Learning); when Prometheus is attempting to scrape the endpoints, we have the following settings for the Istio ConfigMap:
outboundTrafficPolicy:
mode: ALLOW_ANY
This should, as per the documentation, allow Prometheus to scrape any endpoint which does not have an associated Service associated with it (via VirtualServices). However, I consistently get the following alert blasting the access logs:
2020-05-27 12:56:43
[2020-05-27T16:56:42.448Z] "- - -" 0 - "-" 247 0 0 - "-" "-" "-" "-" "10.51.105.4:6001" PassthroughCluster 10.51.105.4:35708 10.51.105.4:6001 10.51.106.2:51760 -
Show context
2020-05-27 12:56:43
[2020-05-27T16:56:42.448Z] "- - -" 0 UO "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.51.105.4:6001 10.51.105.4:35708 -
Now, when I bash-exec into a container inside the pod, the destination, localhost:6001
is reachable, and is formatted for scraping by Prometheus.
In addition, the DestinationRule
has no short-circuit limitations:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
generation: 1
name: destination-pod
ownerReferences:
- apiVersion: machinelearning.seldon.io/v1
blockOwnerDeletion: true
controller: true
kind: SeldonDeployment
name: destination
spec:
host: destination-pod
subsets:
- labels:
version: 41949a6
name: pod
What might be the reason that Istio is appearing to block this?