I am having issues when setting the Istio PeerAuthentication policy to STRICT cluster-wide. I am setting this in the istio-system namespace, so that the entire service mesh is forced to communicate only with mTLS. I am seeing an issue in the Kiali dashboard, where I can view the services/apps/workloads that are deployed in the cluster, but the graph is empty (unless i show idle nodes) and it looks like I do not have any info about the traffic in the cluster.
I guess my main questions are:
- Where does Kiali get the traffic data from? I thought it was Prometheus, but I do not see any errors regarding the Kiali -> Prometheus connection. Any ideas/suggestions on where to look for this info would be greatly appreciated.
- Is it generally recommended to enable mTLS this way (setting it to strict throughout the entire service mesh?). There will obviously be the observability tools mentioned here (Kiali, Prometheus, etc.), and then business applications. Is it best practice to have both of these types of applications apart of the service mesh, and forcing mTLS?
- How can I tell that the issue is with Kiali vs. Prometheus? What metrics can I search for in Prometheus to make sure that it is collecting the correct data? For example, I do not see the "istio_total_requests" metric in the istio dashboard. Is this enough to say that the issue is with Prometheus rather than Kiali?
- Can istiod be accessed when STRICT mTLS is enabled in the istio-system namespace? istiod is not part of the mesh, as it is in the istio-system namespace. Is this causing an issue with services trying to access it, when they are in other namespaces, apart of the mesh, due to the STRICT policy?
I have taken out the PeerAuthentication policy, and things begin to work, so I am assuming there is some issue with one of the tools mentioned due to this policy, but I cannot pinpoint what that may be at this time.