I am very new to ISTIO and would like to get clarified with my following doubts.
Details
Current AKS version 1.18.14
planning upgrade to AKS 1.19.11
Current istio version 1.7
Planning upgrade to 1.8
We are planning to upgrade Istio version from 1.7 to 1.8 in our AKS cluster 1.18.14 in our production.
But I am not sure about the proper method of upgrade to follow in production since there are multiple methods are given by Istio.
I don't have any clue about the how the current Istio setup is done in my environment and what profile settings we used as it was done long before. Could understand below are the steps followed to install istio..
Istio was installed following way:
Created manifest:
istioctl manifest generate --set profile=default -f /manifests/overlay/overlay.yaml > $HOME/generated-manifest.yaml
Installed istio:
istioctl install --set profile=default -f /manifests/overlay/overlay.yaml
Verified istio against the deployed manifest:
istioctl verify-install -f $HOME/generated-manifest.yaml
Is there any method to export all the existing settings (the one currently running) and do the upgrade?
So I am looking for a production ready approach to upgrade Istio with all existing settings in placed.
Important
Consider replicating the environment and performing the upgrade on dev/stage first to make sure it works for you and your infrastructure.
Check what exactly you have installed
Can be done by getting
installed state custom resource
and all settings:kubectl -n istio-system get IstioOperator installed-state -o yaml > installed-state.yaml
Below are steps based on official documentation to upgrade using
istioctl
From 1.7.3 to 1.8.6, this will be similar for other versions, however upgrades should be no more than 1 minor version of difference e.g. 1.5 to 1.6.
Available versions and releases can be checked in Istio Github.
1 - install
istioctl
version 1.8.6: Get required binaries:curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.8.6 TARGET_ARCH=x86_64 sh -
and copy
istiolctl
bin:sudo cp bin/istioctl /usr/local/bin/
2 - run
istioctl version
to confirmistioctl
version and control/data plane versions:3 - run
istioctl x precheck
to see if revision was set (it may be different if set - see warning at the end of section)There are two main upgrade strategies:
Vendor suggests to go with canary as it's more safe and can be tested before final migration.
4 - Create a backup:
Can be restored with:
5 - Control plane - install Canary version
Check it installed successfully by running following commands:
kubectl get pods -n istio-system -l app=istiod
kubectl get svc -n istio-system -l app=istiod
kubectl get mutatingwebhookconfigurations
6 - Data plane
istioctl proxy-status | grep $(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items..metadata.name}') | awk '{print $7}'
kubectl label namespace NAME_SPACE istio-injection- istio.io/rev=1-8-6
Once namespace(s) updated, pods need to be re-injected. This can be done by restarting them, e.g. with:
kubectl rollout restart deployment -n NAME_SPACE
Verify pods are now using
canary
istiod:istioctl proxy-status
7 - Uninstall old control-plane
Run:
istioctl x uninstall -f manifests/profiles/default.yaml
Check only canary control-plane is running:
kubectl get pods -n istio-system -l app=istiod
Other types of istio installation available:
Please get familiar with istio installation methods' pros and cons.
Useful links
Update
Moving this from comments. There are more challenges with updating from 1.7.3 to 1.8.6 istio versions. For removing current control-plane
-f
with previous manifest should be used. When applying for the same manifest to1.8.6
versions, there are errors aboutpolicy
andtelemetry
components:After digging, it appeared, even though api version is used the same -
v1alpha1
, newer version ofistioctl operator
can't validate manifest from1.7.3
.I took
installed-state.yaml
as it's described at the beginning of the asnwer from1.7.3
and1.8.6
istio installations and gotdiff
between them:policy
andtelemetry
components are completely missing in1.8.6
which explains the errors. Also there are some changes as well. Github link to the diff file, left is1.7.3
, right is1.8.6
.In that case it's probably impossible to upgrade without manual work with manifests:
1 - check if manifest which was applied is default or has changes. Get a default profile (note! istioctl should be used
1.7.3
):istioctl profile dump default > default-profile.yaml
2 - If manifest is default, then safely proceed to install
canary
with--set profile=default
.3 - Manifest is not default and has customization. Using
istioctl 1.8.6
get a dump of default profile:istioctl profile dump default > default-profile-186.yaml
"Adapt" it to current existing manifest and then proceed to install
canary
with-f
option andadapted
manifest.