Upgrade Istio 1.10 to 1.16 using Helm + Terraform

Istio is a popular open-source service mesh that provides a unified control plane for managing microservices in a Kubernetes cluster. Upgrading Istio from one version to another is a critical process that ensures your microservices are running with the latest features and bug fixes. In this blog, we’ll walk you through the steps required to upgrade Istio from version 1.10 to 1.16. We will be using istio upgrade with downtime using helm upgrade.

Recommended upgrade to 2 minor version at a time

Step 1: Backup the Istio Configuration

Before upgrading Istio, it’s essential to take a backup of the existing Istio configuration. This backup can be helpful in case you face any issues during the upgrade process, and you need to roll back to the previous version.

To backup Istio configuration, run the following command:

kubectl get configmap istio -n istio-system -o yaml > istio-config.yaml

This command will create a YAML file named istio-config.yaml, which contains the current Istio configuration.

Step 2: Install Istio Binary file

Download the Istio 1.16 release files:

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.16.0 sh -
cd istio-1.16.0
export PATH="/Users/nirav/istio/istio-1.16.0/bin:$PATH"

Step 3: Check upgrade compatibility

istioctl version
client version: 1.16.0
control plane version: 1.10.2

istioctl x precheck
✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
  To get started, check out https://istio.io/latest/docs/setup/getting-started/

Step 4: Upgrade Istio crd

kubectl apply -f manifests/charts/base/crds

Step 5: Modify helm chart

Below changes done for the istio.tf file.

resource "helm_release" "istio_base" {
  chart            = "base"
  namespace        = "istio-system"
  name             = "istio-base"
  version          = var.istio_base_version
  create_namespace = true
  repository       = "https://istio-release.storage.googleapis.com/charts"
  # Add below changes to avoid errors while deployment
  skip_crds        = true
}

resource "helm_release" "istiod" {
  chart            = "istiod"
  namespace        = "istio-system"
  name             = "istiod"
  version          = var.istiod_version
  create_namespace = true
  repository       = "https://istio-release.storage.googleapis.com/charts"
  values = [
    templatefile("templates/istiod-chart-values.yaml", {
      ingresses = var.istio_ingresses
    })
  ]
  depends_on = [
    helm_release.istio_base
  ]
}

#custom ingress implementation
resource "helm_release" "istio_ingress" {
  for_each         = var.istio_ingresses
  # Verify chart name has changed to gateway
  chart            = "gateway"
  namespace        = "istio-system"
  name             = "${each.key}-istio-ingress"
  version          = var.istio_ingress_version
  create_namespace = true
  repository       = "https://istio-release.storage.googleapis.com/charts"
  values = [
    templatefile("templates/istio-ingress-chart-v2-values.yaml", {
      stage        = lookup(each.value, "stage", "")
      ssl-cert     = lookup(each.value, "ssl_cert", "")
      subnets      = lookup(each.value, "subnets", "")
      name         = each.key
      autoscaleMin = lookup(each.value, "min", "1")
      autoscaleMax = lookup(each.value, "max", "5")
      isInternal   = lookup(each.value, "internal", "true")
    })
  ]
  depends_on = [
    helm_release.istiod
  ]
}

Modify tfvars with latest istio version

Run terraform apply & verify new app version available

Step 6: Restart pods

The changes have been applied and istio upgraded. However istio proxy sidecar is yet an older version. To apply it to all the pods run the below command

for namespace in `kubectl get ns| awk '{print $1}'`; do for deployment in `kubectl get deployments -n $namespace | grep -v NAME | awk '{print $1}'`; do echo kubectl rollout restart deployment/$deployment -n $namespace; done; done

or readable like below:

for namespace in `kubectl get ns| awk '{print $1}'`; 
  do 
  for deployment in `kubectl get deployments -n $namespace | grep -v NAME | awk '{print $1}'`; 
    do 
    echo kubectl rollout restart deployment/$deployment -n $namespace; 
   done; 
done

Step 7: Revert skip_crds

skip_crds needed for upgrade setup only, hence it is advisable to revert the changes to support new environment creation with the same script.

resource "helm_release" "istio_base" {
  chart            = "base"
  namespace        = "istio-system"
  name             = "istio-base"
  version          = var.istio_base_version
  create_namespace = true
  repository       = "https://istio-release.storage.googleapis.com/charts"
  skip_crds        = false
}

Step 8: Verify the Istio Upgrade

After upgrading Istio to version 1.16, it’s essential to verify that everything is working as expected. You can use the following command to check the Istio status:

This command will analyze the Istio configuration and highlight any issues or errors.

istioctl analyze -A

You can also run some basic tests to ensure the microservices are running correctly.

Ignorable warnings

1. Ignore namespaces which do not require istio_proxy enabled

Info [IST0102] (Namespace coder) The namespace is not enabled for Istio injection. Run 'kubectl label namespace coder istio-injection=enabled' to enable it, or 'kubectl label namespace coder istio-injection=disabled' to explicitly mark it as not needing injection.

2. Ignore INSERT_FIRST for Envoy_filter

Warning [IST0151] (EnvoyFilter istio-system/filter-ratelimit) This EnvoyFilter does not have a priority and has a relative patch operation set which can cause the EnvoyFilter not to be applied. Using the INSERT_FIRST of ADD option or setting the priority may help in ensuring the EnvoyFilter is applied correctly.

3. Ignore if this pods does not need istio-proxy side car

Warning [IST0103] (Pod service-signer/signer) The pod service-signer/signer-xser is missing the Istio proxy. This can often be resolved by restarting or redeploying the workload.

4. In future deployments change sidecar.istio.io/inject to labels

Info [IST0135] (Pod test-autosuite/autop0mt-stackas-6c7f5bbbf7-hl6sb) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions.

5. In future deployments change port conventions

Info [IST0118] (Service monitoring/prometheus-prometheus-node-exporter) Port name metrics (port: 9100, targetPort: 9100) doesn't follow the naming convention of Istio port.

Actionable warnings

Warning [IST0105] (Pod test-autosuite/autop0mt-stackas-6c7f5bbbf7-hl6sb) The image of the Istio proxy running on the pod does not match the image defined in the injection configuration (pod image: docker.io/istio/proxyv2:1.10.2; injection configuration image: docker.io/istio/proxyv2:1.16.0). This often happens after upgrading the Istio control-plane and can be fixed by redeploying the pod.

Solution: Re-deploy pod/deployment

Conclusion

Upgrading Istio from version 1.10 to 1.16 requires a few critical steps, including backing up the existing Istio configuration, upgrading the Istio control plane and data plane, and verifying the upgrade. Following these steps can help ensure a smooth upgrade process and minimize the risk of issues or downtime.

References:

Istio 1.11 Upgrade Notes

Istio 1.12 Upgrade Notes

Istio 1.13 Upgrade Notes

Istio 1.14 Upgrade Notes

Upgrade with Helm