How to implement Canary for Kubernetes apps using Istio

In our last blog, we have described What is canary and why it is important for CI/CD. We also discussed the tools required for implementing canary- a traffic routing and an automated deployment tool. 

In this blog, we will focus on implementing canary deployment using Istio service mesh and Argo CD Rollouts (automated deployment tool). Please note, Argo Rollouts is not a pure play deployment tool, it is a progressive delivery tool used for implementing canary and blue/green strategy for Kubernetes applications. If you want to deploy using GitOps (we recommend GitOps for Kubernetes deployment), then check out Argo CD. 

Canary deployment steps to automate 

You can implement canary deployment by automating the 4 phases of canary deployment strategy (refer to Fig A). But for our blog, we will focus on step-1, and 2. 

1. Configure network using Istio: We will use Istio resources to specify the rules to split and route the traffic. 
2. Configure deployment using Argo Rollouts: We will create a yaml file of Argo Rollouts kind to deploy software (baseline version). 
3. Perform canary analysis software (out-of-scope): Analyzing metrics and logs of the canary to validate the new release usually involves aggregating the data from the monitoring system and manually analyzing it (you can also use AI-based solutions like OpsMx ISD). But this will remain out-of-scope of this blog. 

Fig A: Phases of Canary deployment in CI/CD

Configure network using Istio gateway 

Istio gateway helps in splitting the traffic in the run-time. Please note Istio provides two ways to implement canary:

• By treating two versions of an application- canary (new version) and stable as two separate services in Kubernetes. 
• By treating two versions as the subset of the single application. In this case service in Kubernetes will have two different Deployments attached to it. You can read the blog here.

For this blog, we will consider the former option, deploying a new application as a new version. 

To attain the traffic splitting, Istio offers CRDs- Istio Gateway and Virtual services. Please refer to the Istio gateway (named as imesh-gateway) and the virtual service (imesh-vsvc) resources that we have used to split the traffic. 

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: imesh-gateway
spec:
selector:
  istio: ingressgateway # using Istio IngressGateway
servers:
- port:
    number: 80
    name: http
    protocol: HTTP
  hosts:
  - "*"

The virtual service would like something below. If you notice we have referred to deployments called imesh-rollout (Rollouts kind) and services canary-svc and stable-svc.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: imesh-vsvc
  Namespace: deb-ns
spec:
  gateways:
  - imesh-gateway
  hosts:
  - imesh-rollout
  http:
  - name: primary        # referenced in canary.trafficRouting.istio.virtualService.routes
    route:
    - destination:
        host: stable-svc # referenced in canary.stableService
      weight: 100
    - destination:
        host: canary-svc # referenced in canary.canaryService
      weight: 0

You can use the following command to deploy the Istio gateway and virtual service resources. 

kubectl apply -n deb-ns -f imesh-gateway.yamlkubectl apply -n deb-ns -f imesh-vsvc.yaml

Configure deployments using Argo Rollouts 

Argo Rollouts is an open source software under the Argo project (graduated by CNCF in Dec, 2022). Argo Rollouts offers a Kubernetes controller and a set of custom resources (CRDs) to implement progressive deliveries such as canary and blue/green. Two resources which are important for implementing canary are:

• Rollouts: Rollouts is a workload resource which is an extended version of ReplicaSet workloads resource. Read about Rollouts here.
• Analysis Template: For automating the analysis (this is out of scope), one can use Analysis Template to define various ways to perform canary. For example fetching data from the monitoring tools (like Prometheus, Appdynamics) and logging tools (like Splunk, Sumo logic).

You can use the following Rollouts yaml to deploy an application. If you notice we have specified the two services called canary-svc and stable-svc ( we will see in the following section).

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: imesh-rollout
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.15.4
        ports:
        - containerPort: 80
  minReadySeconds: 30
  revisionHistoryLimit: 3
  strategy:
    canary:
      canaryService: canary-svc 
      stableService: stable-svc  # required
      trafficRouting:
        istio:
          virtualService:
            name: imesh-vsvc   # required
            routes:
            - primary  
      steps:
      - setWeight: 50   # overrides the weight provided in virtualservice
      - pause:
          duration: 1h # 1 hour

You can use the following command to deploy the rollout resource.

kubectl apply -n deb-ns -f rollout.yaml

After this we will configure the two services- canary-svc and stable-svc. 

Configuring services in Kubernetes canary and stable versions

Now we will configure services in the same Kubernetes cluster to point to the deployments/ Rollouts (imesh-rollout). We have configured the following in rollout-services.yaml. 

apiVersion: v1
kind: Service
metadata:
  name: canary-svc
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: imesh-rollout
 
---
apiVersion: v1
kind: Service
metadata:
  name: stable-svc
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: imesh-rollout
   

We can create the services by executing the command:

kubectl apply -n deb-ns -f rollout-services.yaml

The canary deployment is configured.

Once we will update the image in the Argo Rollouts, then the new deployment will happen using canary strategy. You can update the image using the following command:

kubectl argo rollouts set image imesh-rollout nginx=nginx:1.15.5

Once you update the image to the new version, then Istio will direct Istio to route only 5% of the traffic to canary. And after 1 hour, the traffic will be increased by another 5%. If not rolled back, then the amount of traffic to the new version will be increased to 100% gradually. 

The output will look something like below:

$ kubectl argo rollouts get rollout imesh-rollout

Name:            imesh-rollout
Namespace:       deb-ns
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/2
  SetWeight:     50
  ActualWeight:  50
Images:          docker.io/nginx:1.15.4 (stable)
                docker.io/nginx:1.15.5 (canary)
Replicas:
  Desired:       2
  Current:       3
  Updated:       1
  Ready:         3
  Available:     3

NAME                                       KIND        STATUS     AGE  INFO
⟳ imesh-rollout                            Rollout     ॥ Paused   57m 
├──# revision:2                                                       
│  └──⧉ imesh-rollout-7sk234kd90           ReplicaSet  ✔ Healthy  6s  canary
│     └──□ imesh-rollout-7sk234kd90-bp5lv  Pod         ✔ Running  6s  ready:1/1
└──# revision:1                                                       
  └──⧉ imesh-rollout-90sdfq3f89           ReplicaSet  ✔ Healthy  56m  stable
      ├──□ imesh-rollout-90sdfq3f89-xb1zf  Pod         ✔ Running  56m  ready:1/1
      └──□ imesh-rollout-90sdfq3f89-ws97n  Pod         ✔ Running  56m  ready:1/1

If you notice Argo Rollout has created new pods under canary ( 6 seconds ago) and while there were 2 other pods under the stable service (created 56 minutes ago). Argo Rollout which points to Istio virtual service, routes 50% of the traffic to canary and 50% to the stable version. It would wait for 1 hour before rolling out the canary fully i.e. routing 100% of the traffic. If you want to stop the deployment based on canary analysis, you can easily do so by rolling back to the stable version from the Argo UI or using kubectl command.

Conclusion

Implementing the Canary deployment strategy is one of the important goals for architects and DevOps team. And adopting a service mesh not only helps in making releasing software safe and risk-free but also opens a new door of opportunities such as security, network management and cross-cluster observability. 

IMESH offers an enterprise version of Istio service mesh for implementing security (mTLS) and abstracting the network management from developers core work. 
In case you are evaluating service mesh and want to understand the cost saving and value from Istio, talk to one of our Istio experts for free. If you want to implement Istio or want us to provide life-cycle management features, then contact us.