In our last blog, we have described What is canary and why it is important for CI/CD. We also discussed the tools required for implementing canary- a traffic routing and an automated deployment tool.
In this blog, we will focus on implementing canary deployment using Istio service mesh and Argo CD Rollouts (automated deployment tool). Please note, Argo Rollouts is not a pure play deployment tool, it is a progressive delivery tool used for implementing canary and blue/green strategy for Kubernetes applications. If you want to deploy using GitOps (we recommend GitOps for Kubernetes deployment), then check out Argo CD.
Canary deployment steps to automate
You can implement canary deployment by automating the 4 phases of canary deployment strategy (refer to Fig A). But for our blog, we will focus on step-1, and 2.
- Configure network using Istio: We will use Istio resources to specify the rules to split and route the traffic.
- Configure deployment using Argo Rollouts: We will create a yaml file of Argo Rollouts kind to deploy software (baseline version).
- Perform canary analysis software (out-of-scope): Analyzing metrics and logs of the canary to validate the new release usually involves aggregating the data from the monitoring system and manually analyzing it (you can also use AI-based solutions like OpsMx ISD). But this will remain out-of-scope of this blog.
Fig A: Phases of Canary deployment in CI/CD
Configure network using Istio gateway
Istio gateway helps in splitting the traffic in the run-time. Please note Istio provides two ways to implement canary:
- By treating two versions of an application- canary (new version) and stable as two separate services in Kubernetes.
- By treating two versions as the subset of the single application. In this case service in Kubernetes will have two different Deployments attached to it. You can read the blog here.
For this blog, we will consider the former option, deploying a new application as a new version.
To attain the traffic splitting, Istio offers CRDs- Istio Gateway and Virtual services. Please refer to the Istio gateway (named as imesh-gateway) and the virtual service (imesh-vsvc) resources that we have used to split the traffic.
apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: imesh-gateway spec: selector: istio: ingressgateway # using Istio IngressGateway servers: - port: number: 80 name: http protocol: HTTP hosts: - "*"
The virtual service would like something below. If you notice we have referred to deployments called imesh-rollout (Rollouts kind) and services canary-svc and stable-svc.
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: imesh-vsvc Namespace: deb-ns spec: gateways: - imesh-gateway hosts: - imesh-rollout http: - name: primary # referenced in canary.trafficRouting.istio.virtualService.routes route: - destination: host: stable-svc # referenced in canary.stableService weight: 100 - destination: host: canary-svc # referenced in canary.canaryService weight: 0
You can use the following command to deploy the Istio gateway and virtual service resources.
kubectl apply -n deb-ns -f imesh-gateway.yamlkubectl apply -n deb-ns -f imesh-vsvc.yaml
Configure deployments using Argo Rollouts
Argo Rollouts is an open source software under the Argo project (graduated by CNCF in Dec, 2022). Argo Rollouts offers a Kubernetes controller and a set of custom resources (CRDs) to implement progressive deliveries such as canary and blue/green. Two resources which are important for implementing canary are:
- Rollouts: Rollouts is a workload resource which is an extended version of ReplicaSet workloads resource. Read about Rollouts here.
- Analysis Template: For automating the analysis (this is out of scope), one can use Analysis Template to define various ways to perform canary. For example fetching data from the monitoring tools (like Prometheus, Appdynamics) and logging tools (like Splunk, Sumo logic).
You can use the following Rollouts yaml to deploy an application. If you notice we have specified the two services called canary-svc and stable-svc ( we will see in the following section).
apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: imesh-rollout spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15.4 ports: - containerPort: 80 minReadySeconds: 30 revisionHistoryLimit: 3 strategy: canary: canaryService: canary-svc stableService: stable-svc # required trafficRouting: istio: virtualService: name: imesh-vsvc # required routes: - primary steps: - setWeight: 50 # overrides the weight provided in virtualservice - pause: duration: 1h # 1 hour
You can use the following command to deploy the rollout resource.
kubectl apply -n deb-ns -f rollout.yaml
After this we will configure the two services- canary-svc and stable-svc.
Configuring services in Kubernetes canary and stable versions
Now we will configure services in the same Kubernetes cluster to point to the deployments/ Rollouts (imesh-rollout). We have configured the following in rollout-services.yaml.
apiVersion: v1 kind: Service metadata: name: canary-svc spec: ports: - port: 80 targetPort: http protocol: TCP name: http selector: app: imesh-rollout --- apiVersion: v1 kind: Service metadata: name: stable-svc spec: ports: - port: 80 targetPort: http protocol: TCP name: http selector: app: imesh-rollout
We can create the services by executing the command:
kubectl apply -n deb-ns -f rollout-services.yaml
The canary deployment is configured.
Once we will update the image in the Argo Rollouts, then the new deployment will happen using canary strategy. You can update the image using the following command:
kubectl argo rollouts set image imesh-rollout nginx=nginx:1.15.5
Once you update the image to the new version, then Istio will direct Istio to route only 5% of the traffic to canary. And after 1 hour, the traffic will be increased by another 5%. If not rolled back, then the amount of traffic to the new version will be increased to 100% gradually.
The output will look something like below:
$ kubectl argo rollouts get rollout imesh-rollout Name: imesh-rollout Namespace: deb-ns Status: ॥ Paused Message: CanaryPauseStep Strategy: Canary Step: 1/2 SetWeight: 50 ActualWeight: 50 Images: docker.io/nginx:1.15.4 (stable) docker.io/nginx:1.15.5 (canary) Replicas: Desired: 2 Current: 3 Updated: 1 Ready: 3 Available: 3 NAME KIND STATUS AGE INFO ⟳ imesh-rollout Rollout ॥ Paused 57m ├──# revision:2 │ └──⧉ imesh-rollout-7sk234kd90 ReplicaSet ✔ Healthy 6s canary │ └──□ imesh-rollout-7sk234kd90-bp5lv Pod ✔ Running 6s ready:1/1 └──# revision:1 └──⧉ imesh-rollout-90sdfq3f89 ReplicaSet ✔ Healthy 56m stable ├──□ imesh-rollout-90sdfq3f89-xb1zf Pod ✔ Running 56m ready:1/1 └──□ imesh-rollout-90sdfq3f89-ws97n Pod ✔ Running 56m ready:1/1
If you notice Argo Rollout has created new pods under canary ( 6 seconds ago) and while there were 2 other pods under the stable service (created 56 minutes ago). Argo Rollout which points to Istio virtual service, routes 50% of the traffic to canary and 50% to the stable version. It would wait for 1 hour before rolling out the canary fully i.e. routing 100% of the traffic. If you want to stop the deployment based on canary analysis, you can easily do so by rolling back to the stable version from the Argo UI or using kubectl command.