Service Mesh Microservices Architecture from Development to Canary Release Best Practices (Part 2)¶
Introduction¶
This article will introduce the overall situation of canary release, using Guance for observability of microservices' Metrics, traces, and logs. All operations related to Rancher in this guide are performed on the k8s-solution-cluster cluster, and will not be repeated.
Canary Release¶
To achieve a canary release, add the label app=reviews to the Deployment of the microservice to distinguish the service name. For the first deployment version, add the label version=v1, and for the second deployment version, add the label version=v2. This allows controlling the traffic distribution between versions based on labels. For example, after deploying v2, direct 90% of the traffic to v1 and 10% to v2. Once verified, switch all traffic to v2 and decommission v1. The entire release process is complete.
Step One: Delete reviews¶
In the previous operation, to explain Gitlab-CI automated deployment, three versions of reviews were deployed. Before this operation, you need to delete these three deployments. Log in to 'Rancher', navigate through 'Workloads' -> 'Deployments' to find reviews-v1, and select 'Delete' on the right side. Then similarly delete reviews-v2 and reviews-v3.
Step Two: Deploy reviews-v1¶
Log in to 'GitLab', find the bookinfo-views project, change the value of APP_VERSION in the .gitlab-ci.yml file to "v1", and commit the code. Log in to 'Rancher', navigate through 'Workloads' -> 'Deployments' to see that reviews-v1 has been deployed.
Step Three: Create DestinationRule¶
Define target addresses by creating subsets for the reviews Service during service discovery, specifically v1 and v2. To deploy this resource using kubectl, save the following content to the destination-rule-reviews.yaml file.
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
namespace: prod
spec:
host: reviews
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
Step Four: Create VirtualService¶
Before deploying v2, route all traffic to v1. Save the following content into the virtual-service-reviews.yaml file.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
namespace: prod
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
Visit http://8.136.193.105:32156/productpage.
Step Five: Deploy reviews-v2¶
Log in to 'GitLab', find the bookinfo-views project, change the value of APP_VERSION in the .gitlab-ci.yml file to "v2", and commit the code. Log in to 'Rancher', navigate through 'Workloads' -> 'Deployments' to see that reviews-v2 has been deployed. Even though v2 has been deployed, visiting http://8.136.193.105:32156/productpage still directs traffic only to the V1 version of reviews.
Step Six: Route 10% of Traffic to reviews-v2¶
Modify the virtual-service-reviews.yaml file as follows:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
namespace: prod
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 90
- destination:
host: reviews
subset: v2
weight: 10
Redeploy.
Multiple visits to http://8.136.193.105:32156/productpage will show that the v1 and v2 versions of reviews receive 90% and 10% of the traffic respectively,
Step Seven: Observe reviews-v2¶
1. Application Performance Monitoring (APM)¶
Log in to 'Guance' -> 'APM' -> click the topology chart in the top-right corner. Enable the environment and version distinction switch. Reviews has two versions, where reviews:test:v2 calls the ratings service.
Click on 'Traces' at the top. This time, use the resource search feature, select reviews.prod, find the trace for the v2 version of reviews, and click into it.
Observe the flame graph in the details interface. If there are any errors or timeouts in the trace calls, they will be clearly visible. The project, version, and env labels are defined in the annotations of the deployment.yaml file in the bookinfo-views project in GitLab.
View the execution duration of each Span in the Span list.
In the service call relationships, you can see a clear topology map.
2 Istio Mesh Monitoring View¶
Log in to 'Guance', click 'Scenarios' -> 'Create Dashboard', and select Istio Mesh Monitoring View. In this view, you can see that the ratio of calls to reviews-v1 and reviews-v2 is approximately 9:1.
Step Eight: Complete the Release¶
After verifying that the reviews-v2 microservice works normally, all traffic can be switched to v2. Modify the virtual-service-reviews.yaml file as follows:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
namespace: prod
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v2
Redeploy.
Note: If there are issues with the reviews-v2 version, log in to 'Guance' and refer to the last section of this chapter for trace timeout analysis to diagnose the problem. Refer to Step Six to completely revert traffic back to reviews-v1 until the issue is fixed and then redeploy.
Metrics¶
When deploying Bookinfo, custom configuration was used to start Pods, and measurement_name = "istio_prom" was added in the annotations configuration. This collects metrics into the istio_prom Measurement set. Log in to 'Guance' -> 'Metrics', and view the istio_prom Measurement set.
These metrics can be used to create similar Istio Mesh Monitoring Views based on your project's needs.
Traces¶
RUM¶
Log in to 'Guance', enter 'RUM', find the devops-bookinfo application, and click to enter.
View UV, PV, session counts, and visited pages.
Tip: For front-end and back-end separated projects, you can integrate backend traces and logs in the Explorer. Detailed steps can be found in Kubernetes Application RUM-APM-LOG Correlation Analysis.
APM¶
Log in to 'Guance', enter 'APM'. Through APM, view trace data.
Logs¶
stdout¶
According to the DataKit deployment configuration, logs output to /dev/stdout are collected by default. Log in to 'Guance', enter 'Logs', and view log information.
Tip: For more log collection methods, refer to:
Pod Log Collection Best Practices
Several Ways to Collect Logs in Kubernetes Clusters
Trace Timeout Analysis¶
The Bookinfo project includes a timeout demonstration. Using the jason user to log in causes the ratings service to time out. Create a new virtual-service-ratings-test-delay.yaml with the following content:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: ratings
namespace: prod
spec:
hosts:
- ratings
http:
- match:
- headers:
end-user:
exact: jason
fault:
delay:
percentage:
value: 100.0
fixedDelay: 7s
route:
- destination:
host: ratings
subset: v1
- route:
- destination:
host: ratings
subset: v1
Create the resource.
Log in using the jason user with an empty password.
Visit http://8.136.193.105:32156/productpage, and now the ratings service is unreachable.
Log in to 'Guance', enter 'APM'. Click on the timed-out trace.
Observe the flame graph to identify the timeout call.
