Istio的熔断

一、理解熔断
　　熔断（Circuit Breaker），原是指当电流超过规定值时断开电路，进行短路保护或严重过载时的一种保护机制。后来熔断也广泛应用于金融领域，指当股指波幅达到规定的熔断点时，交易所为控制风险采取的暂停交易措施。而在软件领域，熔断则是指当服务达到系统负载阈值时，为避免整个软件系统不可用，而采取的一种主动保护措施。对于微服务系统而言，熔断尤为重要，它可以使系统在遭遇某些模块故障时，通过服务降级等方式来提高系统核心功能的可用性，得以应对来自故障、潜在峰值或其他未知网络因素的影响。
在这里插入图片描述

二、配置熔断

1）部署httpbin服务

➜  kubectl apply -f ../../samples/httpbin/httpbin.yaml 
serviceaccount/httpbin unchanged
service/httpbin unchanged
deployment.apps/httpbin unchanged➜  kubectl describe pods httpbin-74fb669cc6-vn897 
Name:         httpbin-74fb669cc6-vn897
Namespace:    default
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Sat, 03 Jul 2021 17:23:17 +0800
Labels:       app=httpbinistio.io/rev=defaultpod-template-hash=74fb669cc6security.istio.io/tlsMode=istioservice.istio.io/canonical-name=httpbinservice.istio.io/canonical-revision=v1version=v1
Annotations:  kubectl.kubernetes.io/default-container: httpbinkubectl.kubernetes.io/default-logs-container: httpbinprometheus.io/path: /stats/prometheusprometheus.io/port: 15020prometheus.io/scrape: truesidecar.istio.io/status:{"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istiod-ca-cert"],"ima...
Status:       Running
IP:           10.1.8.151
IPs:IP:           10.1.8.151
Controlled By:  ReplicaSet/httpbin-74fb669cc6
Init Containers:istio-init:Container ID:  docker://538c2e169bfedaf636a942bee92183c59f79b47bea25a286f242e74c22abddadImage:         docker.io/istio/proxyv2:1.10.1Image ID:      docker-pullable://istio/proxyv2@sha256:d9b295da022ad826c54d5bb49f1f2b661826efd8c2672b2f61ddc2aedac78cfcPort:          <none>Host Port:     <none>Args:istio-iptables-p15001-z15006-u1337-mREDIRECT-i*-x-b*-d15090,15021,15020State:          TerminatedReason:       CompletedExit Code:    0Started:      Sat, 24 Jul 2021 17:03:16 +0800Finished:     Sat, 24 Jul 2021 17:03:17 +0800Ready:          TrueRestart Count:  1Limits:cpu:     2memory:  1GiRequests:cpu:        10mmemory:     40MiEnvironment:  <none>Mounts:/var/run/secrets/kubernetes.io/serviceaccount from httpbin-token-qk94s (ro)
Containers:httpbin:Container ID:   docker://3a3e8c32b05eb6f022ef6f660bad99128a7459452949bf3d43c0693dac5c82b1Image:          docker.io/kennethreitz/httpbinImage ID:       docker-pullable://kennethreitz/httpbin@sha256:599fe5e5073102dbb0ee3dbb65f049dab44fa9fc251f6835c9990f8fb196a72bPort:           80/TCPHost Port:      0/TCPState:          RunningStarted:      Sat, 24 Jul 2021 17:03:22 +0800Last State:     TerminatedReason:       ErrorExit Code:    255Started:      Sat, 03 Jul 2021 17:26:41 +0800Finished:     Sat, 24 Jul 2021 16:41:32 +0800Ready:          TrueRestart Count:  1Environment:    <none>Mounts:/var/run/secrets/kubernetes.io/serviceaccount from httpbin-token-qk94s (ro)istio-proxy:Container ID:  docker://fce4a591308062ef5f4c6aebb1e738e2fc79a4c8cfcc2498f6481a31d7ddb7aaImage:         docker.io/istio/proxyv2:1.10.1Image ID:      docker-pullable://istio/proxyv2@sha256:d9b295da022ad826c54d5bb49f1f2b661826efd8c2672b2f61ddc2aedac78cfcPort:          15090/TCPHost Port:     0/TCPArgs:proxysidecar--domain$(POD_NAMESPACE).svc.cluster.local--serviceClusterhttpbin.$(POD_NAMESPACE)--proxyLogLevel=warning--proxyComponentLogLevel=misc:error--log_output_level=default:info--concurrency2State:          RunningStarted:      Sat, 24 Jul 2021 17:03:24 +0800Last State:     TerminatedReason:       ErrorExit Code:    255Started:      Sat, 03 Jul 2021 17:26:41 +0800Finished:     Sat, 24 Jul 2021 16:41:32 +0800Ready:          TrueRestart Count:  1Limits:cpu:     2memory:  1GiRequests:cpu:      10mmemory:   40MiReadiness:  http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30Environment:JWT_POLICY:                    first-party-jwtPILOT_CERT_PROVIDER:           istiodCA_ADDR:                       istiod.istio-system.svc:15012POD_NAME:                      httpbin-74fb669cc6-vn897 (v1:metadata.name)POD_NAMESPACE:                 default (v1:metadata.namespace)INSTANCE_IP:                    (v1:status.podIP)SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)HOST_IP:                        (v1:status.hostIP)CANONICAL_SERVICE:              (v1:metadata.labels['service.istio.io/canonical-name'])CANONICAL_REVISION:             (v1:metadata.labels['service.istio.io/canonical-revision'])PROXY_CONFIG:                  {}ISTIO_META_POD_PORTS:          [{"containerPort":80,"protocol":"TCP"}]ISTIO_META_APP_CONTAINERS:     httpbinISTIO_META_CLUSTER_ID:         KubernetesISTIO_META_INTERCEPTION_MODE:  REDIRECTISTIO_META_WORKLOAD_NAME:      httpbinISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/default/deployments/httpbinISTIO_META_MESH_ID:            cluster.localTRUST_DOMAIN:                  cluster.localMounts:/etc/istio/pod from istio-podinfo (rw)/etc/istio/proxy from istio-envoy (rw)/var/lib/istio/data from istio-data (rw)/var/run/secrets/istio from istiod-ca-cert (rw)/var/run/secrets/kubernetes.io/serviceaccount from httpbin-token-qk94s (ro)
Conditions:Type              StatusInitialized       True Ready             True ContainersReady   True PodScheduled      True 
Volumes:istio-envoy:Type:       EmptyDir (a temporary directory that shares a pod's lifetime)Medium:     MemorySizeLimit:  <unset>istio-data:Type:       EmptyDir (a temporary directory that shares a pod's lifetime)Medium:     SizeLimit:  <unset>istio-podinfo:Type:  DownwardAPI (a volume populated by information about the pod)Items:metadata.labels -> labelsmetadata.annotations -> annotationslimits.cpu -> cpu-limitrequests.cpu -> cpu-requestistiod-ca-cert:Type:      ConfigMap (a volume populated by a ConfigMap)Name:      istio-ca-root-certOptional:  falsehttpbin-token-qk94s:Type:        Secret (a volume populated by a Secret)SecretName:  httpbin-token-qk94sOptional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

2）创建熔断器
　　创建一个目标规则，定义macConnections：1和httpMaxPendingRequests：1，当并发的连接数和请求数超过1个时，熔断功能将会生效。

➜  kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:name: httpbin
spec:host: httpbintrafficPolicy:connectionPool:tcp:maxConnections: 1http:http1MaxPendingRequests: 1maxRequestsPerConnection: 1
EOF
destinationrule.networking.istio.io/httpbin created➜  kubectl get dr -o wide  
NAME                   HOST                                                 AGE
dr-for-egressgateway   istio-egressgateway.istio-system.svc.cluster.local   21d
httpbin                httpbin                                              21skubectl describe dr httpbin                    
Name:         httpbin
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
Metadata:Creation Timestamp:  2021-07-24T10:38:06ZGeneration:          1Managed Fields:API Version:  networking.istio.io/v1alpha3Fields Type:  FieldsV1fieldsV1:f:metadata:f:annotations:.:f:kubectl.kubernetes.io/last-applied-configuration:f:spec:.:f:host:f:trafficPolicy:.:f:connectionPool:.:f:http:.:f:http1MaxPendingRequests:f:maxRequestsPerConnection:f:tcp:.:f:maxConnections:Manager:         kubectl-client-side-applyOperation:       UpdateTime:            2021-07-24T10:38:06ZResource Version:  654190Self Link:         /apis/networking.istio.io/v1beta1/namespaces/default/destinationrules/httpbinUID:               fd21f923-2d3b-4a59-a11c-747662302bc8
Spec:Host:  httpbinTraffic Policy:Connection Pool:Http:http1MaxPendingRequests:      1Max Requests Per Connection:  1Tcp:Max Connections:  1
Events:                   <none>

3）部署客户端程序
　　使用Fortio作为客户端进行测试，Fortio是一款优秀的负载测试工具，它起初时Istio项目的一部分，现在已经独立运营。

➜  kubectl apply -f ../../samples/httpbin/sample-client/fortio-deploy.yaml 
service/fortio created
deployment.apps/fortio-deploy created➜  kubectl describe deploy fortio-deploy 
Name:                   fortio-deploy
Namespace:              default
CreationTimestamp:      Sat, 24 Jul 2021 18:44:19 +0800
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=fortio
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:Labels:       app=fortioAnnotations:  sidecar.istio.io/statsInclusionPrefixes:cluster.outbound,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpcContainers:fortio:Image:        fortio/fortio:latest_releasePorts:        8080/TCP, 8079/TCPHost Ports:   0/TCP, 0/TCPEnvironment:  <none>Mounts:       <none>Volumes:        <none>
Conditions:Type           Status  Reason----           ------  ------Available      True    MinimumReplicasAvailableProgressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   fortio-deploy-576dbdfbc4 (1/1 replicas created)
Events:Type    Reason             Age   From                   Message----    ------             ----  ----                   -------Normal  ScalingReplicaSet  45s   deployment-controller  Scaled up replica set fortio-deploy-576dbdfbc4 to 1

4）正常访问httpbin服务

➜  FORTIO_POD=$(kubectl get pod | grep fortio | awk '{ print $1 }')
➜  echo $FORTIO_POD
➜  kubectl exec -it $FORTIO_POD  -c fortio -- /usr/bin/fortio load -curl  http://httpbin:8000/get
HTTP/1.1 200 OK
server: envoy
date: Sat, 24 Jul 2021 10:46:09 GMT
content-type: application/json
content-length: 594
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 107{"args": {}, "headers": {"Host": "httpbin:8000", "User-Agent": "fortio.org/fortio-1.11.3", "X-B3-Parentspanid": "fdf290f67dc89b84", "X-B3-Sampled": "1", "X-B3-Spanid": "7e7539083d56c52d", "X-B3-Traceid": "11f7193d9ab44f05fdf290f67dc89b84", "X-Envoy-Attempt-Count": "1", "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin;Hash=61abe9d5c14976b3980c6ab04bdf6be3ee928a4199571afb32ea01da9af81e29;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/default"}, "origin": "127.0.0.6", "url": "http://httpbin:8000/get"
}

5）测试熔断功能
　　发送并发数为30的连接数（-c 30），请求300次（-n 300），测试结果如下，大概只有3%的成功率。在DestinationRule配置中，定义了maxConnections为1和http1MaxPendingRequests为1。这些规则意味着，如果并发的连接数和请求数超过了1个时，在istio-proxy进行进一步的请求和连接将被阻止。于是，当并发数为30时，成功率只有1/30，也就是3.3%左右；如果你看到的成功率并非3.3%，如果4.3%，也是正常的，istio-proxy确实允许存在一定的误差。

➜  kubectl exec -it $FORTIO_POD  -c fortio -- /usr/bin/fortio load -c 30 -qps 0 -n 300 -loglevel Warning http://httpbin:8000/get
10:48:43 I logger.go:127> Log level is now 3 Warning (was 2 Info)
Fortio 1.11.3 running at 0 queries per second, 4->4 procs, for 300 calls: http://httpbin:8000/get
Starting at max qps with 30 thread(s) [gomax 4] for exactly 300 calls (10 per thread + 0)
10:48:43 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
10:48:43 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
10:48:43 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
10:48:43 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
.................................
Ended after 947.7294ms : 300 calls. qps=316.55
Aggregated Function Time : count 300 avg 0.078995054 +/- 0.05082 min 0.0060063 max 0.2884903 sum 23.6985163
# range, mid point, percentile, count
>= 0.0060063 <= 0.007 , 0.00650315 , 1.33, 4
> 0.007 <= 0.008 , 0.0075 , 1.67, 1
> 0.009 <= 0.01 , 0.0095 , 2.33, 2
> 0.01 <= 0.011 , 0.0105 , 2.67, 1
> 0.011 <= 0.012 , 0.0115 , 3.00, 1
> 0.012 <= 0.014 , 0.013 , 4.67, 5
> 0.014 <= 0.016 , 0.015 , 5.67, 3
> 0.016 <= 0.018 , 0.017 , 7.00, 4
> 0.018 <= 0.02 , 0.019 , 9.00, 6
> 0.02 <= 0.025 , 0.0225 , 13.67, 14
> 0.025 <= 0.03 , 0.0275 , 16.33, 8
> 0.03 <= 0.035 , 0.0325 , 20.33, 12
> 0.035 <= 0.04 , 0.0375 , 23.67, 10
> 0.04 <= 0.045 , 0.0425 , 27.00, 10
> 0.045 <= 0.05 , 0.0475 , 31.33, 13
> 0.05 <= 0.06 , 0.055 , 41.33, 30
> 0.06 <= 0.07 , 0.065 , 53.67, 37
> 0.07 <= 0.08 , 0.075 , 60.00, 19
> 0.08 <= 0.09 , 0.085 , 66.67, 20
> 0.09 <= 0.1 , 0.095 , 73.33, 20
> 0.1 <= 0.12 , 0.11 , 79.67, 19
> 0.12 <= 0.14 , 0.13 , 84.33, 14
> 0.14 <= 0.16 , 0.15 , 91.67, 22
> 0.16 <= 0.18 , 0.17 , 95.00, 10
> 0.18 <= 0.2 , 0.19 , 97.67, 8
> 0.2 <= 0.25 , 0.225 , 99.67, 6
> 0.25 <= 0.28849 , 0.269245 , 100.00, 1
# target 50% 0.067027
# target 75% 0.105263
# target 90% 0.155455
# target 99% 0.233333
# target 99.9% 0.276943
Sockets used: 285 (for perfect keepalive, would be 30)
Jitter: false
Code 200 : 18 (6.0 %)
Code 503 : 282 (94.0 %)
Response Header Sizes : count 300 avg 13.88 +/- 54.94 min 0 max 232 sum 4164
Response Body/Total Sizes : count 300 avg 276.06 +/- 138.8 min 241 max 826 sum 82818
All done 300 calls (plus 0 warmup) 78.995 ms avg, 316.5 qps

6）清理

➜  kubectl delete destinationrule httpbin
destinationrule.networking.istio.io "httpbin" deleted➜  kubectl delete deploy httpbin fortio-deploy
deployment.apps "httpbin" deleted
deployment.apps "fortio-deploy" deleted