基于Sentinel自研组件的系统限流、降级、负载保护最佳实践探索 | 京东云技术团队

作者:京东物流 杨建民

一、Sentinel简介

Sentinel 以流量为切入点,从流量控制熔断降级系统负载保护等多个维度保护服务的稳定性。

Sentinel 具有以下特征:

  • 丰富的应用场景:秒杀(即突发流量控制在系统容量可以承受的范围)、消息削峰填谷、集群流量控制、实时熔断下游不可用应用等。
  • 完备的实时监控:Sentinel 同时提供实时的监控功能。您可以在控制台中看到接入应用的单台机器秒级数据,甚至 500 台以下规模的集群的汇总运行情况。
  • 广泛的开源生态:Sentinel 提供开箱即用的与其它开源框架/库的整合模块,例如与 Spring Cloud、Apache Dubbo、gRPC、Quarkus 的整合。您只需要引入相应的依赖并进行简单的配置即可快速地接入 Sentinel。同时 Sentinel 提供 Java/Go/C++ 等多语言的原生实现。
  • 完善的 SPI 扩展机制:Sentinel 提供简单易用、完善的 SPI 扩展接口。您可以通过实现扩展接口来快速地定制逻辑。例如定制规则管理、适配动态数据源等

有关Sentinel的详细介绍以及和Hystrix的区别可以自行网上检索,推荐一篇文章:https://mp.weixin.qq.com/s/Q7Xv8cypQFrrOQhbd9BOXw

本次主要使用了Sentinel的降级、限流、系统负载保护功能

二、Sentinel关键技术源码解析

无论是限流、降级、负载等控制手段,大致流程如下:

•StatisticSlot 则用于记录、统计不同维度的 runtime 指标监控信息

•责任链依次触发后续 slot 的 entry 方法,如 SystemSlot、FlowSlot、DegradeSlot 等的规则校验;

•当后续的 slot 通过,没有抛出 BlockException 异常,说明该资源被成功调用,则增加执行线程数和通过的请求数等信息。

关于数据统计,主要会牵扯到 ArrayMetric、BucketLeapArray、MetricBucket、WindowWrap 等类。

项目结构

以下主要分析core包里的内容

2.1注解入口

2.1.1 Entry、Context、Node

SphU门面类的方法出参都是Entry,Entry可以理解为每次进入资源的一个凭证,如果调用SphO.entry()或者SphU.entry()能获取Entry对象,代表获取了凭证,没有被限流,否则抛出一个BlockException。

Entry中持有本次对资源调用的相关信息:

•createTime:创建该Entry的时间戳。

•curNode:Entry当前是在哪个节点。

•orginNode:Entry的调用源节点。

•resourceWrapper:Entry关联的资源信息。

Entry是一个抽象类,CtEntry是Entry的实现,CtEntry持有Context和调用链的信息

Context的源码注释如下,

1This class holds metadata of current invocation 2 3

Node的源码注释

1Holds real-time statistics for resources 2 3

Node中保存了对资源的实时数据的统计,Sentinel中的限流或者降级等功能就是通过Node中的数据进行判断的。Node是一个接口,里面定义了各种操作request、exception、rt、qps、thread的方法。

在细看Node实现时,不难发现LongAddr的使用,关于LongAddr和DoubleAddr都是java8 java.util.concurrent.atomic里的内容,感兴趣的小伙伴可以再深入研究一下,这两个是高并发下计数功能非常优秀的数据结构,实际应用场景里需要计数时可以考虑使用。

关于Node的介绍后续还会深入,此处大致先提一下这个概念。

2.2 初始化

2.2.1 Context初始化

在初始化slot责任链部分前,还执行了context的初始化,里面涉及几个重要概念,需要解释一下:

可以发现在Context初始化的过程中,会把EntranceNode加入到Root子节点中(实际Root本身是一个特殊的EntranceNode),并把EntranceNode放到contextNameNodeMap中。

之前简单提到过Node,是用来统计数据用的,不同Node功能如下:

•Node:用于完成数据统计的接口

•StatisticNode:统计节点,是Node接口的实现类,用于完成数据统计

•EntranceNode:入口节点,一个Context会有一个入口节点,用于统计当前Context的总体流量数据

•DefaultNode:默认节点,用于统计一个资源在当前Context中的流量数据

•ClusterNode:集群节点,用于统计一个资源在所有Context中的总体流量数据

1protected static Context trueEnter(String name, String origin) { 2 Context context = contextHolder.get(); 3 if (context == null) { 4 Map<String, DefaultNode> localCacheNameMap = contextNameNodeMap; 5 DefaultNode node = localCacheNameMap.get(name); 6 if (node == null) { 7 if (localCacheNameMap.size() > Constants.MAX_CONTEXT_NAME_SIZE) { 8 setNullContext(); 9 return NULL_CONTEXT; 10 } else { 11 LOCK.lock(); 12 try { 13 node = contextNameNodeMap.get(name); 14 if (node == null) { 15 if (contextNameNodeMap.size() > Constants.MAX_CONTEXT_NAME_SIZE) { 16 setNullContext(); 17 return NULL_CONTEXT; 18 } else { 19 node = new EntranceNode(new StringResourceWrapper(name, EntryType.IN), null); 20 // Add entrance node. 21 Constants.ROOT.addChild(node); 22 23 Map<String, DefaultNode> newMap = new HashMap<>(contextNameNodeMap.size() + 1); 24 newMap.putAll(contextNameNodeMap); 25 newMap.put(name, node); 26 contextNameNodeMap = newMap; 27 } 28 } 29 } finally { 30 LOCK.unlock(); 31 } 32 } 33 } 34 context = new Context(node, name); 35 context.setOrigin(origin); 36 contextHolder.set(context); 37 } 38 39 return context; 40 } 41 42

2.2.2 通过SpiLoader默认初始化8个slot

每个slot的主要职责如下:

•NodeSelectorSlot 负责收集资源的路径,并将这些资源的调用路径,以树状结构存储起来,用于根据调用路径来限流降级

•ClusterBuilderSlot 则用于存储资源的统计信息以及调用者信息,例如该资源的 RT, QPS, thread count 等等,这些信息将用作为多维度限流,降级的依据

•StatisticSlot 则用于记录、统计不同纬度的 runtime 指标监控信息

•FlowSlot 则用于根据预设的限流规则以及前面 slot 统计的状态,来进行流量控制

•AuthoritySlot 则根据配置的黑白名单和调用来源信息,来做黑白名单控制

•DegradeSlot 则通过统计信息以及预设的规则,来做熔断降级

•SystemSlot 则通过系统的状态,例如 集群QPS、线程数、RT、负载 等,来控制总的入口流量

2.3 StatisticSlot

2.3.1 Node

深入看一下Node,因为统计信息都在里面,后面不论是限流、熔断、负载保护等都是结合规则+统计信息判断是否要执行

从Node的源码注释看,它会持有资源维度的实时统计数据,以下是接口里的方法定义,可以看到totalRequest、totalPass、totalSuccess、blockRequest、totalException、passQps等很多request、qps、thread的相关方法:

1/** 2 * Holds real-time statistics for resources. 3 * 4 * @author qinan.qn 5 * @author leyou 6 * @author Eric Zhao 7 */ 8public interface Node extends OccupySupport, DebugSupport { 9 long totalRequest(); 10 long totalPass(); 11 long totalSuccess(); 12 long blockRequest(); 13 long totalException(); 14 double passQps(); 15 double blockQps(); 16 double totalQps(); 17 double successQps(); 18 …… 19} 20 21

2.3.2 StatisticNode

我们先从最基础的StatisticNode开始看,源码给出的定位是:

1The statistic node keep three kinds of real-time statistics metrics: 2metrics in second level ({@code rollingCounterInSecond}) 3metrics in minute level ({@code rollingCounterInMinute}) 4thread count 5 6

StatisticNode只有四个属性,除了之前提到过的LongAddr类型的curThreadNum外,还有两个属性是Metric对象,通过入参已经属性命名可以看出,一个用于秒级,一个用于分钟级统计。接下来我们就要看看Metric

1// StatisticNode持有两个Metric,一个秒级一个分钟级,由入参可知,秒级统计划分了两个时间窗口,窗口程度是500ms 2private transient volatile Metric rollingCounterInSecond = new ArrayMetric(SampleCountProperty.SAMPLE_COUNT, 3 IntervalProperty.INTERVAL); 4 5// 分钟级统计划分了60个时间窗口,窗口长度是1000ms 6private transient Metric rollingCounterInMinute = new ArrayMetric(60, 60 * 1000, false); 7 8/** 9 * The counter for thread count. 10 */ 11private LongAdder curThreadNum = new LongAdder(); 12 13/** 14 * The last timestamp when metrics were fetched. 15 */ 16private long lastFetchTime = -1; 17 18

ArrayMetric只有一个属性LeapArray<MetricBucket>,其余都是用于统计的方法,LeapArray是sentinel中统计最基本的数据结构,这里有必要详细看一下,总体就是根据timeMillis去获取一个bucket,分为:没有创建、有直接返回、被废弃后的reset三种场景。

1//以分钟级的统计属性为例,看一下时间窗口初始化过程 2private transient Metric rollingCounterInMinute = new ArrayMetric(60, 60 * 1000, false); 3 4 5public LeapArray(int sampleCount, int intervalInMs) { 6 AssertUtil.isTrue(sampleCount > 0, "bucket count is invalid: " + sampleCount); 7 AssertUtil.isTrue(intervalInMs > 0, "total time interval of the sliding window should be positive"); 8 AssertUtil.isTrue(intervalInMs % sampleCount == 0, "time span needs to be evenly divided"); 9 // windowLengthInMs = 60*1000 / 60 = 1000 滑动窗口时间长度,可见sentinel默认将单位时间分为了60个滑动窗口进行数据统计 10 this.windowLengthInMs = intervalInMs / sampleCount; 11 // 60*1000 12 this.intervalInMs = intervalInMs; 13 // 60 14 this.intervalInSecond = intervalInMs / 1000.0; 15 // 60 16 this.sampleCount = sampleCount; 17 // 数组长度60 18 this.array = new AtomicReferenceArray<>(sampleCount); 19 } 20 21/** 22 * Get bucket item at provided timestamp. 23 * 24 * @param timeMillis a valid timestamp in milliseconds 25 * @return current bucket item at provided timestamp if the time is valid; null if time is invalid 26 */ 27 public WindowWrap<T> currentWindow(long timeMillis) { 28 if (timeMillis < 0) { 29 return null; 30 } 31 // 根据当前时间戳算一个数组索引 32 int idx = calculateTimeIdx(timeMillis); 33 // Calculate current bucket start time. 34 // timeMillis % 1000 35 long windowStart = calculateWindowStart(timeMillis); 36 37 /* 38 * Get bucket item at given time from the array. 39 * 40 * (1) Bucket is absent, then just create a new bucket and CAS update to circular array. 41 * (2) Bucket is up-to-date, then just return the bucket. 42 * (3) Bucket is deprecated, then reset current bucket. 43 */ 44 while (true) { 45 WindowWrap<T> old = array.get(idx); 46 if (old == null) { 47 /* 48 * B0 B1 B2 NULL B4 49 * ||_______|_______|_______|_______|_______||___ 50 * 200 400 600 800 1000 1200 timestamp 51 * ^ 52 * time=888 53 * bucket is empty, so create new and update 54 * 55 * If the old bucket is absent, then we create a new bucket at {@code windowStart}, 56 * then try to update circular array via a CAS operation. Only one thread can 57 * succeed to update, while other threads yield its time slice. 58 */ 59 // newEmptyBucket 方法重写,秒级和分钟级统计对象实现不同 60 WindowWrap<T> window = new WindowWrap<T>(windowLengthInMs, windowStart, newEmptyBucket(timeMillis)); 61 if (array.compareAndSet(idx, null, window)) { 62 // Successfully updated, return the created bucket. 63 return window; 64 } else { 65 // Contention failed, the thread will yield its time slice to wait for bucket available. 66 Thread.yield(); 67 } 68 } else if (windowStart == old.windowStart()) { 69 /* 70 * B0 B1 B2 B3 B4 71 * ||_______|_______|_______|_______|_______||___ 72 * 200 400 600 800 1000 1200 timestamp 73 * ^ 74 * time=888 75 * startTime of Bucket 3: 800, so it's up-to-date 76 * 77 * If current {@code windowStart} is equal to the start timestamp of old bucket, 78 * that means the time is within the bucket, so directly return the bucket. 79 */ 80 return old; 81 } else if (windowStart > old.windowStart()) { 82 /* 83 * (old) 84 * B0 B1 B2 NULL B4 85 * |_______||_______|_______|_______|_______|_______||___ 86 * ... 1200 1400 1600 1800 2000 2200 timestamp 87 * ^ 88 * time=1676 89 * startTime of Bucket 2: 400, deprecated, should be reset 90 * 91 * If the start timestamp of old bucket is behind provided time, that means 92 * the bucket is deprecated. We have to reset the bucket to current {@code windowStart}. 93 * Note that the reset and clean-up operations are hard to be atomic, 94 * so we need a update lock to guarantee the correctness of bucket update. 95 * 96 * The update lock is conditional (tiny scope) and will take effect only when 97 * bucket is deprecated, so in most cases it won't lead to performance loss. 98 */ 99 if (updateLock.tryLock()) { 100 try { 101 // Successfully get the update lock, now we reset the bucket. 102 return resetWindowTo(old, windowStart); 103 } finally { 104 updateLock.unlock(); 105 } 106 } else { 107 // Contention failed, the thread will yield its time slice to wait for bucket available. 108 Thread.yield(); 109 } 110 } else if (windowStart < old.windowStart()) { 111 // Should not go through here, as the provided time is already behind. 112 return new WindowWrap<T>(windowLengthInMs, windowStart, newEmptyBucket(timeMillis)); 113 } 114 } 115 } 116// 持有一个时间窗口对象的数据,会根据当前时间戳除以时间窗口长度然后散列到数组中 117private int calculateTimeIdx(/*@Valid*/ long timeMillis) { 118 long timeId = timeMillis / windowLengthInMs; 119 // Calculate current index so we can map the timestamp to the leap array. 120 return (int)(timeId % array.length()); 121 } 122 123

WindowWrap持有了windowLengthInMs, windowStart和LeapArray(分钟统计实现是BucketLeapArray,秒级统计实现是OccupiableBucketLeapArray),对于分钟级别的统计,MetricBucket维护了一个longAddr数组和一个配置的minRT

1/** 2 * The fundamental data structure for metric statistics in a time span. 3 * 4 * @author jialiang.linjl 5 * @author Eric Zhao 6 * @see LeapArray 7 */ 8public class BucketLeapArray extends LeapArray<MetricBucket> { 9 10 public BucketLeapArray(int sampleCount, int intervalInMs) { 11 super(sampleCount, intervalInMs); 12 } 13 14 @Override 15 public MetricBucket newEmptyBucket(long time) { 16 return new MetricBucket(); 17 } 18 19 @Override 20 protected WindowWrap<MetricBucket> resetWindowTo(WindowWrap<MetricBucket> w, long startTime) { 21 // Update the start time and reset value. 22 w.resetTo(startTime); 23 w.value().reset(); 24 return w; 25 } 26} 27 28

对于秒级统计,QPS=20场景下,如何准确统计的问题,此处用到了另外一个LeapArry实现FutureBucketLeapArray,至于秒级统计如何保证没有统计误差,读者可以再研究一下FutureBucketLeapArray的上下文就好。

2.4 FlowSlot

2.4.1 常见限流算法

介绍sentinel限流实现前,先介绍一下常见限流算法,基本分为三种:计数器、漏斗、令牌桶。

计数器算法

顾名思义,计数器算法就是统计某个时间段内的请求,每单位时间加1,然后与配置的限流值(最大QPS)进行比较,如果超出则触发限流。但是这种算法不能做到“平滑限流”,以1s为单位时间,100QPS为限流值为例,如下图,会出现某时段超出限流值的情况

因此在单纯计数器算法上,又出现了滑动窗口计数器算法,我们将统计时间细分,比如将1s统计时长分为5个时间窗口,通过滚动统计所有时间窗口的QPS作为系统实际的QPS的方式,就能解决上述临界统计问题,后续我们看sentinel源码时也能看到类似操作。

漏斗算法

不论流量有多大都会先到漏桶中,然后以均匀的速度流出。如何在代码中实现这个匀速呢?比如我们想让匀速为100q/s,那么我们可以得到每流出一个流量需要消耗10ms,类似一个队列,每隔10ms从队列头部取出流量进行放行,而我们的队列也就是漏桶,当流量大于队列的长度的时候,我们就可以拒绝超出的部分。

漏斗算法同样的也有一定的缺点:无法应对突发流量。比如一瞬间来了100个请求,在漏桶算法中只能一个一个的过去,当最后一个请求流出的时候时间已经过了一秒了,所以漏斗算法比较适合请求到达比较均匀,需要严格控制请求速率的场景。

令牌桶算法

令牌桶算法和漏斗算法比较类似,区别是令牌桶存放的是令牌数量不是请求数量,令牌桶可以根据自身需求多样性得管理令牌的生产和消耗,可以解决突发流量的问题。

2.4.2 单机限流模式

接下来我们看一下Sentinel中的限流实现,相比上述基本限流算法,Sentinel限流的第一个特性就是引入“资源”的概念,可以细粒度多样性的支持特定资源、关联资源、指定链路的限流。

FlowSlot的主要逻辑都在FlowRuleChecker里,介绍之前,我们先看一下Sentinel关于规则的模型描述,下图分别是限流、访问控制规则、系统保护规则(Linux负载)、降级规则

1 /** 2 * 流量控制两种模式 3 * 0: thread count(当调用该api的线程数达到阈值的时候,进行限流) 4 * 1: QPS(当调用该api的QPS达到阈值的时候,进行限流) 5 */ 6 private int grade = RuleConstant.FLOW_GRADE_QPS; 7 8 /** 9 * 流量控制阈值,值含义与grade有关 10 */ 11 private double count; 12 13 /** 14 * 调用关系限流策略(可以支持关联资源或指定链路的多样性限流需求) 15 * 直接(api 达到限流条件时,直接限流) 16 * 关联(当关联的资源达到限流阈值时,就限流自己) 17 * 链路(只记录指定链路上的流量) 18 * {@link RuleConstant#STRATEGY_DIRECT} for direct flow control (by origin); 19 * {@link RuleConstant#STRATEGY_RELATE} for relevant flow control (with relevant resource); 20 * {@link RuleConstant#STRATEGY_CHAIN} for chain flow control (by entrance resource). 21 */ 22 private int strategy = RuleConstant.STRATEGY_DIRECT; 23 24 /** 25 * Reference resource in flow control with relevant resource or context. 26 */ 27 private String refResource; 28 29 /** 30 * 流控效果: 31 * 0. default(reject directly),直接拒绝,抛异常FlowException 32 * 1. warm up, 慢启动模式(根据coldFactor(冷加载因子,默认3)的值,从阈值/coldFactor,经过预热时长,才达到设置的QPS阈值) 33 * 2. rate limiter 排队等待 34 * 3. warm up + rate limiter 35 */ 36 private int controlBehavior = RuleConstant.CONTROL_BEHAVIOR_DEFAULT; 37 38 private int warmUpPeriodSec = 10; 39 40 /** 41 * Max queueing time in rate limiter behavior. 42 */ 43 private int maxQueueingTimeMs = 500; 44 45 /** 46 * 是否集群限流,默认为否 47 */ 48 private boolean clusterMode; 49 /** 50 * Flow rule config for cluster mode. 51 */ 52 private ClusterFlowConfig clusterConfig; 53 54 /** 55 * The traffic shaping (throttling) controller. 56 */ 57 private TrafficShapingController controller; 58 59

接着我们继续分析FlowRuleChecker

canPassCheck第一步会好看limitApp,这个是结合访问授权限制规则使用的,默认是所有。

1private static boolean passLocalCheck(FlowRule rule, Context context, DefaultNode node, int acquireCount, 2 boolean prioritized) { 3 // 根据策略选择Node来进行统计(可以是本身Node、关联的Node、指定的链路) 4 Node selectedNode = selectNodeByRequesterAndStrategy(rule, context, node); 5 if (selectedNode == null) { 6 return true; 7 } 8 9 return rule.getRater().canPass(selectedNode, acquireCount, prioritized); 10 } 11 12 13static Node selectNodeByRequesterAndStrategy(/*@NonNull*/ FlowRule rule, Context context, DefaultNode node) { 14 // limitApp是访问控制使用的,默认是default,不限制来源 15 String limitApp = rule.getLimitApp(); 16 // 拿到限流策略 17 int strategy = rule.getStrategy(); 18 String origin = context.getOrigin(); 19 // 基于调用来源做鉴权 20 if (limitApp.equals(origin) && filterOrigin(origin)) { 21 if (strategy == RuleConstant.STRATEGY_DIRECT) { 22 // Matches limit origin, return origin statistic node. 23 return context.getOriginNode(); 24 } 25 // 26 return selectReferenceNode(rule, context, node); 27 } else if (RuleConstant.LIMIT_APP_DEFAULT.equals(limitApp)) { 28 if (strategy == RuleConstant.STRATEGY_DIRECT) { 29 // Return the cluster node. 30 return node.getClusterNode(); 31 } 32 33 return selectReferenceNode(rule, context, node); 34 } else if (RuleConstant.LIMIT_APP_OTHER.equals(limitApp) 35 && FlowRuleManager.isOtherOrigin(origin, rule.getResource())) { 36 if (strategy == RuleConstant.STRATEGY_DIRECT) { 37 return context.getOriginNode(); 38 } 39 40 return selectReferenceNode(rule, context, node); 41 } 42 43 return null; 44 } 45 46static Node selectReferenceNode(FlowRule rule, Context context, DefaultNode node) { 47 String refResource = rule.getRefResource(); 48 int strategy = rule.getStrategy(); 49 50 if (StringUtil.isEmpty(refResource)) { 51 return null; 52 } 53 54 if (strategy == RuleConstant.STRATEGY_RELATE) { 55 return ClusterBuilderSlot.getClusterNode(refResource); 56 } 57 58 if (strategy == RuleConstant.STRATEGY_CHAIN) { 59 if (!refResource.equals(context.getName())) { 60 return null; 61 } 62 return node; 63 } 64 // No node. 65 return null; 66 } 67 68// 此代码是load限流规则时根据规则初始化流量整形控制器的逻辑,rule.getRater()返回TrafficShapingController 69private static TrafficShapingController generateRater(/*@Valid*/ FlowRule rule) { 70 if (rule.getGrade() == RuleConstant.FLOW_GRADE_QPS) { 71 switch (rule.getControlBehavior()) { 72 // 预热模式返回WarmUpController 73 case RuleConstant.CONTROL_BEHAVIOR_WARM_UP: 74 return new WarmUpController(rule.getCount(), rule.getWarmUpPeriodSec(), 75 ColdFactorProperty.coldFactor); 76 // 排队模式返回ThrottlingController 77 case RuleConstant.CONTROL_BEHAVIOR_RATE_LIMITER: 78 return new ThrottlingController(rule.getMaxQueueingTimeMs(), rule.getCount()); 79 // 预热+排队模式返回WarmUpRateLimiterController 80 case RuleConstant.CONTROL_BEHAVIOR_WARM_UP_RATE_LIMITER: 81 return new WarmUpRateLimiterController(rule.getCount(), rule.getWarmUpPeriodSec(), 82 rule.getMaxQueueingTimeMs(), ColdFactorProperty.coldFactor); 83 case RuleConstant.CONTROL_BEHAVIOR_DEFAULT: 84 default: 85 // Default mode or unknown mode: default traffic shaping controller (fast-reject). 86 } 87 } 88 // 默认是DefaultController 89 return new DefaultController(rule.getCount(), rule.getGrade()); 90 } 91 92

Sentinel单机限流算法

上面我们看到根据限流规则controlBehavior属性(流控效果),会初始化以下实现:

•DefaultController:是一个非常典型的滑动窗口计数器算法实现,将当前统计的qps和请求进来的qps进行求和,小于限流值则通过,大于则计算一个等待时间,稍后再试

•ThrottlingController:是漏斗算法的实现,实现思路已经在源码片段中加了备注

•WarmUpController:实现参考了Guava的带预热的RateLimiter,区别是Guava侧重于请求间隔,类似前面提到的令牌桶,而Sentinel更关注于请求数,和令牌桶算法有点类似

•WarmUpRateLimiterController:低水位使用预热算法,高水位使用滑动窗口计数器算法排队。

DefaultController

1 @Override 2 public boolean canPass(Node node, int acquireCount, boolean prioritized) { 3 int curCount = avgUsedTokens(node); 4 if (curCount + acquireCount > count) { 5 if (prioritized && grade == RuleConstant.FLOW_GRADE_QPS) { 6 long currentTime; 7 long waitInMs; 8 currentTime = TimeUtil.currentTimeMillis(); 9 waitInMs = node.tryOccupyNext(currentTime, acquireCount, count); 10 if (waitInMs < OccupyTimeoutProperty.getOccupyTimeout()) { 11 node.addWaitingRequest(currentTime + waitInMs, acquireCount); 12 node.addOccupiedPass(acquireCount); 13 sleep(waitInMs); 14 15 // PriorityWaitException indicates that the request will pass after waiting for {@link @waitInMs}. 16 throw new PriorityWaitException(waitInMs); 17 } 18 } 19 return false; 20 } 21 return true; 22 } 23 24

ThrottlingController

1 public ThrottlingController(int queueingTimeoutMs, double maxCountPerStat) { 2 this(queueingTimeoutMs, maxCountPerStat, 1000); 3 } 4 5 public ThrottlingController(int queueingTimeoutMs, double maxCountPerStat, int statDurationMs) { 6 AssertUtil.assertTrue(statDurationMs > 0, "statDurationMs should be positive"); 7 AssertUtil.assertTrue(maxCountPerStat >= 0, "maxCountPerStat should be >= 0"); 8 AssertUtil.assertTrue(queueingTimeoutMs >= 0, "queueingTimeoutMs should be >= 0"); 9 this.maxQueueingTimeMs = queueingTimeoutMs; 10 this.count = maxCountPerStat; 11 this.statDurationMs = statDurationMs; 12 // Use nanoSeconds when durationMs%count != 0 or count/durationMs> 1 (to be accurate) 13 // 可见配置限流值count大于1000时useNanoSeconds会是true否则是false 14 if (maxCountPerStat > 0) { 15 this.useNanoSeconds = statDurationMs % Math.round(maxCountPerStat) != 0 || maxCountPerStat / statDurationMs > 1; 16 } else { 17 this.useNanoSeconds = false; 18 } 19 } 20 21 @Override 22 public boolean canPass(Node node, int acquireCount) { 23 return canPass(node, acquireCount, false); 24 } 25 26 private boolean checkPassUsingNanoSeconds(int acquireCount, double maxCountPerStat) { 27 final long maxQueueingTimeNs = maxQueueingTimeMs * MS_TO_NS_OFFSET; 28 long currentTime = System.nanoTime(); 29 // Calculate the interval between every two requests. 30 final long costTimeNs = Math.round(1.0d * MS_TO_NS_OFFSET * statDurationMs * acquireCount / maxCountPerStat); 31 32 // Expected pass time of this request. 33 long expectedTime = costTimeNs + latestPassedTime.get(); 34 35 if (expectedTime <= currentTime) { 36 // Contention may exist here, but it's okay. 37 latestPassedTime.set(currentTime); 38 return true; 39 } else { 40 final long curNanos = System.nanoTime(); 41 // Calculate the time to wait. 42 long waitTime = costTimeNs + latestPassedTime.get() - curNanos; 43 if (waitTime > maxQueueingTimeNs) { 44 return false; 45 } 46 47 long oldTime = latestPassedTime.addAndGet(costTimeNs); 48 waitTime = oldTime - curNanos; 49 if (waitTime > maxQueueingTimeNs) { 50 latestPassedTime.addAndGet(-costTimeNs); 51 return false; 52 } 53 // in race condition waitTime may <= 0 54 if (waitTime > 0) { 55 sleepNanos(waitTime); 56 } 57 return true; 58 } 59 } 60 61 // 漏斗算法具体实现 62 private boolean checkPassUsingCachedMs(int acquireCount, double maxCountPerStat) { 63 long currentTime = TimeUtil.currentTimeMillis(); 64 // 计算两次请求的间隔(分为秒级和纳秒级) 65 long costTime = Math.round(1.0d * statDurationMs * acquireCount / maxCountPerStat); 66 67 // 请求的期望的时间 68 long expectedTime = costTime + latestPassedTime.get(); 69 70 if (expectedTime <= currentTime) { 71 // latestPassedTime是AtomicLong类型,支持volatile语义 72 latestPassedTime.set(currentTime); 73 return true; 74 } else { 75 // 计算等待时间 76 long waitTime = costTime + latestPassedTime.get() - TimeUtil.currentTimeMillis(); 77 // 如果大于最大排队时间,则触发限流 78 if (waitTime > maxQueueingTimeMs) { 79 return false; 80 } 81 82 long oldTime = latestPassedTime.addAndGet(costTime); 83 waitTime = oldTime - TimeUtil.currentTimeMillis(); 84 if (waitTime > maxQueueingTimeMs) { 85 latestPassedTime.addAndGet(-costTime); 86 return false; 87 } 88 // in race condition waitTime may <= 0 89 if (waitTime > 0) { 90 sleepMs(waitTime); 91 } 92 return true; 93 } 94 } 95 96 @Override 97 public boolean canPass(Node node, int acquireCount, boolean prioritized) { 98 // Pass when acquire count is less or equal than 0. 99 if (acquireCount <= 0) { 100 return true; 101 } 102 // Reject when count is less or equal than 0. 103 // Otherwise, the costTime will be max of long and waitTime will overflow in some cases. 104 if (count <= 0) { 105 return false; 106 } 107 if (useNanoSeconds) { 108 return checkPassUsingNanoSeconds(acquireCount, this.count); 109 } else { 110 return checkPassUsingCachedMs(acquireCount, this.count); 111 } 112 } 113 114 private void sleepMs(long ms) { 115 try { 116 Thread.sleep(ms); 117 } catch (InterruptedException e) { 118 } 119 } 120 121 private void sleepNanos(long ns) { 122 LockSupport.parkNanos(ns); 123 } 124 125
1long costTime = Math.round(1.0d * statDurationMs * acquireCount / maxCountPerStat); 2 3

由上述计算两次请求间隔的公式我们可以发现,当maxCountPerStat(规则配置的限流值QPS)超过1000后,就无法准确计算出匀速排队模式下的请求间隔时长,因此对应前面介绍的,当规则配置限流值超过1000QPS后,会采用checkPassUsingNanoSeconds,小于1000QPS会采用checkPassUsingCachedMs,对比一下checkPassUsingNanoSeconds和checkPassUsingCachedMs,可以发现主体思路没变,只是统计维度从毫秒换算成了纳秒,因此只看checkPassUsingCachedMs实现就可以

WarmUpController

1 2@Override 3 public boolean canPass(Node node, int acquireCount, boolean prioritized) { 4 long passQps = (long) node.passQps(); 5 6 long previousQps = (long) node.previousPassQps(); 7 syncToken(previousQps); 8 9 // 开始计算它的斜率 10 // 如果进入了警戒线,开始调整他的qps 11 long restToken = storedTokens.get(); 12 if (restToken >= warningToken) { 13 long aboveToken = restToken - warningToken; 14 // 消耗的速度要比warning快,但是要比慢 15 // current interval = restToken*slope+1/count 16 double warningQps = Math.nextUp(1.0 / (aboveToken * slope + 1.0 / count)); 17 if (passQps + acquireCount <= warningQps) { 18 return true; 19 } 20 } else { 21 if (passQps + acquireCount <= count) { 22 return true; 23 } 24 } 25 26 return false; 27 } 28 29protected void syncToken(long passQps) { 30 long currentTime = TimeUtil.currentTimeMillis(); 31 currentTime = currentTime - currentTime % 1000; 32 long oldLastFillTime = lastFilledTime.get(); 33 if (currentTime <= oldLastFillTime) { 34 return; 35 } 36 37 long oldValue = storedTokens.get(); 38 long newValue = coolDownTokens(currentTime, passQps); 39 40 if (storedTokens.compareAndSet(oldValue, newValue)) { 41 long currentValue = storedTokens.addAndGet(0 - passQps); 42 if (currentValue < 0) { 43 storedTokens.set(0L); 44 } 45 lastFilledTime.set(currentTime); 46 } 47 48 } 49 50private long coolDownTokens(long currentTime, long passQps) { 51 long oldValue = storedTokens.get(); 52 long newValue = oldValue; 53 54 // 添加令牌的判断前提条件: 55 // 当令牌的消耗程度远远低于警戒线的时候 56 if (oldValue < warningToken) { 57 newValue = (long)(oldValue + (currentTime - lastFilledTime.get()) * count / 1000); 58 } else if (oldValue > warningToken) { 59 if (passQps < (int)count / coldFactor) { 60 newValue = (long)(oldValue + (currentTime - lastFilledTime.get()) * count / 1000); 61 } 62 } 63 return Math.min(newValue, maxToken); 64 } 65 66

2.4.3 集群限流

passClusterCheck方法(因为clusterService找不到会降级到非集群限流)

1private static boolean passClusterCheck(FlowRule rule, Context context, DefaultNode node, int acquireCount, 2 boolean prioritized) { 3 try { 4 // 获取当前节点是Token Client还是Token Server 5 TokenService clusterService = pickClusterService(); 6 if (clusterService == null) { 7 return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized); 8 } 9 long flowId = rule.getClusterConfig().getFlowId(); 10 // 根据获取的flowId通过TokenService进行申请token。从上面可知,它可能是TokenClient调用的,也可能是ToeknServer调用的。分别对应的类是DefaultClusterTokenClient和DefaultTokenService 11 TokenResult result = clusterService.requestToken(flowId, acquireCount, prioritized); 12 return applyTokenResult(result, rule, context, node, acquireCount, prioritized); 13 // If client is absent, then fallback to local mode. 14 } catch (Throwable ex) { 15 RecordLog.warn("[FlowRuleChecker] Request cluster token unexpected failed", ex); 16 } 17 // Fallback to local flow control when token client or server for this rule is not available. 18 // If fallback is not enabled, then directly pass. 19 return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized); 20 } 21 22//获取当前节点是Token Client还是Token Server。 23//1) 如果当前节点的角色是Client,返回的TokenService为DefaultClusterTokenClient; 24//2)如果当前节点的角色是Server,则默认返回的TokenService为DefaultTokenService。 25private static TokenService pickClusterService() { 26 if (ClusterStateManager.isClient()) { 27 return TokenClientProvider.getClient(); 28 } 29 if (ClusterStateManager.isServer()) { 30 return EmbeddedClusterTokenServerProvider.getServer(); 31 } 32 return null; 33 } 34 35

集群限流模式

Sentinel 集群限流服务端有两种启动方式:

•嵌入模式(Embedded)适合应用级别的限流,部署简单,但对应用性能有影响

•独立模式(Alone)适合全局限流,需要独立部署

考虑到文章篇幅,集群限流有机会再展开详细介绍。

集群限流模式降级

1private static boolean passClusterCheck(FlowRule rule, Context context, DefaultNode node, int acquireCount, 2 boolean prioritized) { 3 try { 4 TokenService clusterService = pickClusterService(); 5 if (clusterService == null) { 6 return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized); 7 } 8 long flowId = rule.getClusterConfig().getFlowId(); 9 TokenResult result = clusterService.requestToken(flowId, acquireCount, prioritized); 10 return applyTokenResult(result, rule, context, node, acquireCount, prioritized); 11 // If client is absent, then fallback to local mode. 12 } catch (Throwable ex) { 13 RecordLog.warn("[FlowRuleChecker] Request cluster token unexpected failed", ex); 14 } 15 // Fallback to local flow control when token client or server for this rule is not available. 16 // If fallback is not enabled, then directly pass. 17 // 可以看到如果集群限流有异常,会降级到单机限流模式,如果配置不允许降级,那么直接会跳过此次校验 18 return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized); 19 } 20 21

2.5 DegradeSlot

CircuitBreaker

大神对断路器的解释:https://martinfowler.com/bliki/CircuitBreaker.html

首先就看到了根据资源名称获取断路器列表,Sentinel的断路器有两个实现:RT模式使用ResponseTimeCircuitBreaker、异常模式使用ExceptionCircuitBreaker

1public interface CircuitBreaker { 2 3 /** 4 * Get the associated circuit breaking rule. 5 * 6 * @return associated circuit breaking rule 7 */ 8 DegradeRule getRule(); 9 10 /** 11 * Acquires permission of an invocation only if it is available at the time of invoking. 12 * 13 * @param context context of current invocation 14 * @return {@code true} if permission was acquired and {@code false} otherwise 15 */ 16 boolean tryPass(Context context); 17 18 /** 19 * Get current state of the circuit breaker. 20 * 21 * @return current state of the circuit breaker 22 */ 23 State currentState(); 24 25 /** 26 * <p>Record a completed request with the context and handle state transformation of the circuit breaker.</p> 27 * <p>Called when a <strong>passed</strong> invocation finished.</p> 28 * 29 * @param context context of current invocation 30 */ 31 void onRequestComplete(Context context); 32 33 /** 34 * Circuit breaker state. 35 */ 36 enum State { 37 /** 38 * In {@code OPEN} state, all requests will be rejected until the next recovery time point. 39 */ 40 OPEN, 41 /** 42 * In {@code HALF_OPEN} state, the circuit breaker will allow a "probe" invocation. 43 * If the invocation is abnormal according to the strategy (e.g. it's slow), the circuit breaker 44 * will re-transform to the {@code OPEN} state and wait for the next recovery time point; 45 * otherwise the resource will be regarded as "recovered" and the circuit breaker 46 * will cease cutting off requests and transform to {@code CLOSED} state. 47 */ 48 HALF_OPEN, 49 /** 50 * In {@code CLOSED} state, all requests are permitted. When current metric value exceeds the threshold, 51 * the circuit breaker will transform to {@code OPEN} state. 52 */ 53 CLOSED 54 } 55} 56 57

以ExceptionCircuitBreaker为例看一下具体实现

1public class ExceptionCircuitBreaker extends AbstractCircuitBreaker { 2 3 // 异常模式有两种,异常率和异常数 4 private final int strategy; 5 // 最小请求数 6 private final int minRequestAmount; 7 // 阈值 8 private final double threshold; 9 10 // LeapArray是sentinel统计数据非常重要的一个结构,主要封装了时间窗口相关的操作 11 private final LeapArray<SimpleErrorCounter> stat; 12 13 public ExceptionCircuitBreaker(DegradeRule rule) { 14 this(rule, new SimpleErrorCounterLeapArray(1, rule.getStatIntervalMs())); 15 } 16 17 ExceptionCircuitBreaker(DegradeRule rule, LeapArray<SimpleErrorCounter> stat) { 18 super(rule); 19 this.strategy = rule.getGrade(); 20 boolean modeOk = strategy == DEGRADE_GRADE_EXCEPTION_RATIO || strategy == DEGRADE_GRADE_EXCEPTION_COUNT; 21 AssertUtil.isTrue(modeOk, "rule strategy should be error-ratio or error-count"); 22 AssertUtil.notNull(stat, "stat cannot be null"); 23 this.minRequestAmount = rule.getMinRequestAmount(); 24 this.threshold = rule.getCount(); 25 this.stat = stat; 26 } 27 28 @Override 29 protected void resetStat() { 30 // Reset current bucket (bucket count = 1). 31 stat.currentWindow().value().reset(); 32 } 33 34 35 @Override 36 public void onRequestComplete(Context context) { 37 Entry entry = context.getCurEntry(); 38 if (entry == null) { 39 return; 40 } 41 Throwable error = entry.getError(); 42 SimpleErrorCounter counter = stat.currentWindow().value(); 43 if (error != null) { 44 counter.getErrorCount().add(1); 45 } 46 counter.getTotalCount().add(1); 47 48 handleStateChangeWhenThresholdExceeded(error); 49 } 50 51 private void handleStateChangeWhenThresholdExceeded(Throwable error) { 52 if (currentState.get() == State.OPEN) { 53 return; 54 } 55 56 if (currentState.get() == State.HALF_OPEN) { 57 // In detecting request 58 if (error == null) { 59 fromHalfOpenToClose(); 60 } else { 61 fromHalfOpenToOpen(1.0d); 62 } 63 return; 64 } 65 66 List<SimpleErrorCounter> counters = stat.values(); 67 long errCount = 0; 68 long totalCount = 0; 69 for (SimpleErrorCounter counter : counters) { 70 71 += counter.errorCount.sum(); 72 totalCount += counter.totalCount.sum(); 73 } 74 if (totalCount < minRequestAmount) { 75 return; 76 } 77 double curCount = errCount; 78 if (strategy == DEGRADE_GRADE_EXCEPTION_RATIO) { 79 // Use errorRatio 80 curCount = errCount * 1.0d / totalCount; 81 } 82 if (curCount > threshold) { 83 transformToOpen(curCount); 84 } 85 } 86 87 static class SimpleErrorCounter { 88 private LongAdder errorCount; 89 private LongAdder totalCount; 90 91 public SimpleErrorCounter() { 92 this.errorCount = new LongAdder(); 93 this.totalCount = new LongAdder(); 94 } 95 96 public LongAdder getErrorCount() { 97 return errorCount; 98 } 99 100 public LongAdder getTotalCount() { 101 return totalCount; 102 } 103 104 public SimpleErrorCounter reset() { 105 errorCount.reset(); 106 totalCount.reset(); 107 return this; 108 } 109 110 @Override 111 public String toString() { 112 return "SimpleErrorCounter{" + 113 "errorCount=" + errorCount + 114 ", totalCount=" + totalCount + 115 '}'; 116 } 117 } 118 119 static class SimpleErrorCounterLeapArray extends LeapArray<SimpleErrorCounter> { 120 121 public SimpleErrorCounterLeapArray(int sampleCount, int intervalInMs) { 122 super(sampleCount, intervalInMs); 123 } 124 125 @Override 126 public SimpleErrorCounter newEmptyBucket(long timeMillis) { 127 return new SimpleErrorCounter(); 128 } 129 130 @Override 131 protected WindowWrap<SimpleErrorCounter> resetWindowTo(WindowWrap<SimpleErrorCounter> w, long startTime) { 132 // Update the start time and reset value. 133 w.resetTo(startTime); 134 w.value().reset(); 135 return w; 136 } 137 } 138} 139 140

2.6 SystemSlot

校验逻辑主要集中在com.alibaba.csp.sentinel.slots.system.SystemRuleManager#checkSystem,以下是片段,可以看到,作为负载保护规则校验,实现了集群的QPS、线程、RT(响应时间)、系统负载的控制,除系统负载以外,其余统计都是依赖StatisticSlot实现,系统负载是通过SystemRuleManager定时调度SystemStatusListener,通过OperatingSystemMXBean去获取

1/** 2 * Apply {@link SystemRule} to the resource. Only inbound traffic will be checked. 3 * 4 * @param resourceWrapper the resource. 5 * @throws BlockException when any system rule's threshold is exceeded. 6 */ 7 public static void checkSystem(ResourceWrapper resourceWrapper, int count) throws BlockException { 8 if (resourceWrapper == null) { 9 return; 10 } 11 // Ensure the checking switch is on. 12 if (!checkSystemStatus.get()) { 13 return; 14 } 15 16 // for inbound traffic only 17 if (resourceWrapper.getEntryType() != EntryType.IN) { 18 return; 19 } 20 21 // total qps 此处是拿到某个资源在集群中的QPS总和,相关概念可以会看初始化关于Node的介绍 22 double currentQps = Constants.ENTRY_NODE.passQps(); 23 if (currentQps + count > qps) { 24 throw new SystemBlockException(resourceWrapper.getName(), "qps"); 25 } 26 27 // total thread 28 int currentThread = Constants.ENTRY_NODE.curThreadNum(); 29 if (currentThread > maxThread) { 30 throw new SystemBlockException(resourceWrapper.getName(), "thread"); 31 } 32 33 double rt = Constants.ENTRY_NODE.avgRt(); 34 if (rt > maxRt) { 35 throw new SystemBlockException(resourceWrapper.getName(), "rt"); 36 } 37 38 // load. BBR algorithm. 39 if (highestSystemLoadIsSet && getCurrentSystemAvgLoad() > highestSystemLoad) { 40 if (!checkBbr(currentThread)) { 41 throw new SystemBlockException(resourceWrapper.getName(), "load"); 42 } 43 } 44 45 // cpu usage 46 if (highestCpuUsageIsSet && getCurrentCpuUsage() > highestCpuUsage) { 47 throw new SystemBlockException(resourceWrapper.getName(), "cpu"); 48 } 49 } 50 51 private static boolean checkBbr(int currentThread) { 52 if (currentThread > 1 && 53 currentThread > Constants.ENTRY_NODE.maxSuccessQps() * Constants.ENTRY_NODE.minRt() / 1000) { 54 return false; 55 } 56 return true; 57 } 58 59 public static double getCurrentSystemAvgLoad() { 60 return statusListener.getSystemAverageLoad(); 61 } 62 63 public static double getCurrentCpuUsage() { 64 return statusListener.getCpuUsage(); 65 } 66 67
1public class SystemStatusListener implements Runnable { 2 3 volatile double currentLoad = -1; 4 volatile double currentCpuUsage = -1; 5 6 volatile String reason = StringUtil.EMPTY; 7 8 volatile long processCpuTime = 0; 9 volatile long processUpTime = 0; 10 11 public double getSystemAverageLoad() { 12 return currentLoad; 13 } 14 15 public double getCpuUsage() { 16 return currentCpuUsage; 17 } 18 19 @Override 20 public void run() { 21 try { 22 OperatingSystemMXBean osBean = ManagementFactory.getPlatformMXBean(OperatingSystemMXBean.class); 23 currentLoad = osBean.getSystemLoadAverage(); 24 25 /* 26 * Java Doc copied from {@link OperatingSystemMXBean#getSystemCpuLoad()}:</br> 27 * Returns the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. 28 * A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value 29 * of 1.0 means that all CPUs were actively running 100% of the time during the recent period being 30 * observed. All values between 0.0 and 1.0 are possible depending of the activities going on in the 31 * system. If the system recent cpu usage is not available, the method returns a negative value. 32 */ 33 double systemCpuUsage = osBean.getSystemCpuLoad(); 34 35 // calculate process cpu usage to support application running in container environment 36 RuntimeMXBean runtimeBean = ManagementFactory.getPlatformMXBean(RuntimeMXBean.class); 37 long newProcessCpuTime = osBean.getProcessCpuTime(); 38 long newProcessUpTime = runtimeBean.getUptime(); 39 int cpuCores = osBean.getAvailableProcessors(); 40 long processCpuTimeDiffInMs = TimeUnit.NANOSECONDS 41 .toMillis(newProcessCpuTime - processCpuTime); 42 long processUpTimeDiffInMs = newProcessUpTime - processUpTime; 43 double processCpuUsage = (double) processCpuTimeDiffInMs / processUpTimeDiffInMs / cpuCores; 44 processCpuTime = newProcessCpuTime; 45 processUpTime = newProcessUpTime; 46 47 currentCpuUsage = Math.max(processCpuUsage, systemCpuUsage); 48 49 if (currentLoad > SystemRuleManager.getSystemLoadThreshold()) { 50 writeSystemStatusLog(); 51 } 52 } catch (Throwable e) { 53 RecordLog.warn("[SystemStatusListener] Failed to get system metrics from JMX", e); 54 } 55 } 56 57 private void writeSystemStatusLog() { 58 StringBuilder sb = new StringBuilder(); 59 sb.append("Load exceeds the threshold: "); 60 sb.append("load:").append(String.format("%.4f", currentLoad)).append("; "); 61 sb.append("cpuUsage:").append(String.format("%.4f", currentCpuUsage)).append("; "); 62 sb.append("qps:").append(String.format("%.4f", Constants.ENTRY_NODE.passQps())).append("; "); 63 sb.append("rt:").append(String.format("%.4f", Constants.ENTRY_NODE.avgRt())).append("; "); 64 sb.append("thread:").append(Constants.ENTRY_NODE.curThreadNum()).append("; "); 65 sb.append("success:").append(String.format("%.4f", Constants.ENTRY_NODE.successQps())).append("; "); 66 sb.append("minRt:").append(String.format("%.2f", Constants.ENTRY_NODE.minRt())).append("; "); 67 sb.append("maxSuccess:").append(String.format("%.2f", Constants.ENTRY_NODE.maxSuccessQps())).append("; "); 68 RecordLog.info(sb.toString()); 69 } 70} 71 72

三、京东版最佳实践

3.1 使用方式

Sentinel使用方式本身非常简单,就是一个注解,但是要考虑规则加载和规则持久化的方式,现有的方式有:

•使用Sentinel-dashboard功能:使用面板接入需要维护一个配置规则的管理端,考虑到偏后端的系统需要额外维护一个面板成本较大,如果是像RPC框架这种本身有管理端的接入可以考虑次方案。

•中间件(如:zookepper、nacos、eureka、redis等):Sentinel源码extension包里提供了类似的实现,如下图

结合京东实际,我实现了一个规则热部署的Sentinel组件,实现方式类似zookeeper的方式,将规则记录到ducc的一个key上,在spring容器启动时做第一次规则加载和监听器注册,组件也做一了一些规则读取,校验、实例化不同规则对象的工作

插件使用方式:注解+配置

第一步 引入组件

1<dependency> 2 <groupId>com.jd.ldop.tools</groupId> 3 <artifactId>sentinel-tools</artifactId> 4 <version>1.0.0-SNAPSHOT</version> 5</dependency> 6 7

第二步 初始化sentinelProcess

支持ducc、本地文件读取、直接写入三种方式规则写入方式

目前支持限流规则、熔断降级规则两种模式,系统负载保护模式待开发和验证

1<!-- 基于sentinel的降级、限流、熔断组件 --> 2 <bean id="sentinelProcess" class="com.jd.ldop.sentinel.SentinelProcess"> 3 <property name="ruleResourceWrappers"> 4 <list> 5 <ref bean="degradeRule"/> 6 </list> 7 </property> 8 </bean> 9 10 <!-- 降级或限流规则配置 --> 11 <bean id="degradeRule" class="com.jd.ldop.sentinel.dto.RuleResourceWrapper"> 12 <constructor-arg index="0" value="ducc.degradeRule"/> 13 <constructor-arg index="1" value="0"/> 14 <constructor-arg index="2" value="0"/> 15 </bean> 16 17

ducc上配置如下:

第三步 定义资源和关联类型

通过@SentinelResource可以直接在任意位置定义资源名以及对应的熔断降级或者限流方式、回调方法等,同时也可以指定关联类型,支持直接、关联、指定链路三种

1 @Override 2 @SentinelResource(value = "modifyGetWaybillState", fallback = "executeDegrade") 3 public ExecutionResult<List<Integer>> execute(@NotNull Model imodel) { 4 // 业务逻辑处理 5 } 6 7 public ExecutionResult<List<Integer>> executeDegrade(@NotNull Model imodel) { 8 // 降级业务逻辑处理 9 } 10 11

3.2 应用场景

组件支持任意的业务降级、限流、负载保护

四、Sentinel压测数据

4.1 压测目标

调用量:1.2W/m

应用机器内存稳定在50%以内

机器规格: 8C16G50G磁盘*2

Sentinel降级规则:

count=350-------慢调用临界阈值350ms

timeWindow=180------熔断时间窗口180s

grade=0-----降级模式 慢调用

statIntervalMs=60000------统计时长1min

4.2 压测结果

应用机器监控:

压测分为了两个阶段,分别是组件开启和组件关闭两次,前半部分是组件开启的情况,后半部分是组件关闭的情况

应用进程内存分析,和sentinel有关的前三对象是

com.alibaba.csp.sentinel.node.metric.MetricNode

com.alibaba.csp.sentinel.CtEntry

com.alibaba.csp.sentinel.context.Context

4.3 压测结论

使Sentinel组件实现系统服务自动降级或限流,由于sentinel会按照滑动窗口周期性统计数据,因此会占用一定的机器内存,使用时应设置合理的规则,如:合理的统计时长、避免过多的Sentinel资源创建等。

总体来说,使用sentinel组件对应用cpu和内存影响不大。

点赞
收藏

评论区

加载中...

相关推荐

Oracle 分组与拼接字符串同时使用

SELECTT.,ROWNUMIDFROM(SELECTT.EMPLID,T.NAME,T.BU,T.REALDEPART,T.FORMATDATE,SUM(T.S0)S0,MAX(UPDATETIME)CREATETIME,LISTAGG(TOCHAR(

Sentinel在docker中获取CPU利用率的一个BUG

Sentinel简介微服务治理中限流、熔断、降级是一块非常重要的内容。目前市面上开源的组件也不是很多,简单场景可以使用Guava,复杂场景可以选用Hystrix、Sentinel。今天要说的就是Sentinel,Sentinel是一款阿里开源的产品,只需要做较少的定制开发即可大规模线上使用。从使用感受上来说,它有以下几个优点:轻量级,对性能损耗几乎可以忽略

高并发场景下常见的限流算法及方案介绍

现代互联网很多业务场景,比如秒杀、下单、查询商品详情,最大特点就是高并发,而往往我们的系统不能承受这么大的流量,这时候限流熔断就发挥作用了,限制请求数,快速失败,保证系统满负载又不超限。本文为大家介绍几种常见的限流算法及方案

Spring Boot集成 Sentinel 实现接口流量控制

Hello,大家好,我是麦洛,今天带大家来了解一下SpringBoot如何继承Sentinel来实现接口流量控制Sentinel控制台搭建在我的上一篇文章阿里出品的Sentinel到底是个什么玩意?中,已经介绍过如何准备Sentinel控制台,大家可以直接参考;Sentinel客户端项目搭建首先我们来创建一个测试项目,这里初始化

Spring Cloud Alibaba:Sentinel实现熔断与限流

一、什么是SentinelSentinel,中文翻译为哨兵,是为微服务提供流量控制、熔断降级的功能,它和Hystrix提供的功能一样,可以有效的解决微服务调用产生的“雪崩效应”,为微服务系统提供了稳定性的解决方案。随着Hystrix进入了维护期,不再提供新功能,Sentinel是一个不错的替代方案。通常情况下,Hystrix采用线程池对服务的调用

Dubbo使用Sentinel来对服务进行降级与限流

一、Sentinel是什么Sentinel是阿里中间件团队开源的,面向分布式服务架构的轻量级流量控制产品,主要以流量为切入点,从流量控制、熔断降级、系统负载保护等多个维度来帮助用户保护服务的稳定性。点此地址了解更多Sentinel(https://www.oschina.net/action/GoToLink?ur