Kubernetes自定义调度器 — 初窥门径

Description 通过上一篇文章对scheduler-framework调度框架已经有了大致了解，根据我们的实际生产的一些问题(如计算服务没有被调度到实际CPU最优的节点)和需求，来实现一个简单的基于CPU指标的自定义调度器。自定义调度器通过kubernetes资源指标服务metrics-server来获取各节点的当前的资源情况，并进行打分，然后把Pod调度到分数最高的节点

PreFilter扩展点

对Pod的信息进行预处理，检查Pod或集群是否满足前提条件。

通过pod已声明的Annotations参数 rely.on.namespaces/name 和 rely.on.pod/labs来获取该pod依赖的pod是否就绪，如依赖的pod未就绪，则终止调度，pod处于Pending状态。

1func (pl *sample) PreFilter(ctx context.Context, state *framework.CycleState, p *v1.Pod) *framework.Status {
2
3  namespace := p.Annotations["rely.on.namespaces/name"]
4  podLabs := p.Annotations["rely.on.pod/labs"]
5
6  if namespace == "" || podLabs == "" {
7    return framework.NewStatus(framework.Success, "ont rely")
8  }
9  if prefilter.IsExist(namespace) == false {
10    return framework.NewStatus(framework.Unschedulable, "not found namespace: "+namespace)
11  }
12
13  if prefilter.IsReady(namespace, podLabs) == false {
14    return framework.NewStatus(framework.Unschedulable, "rely pod not ready")
15  }
16  klog.Infoln("rely pod is ready :", namespace, podLabs, prefilter.IsReady(namespace, podLabs))
17  return framework.NewStatus(framework.Success, "rely pod is ready")
18}

调度失败结果：

1Events:
2  Type     Reason            Age   From              Message
3  ----     ------            ----  ----              -------
4  Warning  FailedScheduling  9s    sample-scheduler  0/2 nodes are available: 2 rely pod not ready.
5  Warning  FailedScheduling  9s    sample-scheduler  0/2 nodes are available: 2 rely pod not ready.

如满足Pod的前置条件，则正常调度，进入下一阶段

1I1025 12:26:59.435773       1 plugins.go:50] rely pod is ready : kube-system k8s-app=metrics-server true

Filter扩展点

对不满足Pod调度要求的节点进行过滤掉。

1
2func (pl *sample) Filter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, node *framework.NodeInfo) *framework.Status {
3
4  if node.Node().Labels["cpu"] != "true" {
5    return framework.NewStatus(framework.Unschedulable, "not found labels")
6  }
7  nodeUsedCPU, nodeUsedMen, nodeCPU, nodeMen, cpuRate, menRate := filter.ResourceStatus(node.Node().Name)
8
9  for i := 0; i < len(pod.Spec.Containers); i++ {
10    requestsCPUCore, _ := strconv.ParseFloat(strings.Replace(pod.Spec.Containers[i].Resources.Requests.Cpu().String(), "n", "", 1), 64)
11    requestsCPU := requestsCPUCore * 1000 * (1000 * 1000)
12    requestsMen := pod.Spec.Containers[i].Resources.Requests.Memory().Value() / 1024 / 1024
13    limitsCPUCore, _ := strconv.ParseFloat(strings.Replace(pod.Spec.Containers[i].Resources.Limits.Cpu().String(), "n", "", 1), 64)
14    limitsCPU := limitsCPUCore * 1000 * (1000 * 1000)
15    limitsMen := pod.Spec.Containers[i].Resources.Limits.Memory().Value() / 1024 / 1024
16    if requestsCPU > float64(nodeCPU) || requestsMen > nodeMen {
17      return framework.NewStatus(framework.Unschedulable, "out of Requests resources")
18    }
19    if limitsCPU > float64(nodeCPU) || limitsMen > nodeMen {
20      return framework.NewStatus(framework.Unschedulable, "out of Limits resources")
21    }
22    if requestsCPU > float64(nodeCPU)-nodeUsedCPU || requestsMen > (nodeMen-nodeUsedMen) {
23      return framework.NewStatus(framework.Unschedulable, "out of Requests resources system")
24    }
25    if limitsCPU > float64(nodeCPU)-nodeUsedCPU || limitsMen > (nodeMen-nodeUsedMen) {
26      return framework.NewStatus(framework.Unschedulable, "out of Limits resources system")
27    }
28
29  }
30
31  klog.Infof("node:%s, CPU:%v ,  Memory: %v", node.Node().Name, cpuRate, menRate)
32  cpuThreshold := filter.GetEnvFloat("CPU_THRESHOLD", 0.85)
33  menThreshold := filter.GetEnvFloat("MEN_THRESHOLD", 0.85)
34  if cpuRate > cpuThreshold || menRate > menThreshold {
35    return framework.NewStatus(framework.Unschedulable, "out of system resources")
36  }
37
38  return framework.NewStatus(framework.Success, "Node: "+node.Node().Name)
39}

过滤掉没有 cpu=true labels的节点； Pod调度资源值和限制资源值大于节点当前可用资源的节点，则过滤；默认过滤cpu内存使用率超过 85% 的节点。 CPU_THRESHOLD 和 MEN_THRESHOLD 环境变量设置该值。调度失败结果：

1Events:
2  Type     Reason            Age    From              Message
3  ----     ------            ----   ----              -------
4  Warning  FailedScheduling  9m33s  sample-scheduler  0/2 nodes are available: 1 Insufficient cpu, 1 out of Limits resources system.
5  Warning  FailedScheduling  9m33s  sample-scheduler  0/2 nodes are available: 1 Insufficient cpu, 1 out of Limits resources system.

可以通过日志查看当前获取到的系统资源利用率

1I1025 12:29:21.360979       1 plugins.go:87] node:k8s-test09, CPU:0.06645718025 ,  Memory: 0.33549941504789904
2I1025 12:29:21.361471       1 plugins.go:87] node:k8s-test10, CPU:0.0411259705 ,  Memory: 0.15946609553600768

Score扩展点

用于对已通过过滤阶段的节点进行打分。

1func (pl *sample) Score(ctx context.Context, state *framework.CycleState, p *v1.Pod, nodeName string) (int64, *framework.Status) {
2
3  isSamePod := score.IsSamePod(nodeName, p.Namespace, p.Labels) // max 2
4  cpuLoad := score.CPURate(nodeName)                            // max 3
5  menLoad := score.MemoryRate(nodeName)                         // max 3
6  core := score.CpuCore(nodeName)                               // max 3
7
8  c := isSamePod + cpuLoad + core + menLoad
9  klog.Infoln(nodeName+" score is :", c)
10  return c, framework.NewStatus(framework.Success, nodeName)
11}

打分规则：

配置高的节点权重大
当前资源使用率底的节点权重大
运行多组pod的情况下，运行相同pod的节点权重底
如果Score插件的打分结果不是[0-100]范围内的整数，则调用NormalizeScore进行归一化。

调度结果：

1I1025 12:28:35.620198       1 plugins.go:105] k8s-test10 score is : 6
2I1025 12:28:35.626170       1 plugins.go:105] k8s-test09 score is : 7
3I1025 12:28:35.626394       1 trace.go:205] Trace[232544630]: "Scheduling" namespace:default,name:test-scheduler-84859f9467-rgfpd (25-Oct-2020 12:28:35.488) (total time: 137ms):
4Trace[232544630]: ---"Computing predicates done" 38ms (12:28:00.526)
5Trace[232544630]: ---"Prioritizing done" 99ms (12:28:00.626)
6Trace[232544630]: [137.453956ms] [137.453956ms] END
7I1025 12:28:35.626551       1 default_binder.go:51] Attempting to bind default/test-scheduler-84859f9467-rgfpd to k8s-test09
8I1025 12:28:35.632736       1 eventhandlers.go:205] delete event for unscheduled pod default/test-scheduler-84859f9467-rgfpd
9I1025 12:28:35.632893       1 scheduler.go:597] "Successfully bound pod to node" pod="default/test-scheduler-84859f9467-rgfpd" node="k8s-test09" evaluatedNodes=2 feasibleNodes=2
10I1025 12:28:35.633369       1 eventhandlers.go:225] add event for scheduled pod default/test-scheduler-84859f9467-rgfpd 
11I1025 12:29:21.328791       1 eventhandlers.go:173] add event for unscheduled pod default/test-scheduler-84859f9467-48wlr
12I1025 12:29:21.328897       1 scheduler.go:452] Attempting to schedule pod: default/test-scheduler-84859f9467-48wlr
13I1025 12:29:21.351126       1 plugins.go:50] rely pod is ready : kube-system k8s-app=metrics-server true
14I1025 12:29:21.360979       1 plugins.go:87] node:k8s-test09, CPU:0.06645718025 ,  Memory: 0.33549941504789904
15I1025 12:29:21.361471       1 plugins.go:87] node:k8s-test10, CPU:0.0411259705 ,  Memory: 0.15946609553600768
16I1025 12:29:21.419137       1 plugins.go:105] k8s-test10 score is : 6
17I1025 12:29:21.419210       1 plugins.go:105] k8s-test09 score is : 5
18I1025 12:29:21.419407       1 default_binder.go:51] Attempting to bind default/test-scheduler-84859f9467-48wlr to k8s-test10
19I1025 12:29:21.427243       1 scheduler.go:597] "Successfully bound pod to node" pod="default/test-scheduler-84859f9467-48wlr" node="k8s-test10" evaluatedNodes=2 feasibleNodes=2
20I1025 12:29:21.427441       1 eventhandlers.go:205] delete event for unscheduled pod default/test-scheduler-84859f9467-48wlr
21I1025 12:29:21.427472       1 eventhandlers.go:225] add event for scheduled pod default/test-scheduler-84859f9467-48wlr

完整的示例代码： https://github.com/prodanlabs/scheduler-framework Description

感兴趣的读者可以关注下微信号 Description

Kubernetes自定义调度器 — 初窥门径

PreFilter扩展点

Filter扩展点

Score扩展点

评论区

相关推荐