<section id="nice" data-tool="mdnice编辑器" data-website="https://www.mdnice.com" style="word-spacing: 0px; word-break: break-word; word-wrap: break-word; font-size: 16px; line-height: 1.75em; letter-spacing: 0.5px; text-align: justify; color: #3f3f3f; padding: 0 8px 0 8px; font-family: system-ui, -apple-system, BlinkMacSystemFont, 'Helvetica Neue', 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei UI', 'Microsoft YaHei', Arial, sans-serif !important;"><p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">应用异常时，基本可以分为<strong style="font-weight: bold; color: #4e6cb4;">服务访问不通</strong>和<strong style="font-weight: bold; color: #4e6cb4;">服务响应慢</strong>两个大类。其中服务响应慢的问题定位非常棘手，很多无头案。应用团队有日志和追踪，对于自认为的不可能不合理的事情都会甩给基础设施团队，又由于基础设施团队现有的监控数据缺乏应用的观测视角，通常成为一切「不是我的问题」超自然现象的终极背锅侠，其中以网络团队尤为严重。</p> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 34px; color: #1d51ce; margin-bottom: 0; margin-top: calc(2 * 27px);"><span class="prefix" style="display: none;"></span><span class="content">01</span><span class="suffix"></span></h1> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 18px; margin-top: 5px; margin-bottom: 32px; color: #1d51ce;"><span class="prefix" style="display: none;"></span><span class="content">响应时延</span><span class="suffix"></span></h1> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;"><strong style="font-weight: bold; color: #4e6cb4;">服务为什么响应慢</strong>？？？首先，我们需要一种方式来度量何为响应慢，参考 Google 在 SRE Handbook 中提到过<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">4 个黄金信号</code>及 Weave Cloud 提出来的 <code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">RED 方法</code>，都存在度量的指标（Latency/Duration），后文统称为<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">响应时延</code>。</p> <ul data-tool="mdnice编辑器" style="margin-top: 8px; padding-left: 25px; list-style-type: disc; color: #1d51ce; margin: 0; margin-bottom: 27px;"> <li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">Latency 表达的是<strong style="font-weight: bold; color: #4e6cb4;">服务处理某个请求所需要的时间</strong>，站在的是服务端视角</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">Duration 表达的是<strong style="font-weight: bold; color: #4e6cb4;">每个请求所花费的时间</strong>，站在的是客户端视角</section></li></ul> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">总结下来，不论站在什么视角，<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">响应时延</code>表达的都是处理一个请求所花费的时间，可以用来<strong style="font-weight: bold; color: #4e6cb4;">表征服务响应慢的度量指标</strong>，但若要定位为什么响应慢还需要进一步剖解响应时延：</p> <ul data-tool="mdnice编辑器" style="margin-top: 8px; padding-left: 25px; list-style-type: disc; color: #1d51ce; margin: 0; margin-bottom: 27px;"> <li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">系统时延：系统转发请求/响应的时延消耗</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">网络时延：包含查询 DNS 时延及网络处理的时延</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">应用时延：从不同视角来看，包含客户端应用处理时延 + 服务端应用处理时延</section></li></ul> <img src="https://yunshan-guangzhou.oss-cn-beijing.aliyuncs.com/pub/pic/2023030364019bdb9438d.jpg" alt="响应时延拆解" style="display: block; margin: 0 auto; max-width: 100%;"> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">确定度量指标后，接下就可以分析服务响应慢的原因，此时可以利用<strong style="font-weight: bold; color: #4e6cb4;">分布式链路追踪</strong>能力来快速来<strong style="font-weight: bold; color: #4e6cb4;">定界瓶颈点</strong>，例如可利用 DeepFlow 的分布式追踪能力来快速定界瓶颈点在应用、系统还是网络。</p> <img src="https://yunshan-guangzhou.oss-cn-beijing.aliyuncs.com/pub/pic/2022082363044b24c3b37.png" alt="分布式链路追踪-火焰图" style="display: block; margin: 0 auto; max-width: 100%;">分布式链路追踪-火焰图 <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">完成瓶颈点定界后，则需要去查找根因。对于应用或者系统的问题，可以利用<strong style="font-weight: bold; color: #4e6cb4;">性能剖析（profile）继续追查根因</strong>，而对应网络时延的分析，其中 DNS 时延分析是相对简单的，只需要关注请求的响应时延即可，但网络处理时延瓶颈的定位却缺少了分析的工具，接下来将主要聚焦讨论网络传输时延如何分析。</p> <img src="https://yunshan-guangzhou.oss-cn-beijing.aliyuncs.com/pub/pic/2023030364019bdb941e8.jpg" alt="性能剖析-火焰图" style="display: block; margin: 0 auto; max-width: 100%;">性能剖析-火焰图 <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 34px; color: #1d51ce; margin-bottom: 0; margin-top: calc(2 * 27px);"><span class="prefix" style="display: none;"></span><span class="content">02</span><span class="suffix"></span></h1> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 18px; margin-top: 5px; margin-bottom: 32px; color: #1d51ce;"><span class="prefix" style="display: none;"></span><span class="content">网络时延</span><span class="suffix"></span></h1> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">参考 AWS 中的定义网络时延是指网络通信中的延时，网络时延显示了数据通过网络传输所需的时间。<strong style="font-weight: bold; color: #4e6cb4;">讨论网络时延如何，也是需要可度量的指标</strong>，AWS 也指定了使用“首字节时间”和“往返时间”等指标来衡量网络时延，这两个指标是可以适用于所有网络协议的传输时延的度量，但实际应用 80% 都使用的 TCP 协议，对于 TCP 协议是需要更细粒度的度量指标，下文通过图文的形式，详细的介绍目前可用的度量指标及用法。</p> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">TCP 协议是面向连接的传输层通信协议，对其详细的通信过程分析，时延可分为三大类：</p> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;"><strong style="font-weight: bold; color: #4e6cb4;">1)建连时产生的时延</strong></p> <ul data-tool="mdnice编辑器" style="margin-top: 8px; padding-left: 25px; list-style-type: disc; color: #1d51ce; margin: 0; margin-bottom: 27px;"> <li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">[1] 完整的<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">建连时延</code>包含客户端发出 SYN 包到收到服务端回复的 SYN+ACK 包，并再次回复 ACK 包的整个时间。建连时延拆解开又可分为<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">客户端建连时延</code>与<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">服务端建连时延</code></section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">[2] <code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">客户端建连时延</code>为客户端收到 SYN+ACK 包后，回复 ACK 包的时间</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">[3] <code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">服务端建连时延</code>为服务端收到 SYN 包后，回复 SYN+ACK 包的时间</section></li></ul> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;"><strong style="font-weight: bold; color: #4e6cb4;">2)数据通信时产生的时延，可拆解为</strong><code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">客户端等待时延</code>+<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">数据传输时延</code></p> <ul data-tool="mdnice编辑器" style="margin-top: 8px; padding-left: 25px; list-style-type: disc; color: #1d51ce; margin: 0; margin-bottom: 27px;"> <li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">[1] <code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">客户端等待时延</code>为建连成功后，客户端首次发送请求的时间；为收到服务端的数据包后，客户端再发起数据包的时间</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">[2] <code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">数据传输时延</code>为客户端发送数据包到收到服务端回复数据包的时间</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">[3] 在数据传输时延中还会产生系统协议栈的处理时延，称为<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">系统时延</code></section></li></ul> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;"><strong style="font-weight: bold; color: #4e6cb4;">3)断连时产生的时延：</strong>因为断连的时延并不影响到应用的响应时延，因此并不会单独统计此部分使用</p> <img src="https://yunshan-guangzhou.oss-cn-beijing.aliyuncs.com/pub/pic/2023030364019bdd98b78.jpg" alt="TCP 网络时延解剖" style="display: block; margin: 0 auto; max-width: 100%;">TCP 网络时延解剖 <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;"><strong style="font-weight: bold; color: #4e6cb4;">度量的网络时延的指标已经拆解好了，接下来讨论在哪里采集指标</strong>，网络的报文将在客户端，各种虚拟和物理网络与服务端之间穿梭，因此可报文穿梭的位置点来采集，后续统称为<code style="font-size: 14px; word-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0 2px; font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; word-break: break-all; color: #376bb4; background-color: #f3f3f3;">统计位置</code>。当然统计位置越多，定位网络的瓶颈路径越快，但是统计位置多则随之带来的计算量也是成倍增加，企业在有成本压力时，建议在重要节点进行采集即可，比如 K8s Pod 虚拟网卡、K8s Node 网卡、云服务器网卡、网关（如 LVS/Nginx 等）网卡、硬件防火墙/负载均衡器前后......</p> <img src="https://yunshan-guangzhou.oss-cn-beijing.aliyuncs.com/pub/pic/2023030364019bdbab68c.jpg" alt="统计位置" style="display: block; margin: 0 auto; max-width: 100%;">统计位置 <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">分析到这，<strong style="font-weight: bold; color: #4e6cb4;">基本已经清晰网络时延的详细的度量指标了，回过头结合响应时延再讨论下如何查看网络时延对其的影响</strong>，基本可以分两种情况讨论:<br><br> <strong style="font-weight: bold; color: #4e6cb4;">a)应用发起请求为短连接</strong>：此时分析网络时延需要查看 <strong style="font-weight: bold; color: #4e6cb4;">DNS 时延 + 建连时延 + 客户端等待时延 + 数据传输时延 + 系统时延</strong>，则可快速定位时延发生的具体原因了。</p> <ul data-tool="mdnice编辑器" style="margin-top: 8px; padding-left: 25px; list-style-type: disc; color: #1d51ce; margin: 0; margin-bottom: 27px;"> <li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;"><strong style="font-weight: bold; color: #4e6cb4;">DNS 时延高</strong>，结合统计位置，则可回答是<strong style="font-weight: bold; color: #4e6cb4;">网络传输时延高</strong>还是<strong style="font-weight: bold; color: #4e6cb4;">DNS 服务响应慢</strong></section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;"><strong style="font-weight: bold; color: #4e6cb4;">建连时延高</strong>，结合客户端建连时延 + 服务端建连时延 + 统计位置，则可回答是<strong style="font-weight: bold; color: #4e6cb4;">网络传输时延高</strong>还是<strong style="font-weight: bold; color: #4e6cb4;">客户端系统回复慢</strong>还是<strong style="font-weight: bold; color: #4e6cb4;">服务端处理建连响应慢</strong></section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;"><strong style="font-weight: bold; color: #4e6cb4;">客户端等待时延高</strong>，结合统计位置，则可回答是<strong style="font-weight: bold; color: #4e6cb4;">网络传输时延高</strong>还是<strong style="font-weight: bold; color: #4e6cb4;">客户端请求发送延迟</strong></section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;"><strong style="font-weight: bold; color: #4e6cb4;">数据传输时延高</strong>，结合统计位置，则可回答是<strong style="font-weight: bold; color: #4e6cb4;">网络传输时延高</strong>还是<strong style="font-weight: bold; color: #4e6cb4;">服务端响应慢</strong></section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;"><strong style="font-weight: bold; color: #4e6cb4;">系统时延高</strong>，结合统计位置，则可回答<strong style="font-weight: bold; color: #4e6cb4;">网络传输时延高</strong>还是<strong style="font-weight: bold; color: #4e6cb4;">服务端协议栈处理慢</strong></section></li></ul> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;"><strong style="font-weight: bold; color: #4e6cb4;">b )应用发起请求为长连接</strong>：因为长连接是保持长期活动的 HTTP 连接，不需要考虑 DNS 查询与建连的时延消耗，只需要关注<strong style="font-weight: bold; color: #4e6cb4;">客户端等待时延 + 数据传输时延 + 系统时延</strong>即可</p> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 34px; color: #1d51ce; margin-bottom: 0; margin-top: calc(2 * 27px);"><span class="prefix" style="display: none;"></span><span class="content">03</span><span class="suffix"></span></h1> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 18px; margin-top: 5px; margin-bottom: 32px; color: #1d51ce;"><span class="prefix" style="display: none;"></span><span class="content">案例分析</span><span class="suffix"></span></h1> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">限于笔者时间限制又想早点将<strong style="font-weight: bold; color: #4e6cb4;">应用响应时延背后深藏的网络时延</strong>剖解分享给大家，本文不继续补充实际案例，将在一周后分享在某xx智能终端公司的如何结合 DeepFlow 在<strong style="font-weight: bold; color: #4e6cb4;">服务响应慢时，网络团队在存在可观测性的时延数据时，如何硬气回怼。</strong></p> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 34px; color: #1d51ce; margin-bottom: 0; margin-top: calc(2 * 27px);"><span class="prefix" style="display: none;"></span><span class="content">04</span><span class="suffix"></span></h1> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 18px; margin-top: 5px; margin-bottom: 32px; color: #1d51ce;"><span class="prefix" style="display: none;"></span><span class="content">什么是 DeepFlow</span><span class="suffix"></span></h1> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;"><span class="footnote-word" style="font-weight: bold; color: #3f3f3f;">DeepFlow</span><sup class="footnote-ref" style="line-height: 0; font-weight: bold; color: #3f3f3f;">[1]</sup> 是一款开源的高度自动化的可观测性平台，是为云原生应用开发者建设可观测性能力而量身打造的全栈、全链路、高性能数据引擎。DeepFlow 使用 eBPF、WASM、OpenTelemetry 等新技术，创新的实现了 AutoTracing、AutoMetrics、AutoTagging、SmartEncoding 等核心机制，帮助开发者提升埋点插码的自动化水平，降低可观测性平台的运维复杂度。利用 DeepFlow 的可编程能力和开放接口，开发者可以快速将其融入到自己的可观测性技术栈中。</p> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">GitHub 地址：https://github.com/deepflowys/deepflow</p> <p data-tool="mdnice编辑器" style="padding-top: 8px; padding-bottom: 8px; margin: 0; padding: 0; font-size: 16px; line-height: 1.6em; margin-bottom: 27px; color: #3f3f3f;">访问 <span class="footnote-word" style="font-weight: bold; color: #3f3f3f;">DeepFlow Demo</span><sup class="footnote-ref" style="line-height: 0; font-weight: bold; color: #3f3f3f;">[2]</sup>，体验高度自动化的可观测性新时代。</p> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 34px; color: #1d51ce; margin-bottom: 0; margin-top: calc(2 * 27px);"><span class="prefix" style="display: none;"></span><span class="content">05</span><span class="suffix"></span></h1> <h1 data-tool="mdnice编辑器" style="font-weight: bold; margin: 0; padding: 0; font-size: 18px; margin-top: 5px; margin-bottom: 32px; color: #1d51ce;"><span class="prefix" style="display: none;"></span><span class="content">参考文档</span><span class="suffix"></span></h1> <ul data-tool="mdnice编辑器" style="margin-top: 8px; padding-left: 25px; list-style-type: disc; color: #1d51ce; margin: 0; margin-bottom: 27px;"> <li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">https://aws.amazon.com/cn/what-is/latency/</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">https://baike.baidu.com/item/%E7%B3%BB%E7%BB%9F%E5%93%8D%E5%BA%94%E6%97%B6%E9%97%B4/22026261</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">https://dev.to/aws/why-are-services-slow-sometimes-mn3</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">https://yunlzheng.gitbook.io/prometheus-book/parti-prometheus-ji-chu/promql/prometheus-promql-best-praticase</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-architecture/</section></li><li><section style="padding: 0; margin-top: 5px; margin-bottom: 5px; text-align: left; font-weight: 500; color: #3f3f3f; line-height: 1.6em; margin: 0;">https://aws.amazon.com/what-is/latency/</section></li></ul> <h3 class="footnotes-sep" data-tool="mdnice编辑器" style="margin-top: 30px; margin-bottom: 15px; font-weight: bold; margin: 0; padding: 0; font-size: 16px; color: #1d51ce; border-left: 3px solid #1d51ce; padding-left: 8px;"><span style="display: block;">参考资料</span></h3> <section class="footnotes" data-tool="mdnice编辑器" style="margin: 0; padding: 0;"> <span id="fn1" class="footnote-item" style="display: flex;"><span class="footnote-num" style="display: inline; width: 10%; background: none; font-size: 80%; opacity: 0.6; line-height: 26px; font-family: ptima-Regular, Optima, PingFangSC-light, PingFangTC-light, 'PingFang SC', Cambria, Cochin, Georgia, Times, 'Times New Roman', serif;">[1] </span><p style="padding-top: 8px; padding-bottom: 8px; margin-bottom: 27px; display: inline; font-size: 14px; width: 90%; padding: 0px; margin: 0; line-height: 26px; word-break: break-all; width: calc(100%-50); color: #1d51ce;">DeepFlow: <em style="font-style: italic; color: #888888;">https://github.com/deepflowys/deepflow</em></p> </span> <span id="fn2" class="footnote-item" style="display: flex;"><span class="footnote-num" style="display: inline; width: 10%; background: none; font-size: 80%; opacity: 0.6; line-height: 26px; font-family: ptima-Regular, Optima, PingFangSC-light, PingFangTC-light, 'PingFang SC', Cambria, Cochin, Georgia, Times, 'Times New Roman', serif;">[2] </span><p style="padding-top: 8px; padding-bottom: 8px; margin-bottom: 27px; display: inline; font-size: 14px; width: 90%; padding: 0px; margin: 0; line-height: 26px; word-break: break-all; width: calc(100%-50); color: #1d51ce;">DeepFlow Demo: <em style="font-style: italic; color: #888888;">https://deepflow.yunshan.net/docs/zh/install/overview/</em></p> </span> </section> </section>