OpenSearch Arrow Flight RPC 传输层监控指南:Flight Stats API 指标详解与实战解读
OpenSearch Arrow Flight RPC 传输层监控指南:Flight Stats API 指标详解与实战解读
本指南系统讲解 OpenSearch 的 Arrow Flight RPC 插件(arrow-flight-rpc)如何通过 Flight Stats API 暴露节点间流式传输(node-to-node streaming transport)的全量运行指标,覆盖指标访问方式、六大指标分类、集群级聚合统计以及性能/错误/吞吐三个维度的解读方法。读完本文,你将能够熟练使用 /_flight/stats 系列端点定位高延迟、内存压力、线程池饱和与异常状态码,并借助 _cat/tasks 实时监控流式传输任务。
背景:Arrow Flight RPC 插件与流式传输
Arrow Flight RPC 插件为 OpenSearch 提供了基于 Apache Arrow Flight 协议的节点间流式传输(streaming transport)能力,并可与 Security 插件集成实现基于 TLS 的加密认证传输。从插件主页 README.md 可以看到,该能力仅面向集群内部节点间传输,不适用于 REST API 客户端。
在使用监控能力之前,需要先启用特性开关并安装插件:
# opensearch.yml
opensearch.experimental.feature.transport.stream.enabled: true
开发模式运行:
./gradlew run -PinstalledPlugins="['arrow-flight-rpc']"
手动部署时还需在 JVM 选项中添加 Netty 相关系统属性(插件启动时会校验这些值,见 ServerConfig.java 中的 Netty4Configs.init):
-Dio.netty.allocator.numDirectArenas=1
-Dio.netty.noUnsafe=false
-Dio.netty.tryUnsafe=true
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
访问指标:Flight Stats API
插件通过 REST handler 暴露了四个统计端点(路由定义见 FlightStatsRestHandler.java):
| 端点 | 说明 |
|---|---|
GET /_flight/stats |
返回集群中所有节点的 Flight 指标 |
GET /_flight/stats/{node_id} |
返回指定节点的 Flight 指标 |
GET /_nodes/flight/stats |
以 nodes API 风格返回所有节点指标 |
GET /_nodes/{node_id}/flight/stats |
以 nodes API 风格返回指定节点指标 |
以 /_flight/stats 为例:
GET /_flight/stats
查询单节点:
GET /_flight/stats/{node_id}
该接口还支持 timeout 参数(如 ?timeout=10s),由 FlightStatsRestHandler.java 透传到内部请求。请求会通过 FlightStatsAction 分发给集群中的目标节点,每个节点在 nodeOperation 中调用 FlightStatsCollector.collectStats() 采集本地指标后汇总返回(见 TransportFlightStatsAction.java),该 action 运行在 MANAGEMENT 线程池上,不会抢占搜索线程资源。
监控流式任务:Tasks API
除了统计端点,流式传输任务还可以通过 OpenSearch 既有的 Tasks API 实时观察:
curl "localhost:9200/_cat/tasks?v"
输出示例(流式任务以 stream-transport 类型标识,该类型名定义于 FlightTransportChannel.java):
action task_id parent_task_id type start_time timestamp running_time ip node
indices:data/read/search TVk0SciMQtSwplV6rQwyMA:2165 - transport 1754082449785 21:07:29 169.5ms 127.0.0.1 node-1
indices:data/read/search[phase/query] TVk0SciMQtSwplV6rQwyMA:2166 TVk0SciMQtSwplV6rQwyMA:2165 stream-transport 1754082449786 21:07:29 168.4ms 127.0.0.1 node-1
从输出可见,stream-transport 子任务会挂在父任务(如 indices:data/read/search[phase/query])之下,running_time 列可直接反映单个流式子任务的耗时。这对于定位"哪一次具体查询的流式传输阶段变慢"非常直观。
指标结构:六大分类
指标在响应中组织为 flight_metrics 对象,内部结构与源码 FlightMetrics.java 的字段一一对应。以下逐一说明每类指标的含义。
Client Call Metrics(客户端调用指标)
记录客户端侧发起的 Flight 调用:
| Metric | Description |
|---|---|
started |
Number of client calls started |
completed |
Number of client calls completed |
duration |
Duration statistics for client calls (min, max, avg, sum) |
request_bytes |
Size statistics for requests sent by clients (min, max, avg, sum) |
response |
Total size of responses received by clients (with human-readable format) |
response_bytes |
Total size of responses received by clients (in bytes) |
Client Batch Metrics(客户端批次指标)
记录客户端侧流式批次(batch)的接收与处理:
| Metric | Description |
|---|---|
requested |
Number of batches requested by clients |
received |
Number of batches received by clients |
received_bytes |
Size statistics for batches received by clients (min, max, avg, sum) |
processing_time |
Time statistics for processing received batches (min, max, avg, sum) |
Server Call Metrics(服务端调用指标)
记录服务端侧处理的 Flight 调用:
| Metric | Description |
|---|---|
started |
Number of server calls started |
completed |
Number of server calls completed |
duration |
Duration statistics for server calls (min, max, avg, sum) |
request_bytes |
Size statistics for requests received by servers (min, max, avg, sum) |
response |
Total size of responses sent by servers (with human-readable format) |
response_bytes |
Total size of responses sent by servers (in bytes) |
Server Batch Metrics(服务端批次指标)
记录服务端侧流式批次的发送:
| Metric | Description |
|---|---|
sent |
Number of batches sent by servers |
sent_bytes |
Size statistics for batches sent by servers (min, max, avg, sum) |
processing_time |
Time statistics for processing and sending batches (min, max, avg, sum) |
Status Metrics(状态码指标)
按 gRPC/Arrow Flight 状态码统计调用完成情况:
| Metric | Description |
|---|---|
client.{status} |
Count of client calls completed with each status code (OK, CANCELLED, UNAVAILABLE, etc.) |
server.{status} |
Count of server calls completed with each status code (OK, CANCELLED, UNAVAILABLE, etc.) |
状态码直接取自 Flight 调用结束时的 status 字符串(见 FlightCallTracker.java 的 recordCallEnd(String status)),因此除了 OK 之外,还可能看到 CANCELLED、UNAVAILABLE、RESOURCE_EXHAUSTED、DEADLINE_EXCEEDED 等标准状态码。
Resource Metrics(资源指标)
资源指标是采集时刻的点快照(point-in-time snapshot),并非累加计数器:
| Metric | Description |
|---|---|
arrow_allocated |
Current Arrow memory allocation (human-readable format) |
arrow_allocated_bytes |
Current Arrow memory allocation in bytes |
arrow_peak |
Peak Arrow memory allocation (human-readable format) |
arrow_peak_bytes |
Peak Arrow memory allocation in bytes |
direct_memory |
Current direct memory usage (human-readable format) |
direct_memory_bytes |
Current direct memory usage in bytes |
client_threads_active |
Number of active client threads |
client_threads_total |
Total number of client threads |
server_threads_active |
Number of active server threads |
server_threads_total |
Total number of server threads |
client_channels_active |
Number of active client channels |
server_channels_active |
Number of active server channels |
此外,当 client_threads_total / server_threads_total 大于 0 时,响应还会额外输出 client_thread_utilization_percent 与 server_thread_utilization_percent 两个百分比字段(见 FlightMetrics.java 的 toXContent)。
资源指标的采集逻辑位于 FlightStatsCollector.java:
- Arrow 内存取自 Arrow
BufferAllocator的getAllocatedMemory()与getPeakMemoryAllocation(); - direct memory 取自 JVM
MemoryMXBean的 Non-Heap 使用量(异常时回退为totalMemory() - freeMemory()); - 线程指标按线程池名称精确匹配
flight-client与flight-server(常量定义于 ServerConfig.java); - 通道数由
incrementClientChannelsActive/decrementServerChannelsActive等原子计数器动态维护。
指标的数据形态与底层实现
所有 duration 与 size 字段都同时输出人可读格式(如 "1s"、"24.6kb")与原始数值(纳秒 / 字节)。以时长直方图为例:
"duration": {
"count": 6,
"sum": "1s",
"sum_nanos": 1019,
"min": "9ms",
"min_nanos": 9,
"max": "743.7ms",
"max_nanos": 743,
"avg": "169.8ms",
"avg_nanos": 169
}
字节直方图同理,输出 sum_bytes、min_bytes、max_bytes、avg_bytes 等字段(格式化逻辑见 FlightMetrics.java 中的 addDurationHistogramToXContent 与 addBytesHistogramToXContent)。
从源码结构看,指标系统的设计参考了 gRPC 的 OpenTelemetry 指标模型(类注释明确注明 "inspired by gRPC's OpenTelemetry metrics"),核心构件包括:
- 计数器:使用 JDK
LongAdder(clientCallStarted、serverCallCompleted等),保证高并发写入下的线性伸缩性; - 直方图:内部维护
count、sum、min、max四个原子字段,avg由sum / count计算得出(见FlightMetrics.Histogram内部类); - 按状态码计数:
ConcurrentHashMap<String, LongAdder>按状态码字符串分桶累加; - 单次调用追踪:每次调用通过
FlightCallTracker记录起始时间戳,结束时以System.nanoTime()计算耗时并上报状态码。客户端与服务端的追踪器分别在 FlightClientChannel.java 与 ArrowFlightProducer.java 中创建,响应阶段则由MetricsTrackingResponseHandler包装完成数据点上报。
Cluster-Level Metrics:集群级聚合
GET /_flight/stats 的响应除了 nodes 中各节点的独立指标外,还会包含一个 cluster_stats 段落,由 FlightStatsResponse.java 的 aggregateClusterStats 方法将集群内所有节点的指标求和/求均值后得出:
- Client calls and batches(跨所有节点聚合)
- Server calls and batches(跨所有节点聚合)
- Average durations and throughput
聚合结构示意(client 与 server 两个分组):
"cluster_stats": {
"client": {
"calls": {
"started": 6,
"completed": 6,
"duration": "1s",
"duration_nanos": 1019,
"avg_duration": "169.8ms",
"avg_duration_nanos": 169,
"request": "5.9kb",
"request_bytes": 6132,
"response": "24.6kb",
"response_bytes": 25276
},
"batches": {
"requested": 6,
"received": 6,
"received_size": "24.6kb",
"received_bytes": 25276,
"avg_processing_time": "2ms",
"avg_processing_time_nanos": 2
}
},
"server": {
"calls": {
"started": 6,
"completed": 6,
"duration": "556ms",
"duration_nanos": 556,
"avg_duration": "92.6ms",
"avg_duration_nanos": 92,
"request": "5.9kb",
"request_bytes": 6132,
"response": "24.6kb",
"response_bytes": 25276
},
"batches": {
"sent": 6,
"sent_size": "24.6kb",
"sent_bytes": 25276,
"avg_processing_time": "34.6ms",
"avg_processing_time_nanos": 34
}
}
}
示例响应:单节点指标全貌
一次完整调用 GET /_flight/stats 的响应示例如下(flight_metrics 节点级部分):
{
"cluster_name": "opensearch",
"nodes": {
"node_id": {
"name": "node_name",
"streamAddress": "localhost:9400",
"flight_metrics": {
"client_calls": {
"started": 6,
"completed": 6,
"duration": {
"count": 6,
"sum": "1s",
"sum_nanos": 1019,
"min": "9ms",
"min_nanos": 9,
"max": "743.7ms",
"max_nanos": 743,
"avg": "169.8ms",
"avg_nanos": 169
},
"request_bytes": {
"count": 6,
"sum": "5.9kb",
"sum_bytes": 6132,
"min": "1022b",
"min_bytes": 1022,
"max": "1022b",
"max_bytes": 1022,
"avg": "1022b",
"avg_bytes": 1022
},
"response": "24.6kb",
"response_bytes": 25276
},
"client_batches": {
"requested": 6,
"received": 6,
"received_bytes": {
"count": 6,
"sum": "24.6kb",
"sum_bytes": 25276,
"min": "3.3kb",
"min_bytes": 3477,
"max": "4.2kb",
"max_bytes": 4361,
"avg": "4.1kb",
"avg_bytes": 4212
},
"processing_time": {
"count": 6,
"sum": "12.1ms",
"sum_nanos": 12,
"min": "352micros",
"min_nanos": 0,
"max": "9.5ms",
"max_nanos": 9,
"avg": "2ms",
"avg_nanos": 2
}
},
"server_calls": {
"started": 3,
"completed": 3,
"duration": {
"count": 3,
"sum": "147.9ms",
"sum_nanos": 147,
"min": "6ms",
"min_nanos": 6,
"max": "135.7ms",
"max_nanos": 135,
"avg": "49.3ms",
"avg_nanos": 49
},
"request_bytes": {
"count": 3,
"sum": "2.9kb",
"sum_bytes": 3066,
"min": "1022b",
"min_bytes": 1022,
"max": "1022b",
"max_bytes": 1022,
"avg": "1022b",
"avg_bytes": 1022
},
"response": "12.7kb",
"response_bytes": 13083
},
"server_batches": {
"sent": 3,
"sent_bytes": {
"count": 3,
"sum": "12.7kb",
"sum_bytes": 13083,
"min": "4.2kb",
"min_bytes": 4361,
"max": "4.2kb",
"max_bytes": 4361,
"avg": "4.2kb",
"avg_bytes": 4361
},
"processing_time": {
"count": 3,
"sum": "6.4ms",
"sum_nanos": 6,
"min": "525.4micros",
"min_nanos": 0,
"max": "5.3ms",
"max_nanos": 5,
"avg": "2.1ms",
"avg_nanos": 2
}
},
"status": {
"client": {
"OK": 6
},
"server": {
"OK": 3
}
},
"resources": {
"arrow_allocated": "0b",
"arrow_allocated_bytes": 0,
"arrow_peak": "48kb",
"arrow_peak_bytes": 49152,
"direct_memory": "120.7mb",
"direct_memory_bytes": 126642920,
"client_threads_active": 0,
"client_threads_total": 0,
"server_threads_active": 0,
"server_threads_total": 0,
"client_channels_active": 2,
"server_channels_active": 1
}
}
}
},
"cluster_stats": {
"client": {
"calls": {
"started": 6,
"completed": 6,
"duration": "1s",
"duration_nanos": 1019,
"avg_duration": "169.8ms",
"avg_duration_nanos": 169,
"request": "5.9kb",
"request_bytes": 6132,
"response": "24.6kb",
"response_bytes": 25276
},
"batches": {
"requested": 6,
"received": 6,
"received_size": "24.6kb",
"received_bytes": 25276,
"avg_processing_time": "2ms",
"avg_processing_time_nanos": 2
}
},
"server": {
"calls": {
"started": 6,
"completed": 6,
"duration": "556ms",
"duration_nanos": 556,
"avg_duration": "92.6ms",
"avg_duration_nanos": 92,
"request": "5.9kb",
"request_bytes": 6132,
"response": "24.6kb",
"response_bytes": 25276
},
"batches": {
"sent": 6,
"sent_size": "24.6kb",
"sent_bytes": 25276,
"avg_processing_time": "34.6ms",
"avg_processing_time_nanos": 34
}
}
}
}
值得注意的细节:response/response_bytes 与 client_batches.received_bytes 的 sum 一致,因为客户端收到的响应总字节数正是各批次的字节和(见 FlightMetrics.java 中 response_bytes 取自 clientBatchReceivedBytes 的快照和);同理服务端 response_bytes 取自 serverBatchSentBytes 的快照和。理解这一点有助于跨字段交叉校验数据一致性。
指标解读:从数据到诊断
Performance Monitoring(性能监控)
- 高延迟(High latency):关注 client/server 的
duration指标。max反映极端情况,avg反映整体水平,count与sum结合可估算总体负载。若max与avg差距过大,说明存在明显的长尾请求。 - 内存压力(Memory pressure):监控
arrow_allocated_bytes与arrow_peak_bytes。arrow_peak_bytes代表历史峰值,可用于评估缓冲池容量上限是否需要调整;direct_memory反映 Netty 与 Arrow 共用的堆外内存水位,持续逼近 JVM 堆外内存上限时需警惕OutOfMemoryError。 - 线程池饱和(Thread pool saturation):查看
client_thread_utilization_percent与server_thread_utilization_percent。接近 100% 说明flight-client/flight-server线程池已满负荷,对应线程池的最小/最大线程数与 keep-alive 可通过thread_pool.flight-server.min、thread_pool.flight-server.max、thread_pool.flight-server.keep_alive等设置调整(见 ServerConfig.java)。
Error Detection(错误检测)
- 调用失败(Failed calls):监控
status.client与status.server中非 OK 状态码的计数。任何非零的非 OK 计数都对应一次未正常完成的调用。 - 取消操作(Cancelled operations):关注
CANCELLED状态码计数。客户端主动取消(如超时或用户中断)与服务端拒绝都可能产生该状态码。 - 资源耗尽(Resource exhaustion):警惕
RESOURCE_EXHAUSTED状态码计数,这通常与背压触发、分配器内存耗尽或并发流超限相关,可配合 backpressure.md 文档理解产生背压的机制。
Throughput Analysis(吞吐分析)
- 请求吞吐(Request throughput):观察
client_calls.started与server_calls.started的增速(rate)。两次采样间started的差值除以时间间隔即为吞吐速率。 - 数据吞吐(Data throughput):跟踪
client_calls.request_bytes与server_batches.sent_bytes的增速。这些直方图的sum_bytes字段最适合做字节级吞吐统计。 - 批次效率(Batch efficiency):对比
client_batches.received与client_batches.requested。两者长期不一致说明存在批次请求被中断或未完成的情况;同时对比received_bytes的平均批次大小(avg_bytes)可以评估流式传输的批大小配置是否合理。
参考资料与深入阅读
- 本文核心依据:plugins/arrow-flight-rpc/docs/metrics.md
- 插件总览与安装步骤:plugins/arrow-flight-rpc/README.md
- 指标采集与直方图实现:FlightMetrics.java
- REST 端点定义:FlightStatsRestHandler.java
- 资源指标采集:FlightStatsCollector.java
- 单次调用追踪:FlightCallTracker.java
- 集群聚合统计:FlightStatsResponse.java
- 传输层 action 分发:TransportFlightStatsAction.java
- 线程池/端口/TLS 等配置项:ServerConfig.java
- 流式任务类型标识:FlightTransportChannel.java
- 配套机制文档:架构总览、背压机制、错误处理、安全集成