OpenSearch Arrow Flight RPC 传输层监控指南:Flight Stats API 指标详解与实战解读

原创2026-09-21 21:34:261,997 阅读
文章标签:搜索引擎全文检索可观测性数据分析

OpenSearch Arrow Flight RPC 传输层监控指南:Flight Stats API 指标详解与实战解读

本指南系统讲解 OpenSearch 的 Arrow Flight RPC 插件(arrow-flight-rpc)如何通过 Flight Stats API 暴露节点间流式传输(node-to-node streaming transport)的全量运行指标,覆盖指标访问方式、六大指标分类、集群级聚合统计以及性能/错误/吞吐三个维度的解读方法。读完本文,你将能够熟练使用 /_flight/stats 系列端点定位高延迟、内存压力、线程池饱和与异常状态码,并借助 _cat/tasks 实时监控流式传输任务。

背景:Arrow Flight RPC 插件与流式传输

Arrow Flight RPC 插件为 OpenSearch 提供了基于 Apache Arrow Flight 协议的节点间流式传输(streaming transport)能力,并可与 Security 插件集成实现基于 TLS 的加密认证传输。从插件主页 README.md 可以看到,该能力仅面向集群内部节点间传输,不适用于 REST API 客户端。

在使用监控能力之前,需要先启用特性开关并安装插件:

# opensearch.yml
opensearch.experimental.feature.transport.stream.enabled: true

开发模式运行:

./gradlew run -PinstalledPlugins="['arrow-flight-rpc']"

手动部署时还需在 JVM 选项中添加 Netty 相关系统属性(插件启动时会校验这些值,见 ServerConfig.java 中的 Netty4Configs.init):

-Dio.netty.allocator.numDirectArenas=1
-Dio.netty.noUnsafe=false
-Dio.netty.tryUnsafe=true
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED

访问指标:Flight Stats API

插件通过 REST handler 暴露了四个统计端点(路由定义见 FlightStatsRestHandler.java):

端点 说明
GET /_flight/stats 返回集群中所有节点的 Flight 指标
GET /_flight/stats/{node_id} 返回指定节点的 Flight 指标
GET /_nodes/flight/stats 以 nodes API 风格返回所有节点指标
GET /_nodes/{node_id}/flight/stats 以 nodes API 风格返回指定节点指标

以 /_flight/stats 为例:

GET /_flight/stats

查询单节点:

GET /_flight/stats/{node_id}

该接口还支持 timeout 参数(如 ?timeout=10s),由 FlightStatsRestHandler.java 透传到内部请求。请求会通过 FlightStatsAction 分发给集群中的目标节点,每个节点在 nodeOperation 中调用 FlightStatsCollector.collectStats() 采集本地指标后汇总返回(见 TransportFlightStatsAction.java),该 action 运行在 MANAGEMENT 线程池上,不会抢占搜索线程资源。

监控流式任务:Tasks API

除了统计端点,流式传输任务还可以通过 OpenSearch 既有的 Tasks API 实时观察:

curl "localhost:9200/_cat/tasks?v"

输出示例(流式任务以 stream-transport 类型标识,该类型名定义于 FlightTransportChannel.java):

action                                task_id                     parent_task_id              type             start_time    timestamp running_time ip        node
indices:data/read/search              TVk0SciMQtSwplV6rQwyMA:2165 -                           transport        1754082449785 21:07:29  169.5ms      127.0.0.1 node-1
indices:data/read/search[phase/query] TVk0SciMQtSwplV6rQwyMA:2166 TVk0SciMQtSwplV6rQwyMA:2165 stream-transport 1754082449786 21:07:29  168.4ms      127.0.0.1 node-1

从输出可见,stream-transport 子任务会挂在父任务(如 indices:data/read/search[phase/query])之下,running_time 列可直接反映单个流式子任务的耗时。这对于定位"哪一次具体查询的流式传输阶段变慢"非常直观。

指标结构:六大分类

指标在响应中组织为 flight_metrics 对象,内部结构与源码 FlightMetrics.java 的字段一一对应。以下逐一说明每类指标的含义。

Client Call Metrics(客户端调用指标)

记录客户端侧发起的 Flight 调用:

Metric Description
started Number of client calls started
completed Number of client calls completed
duration Duration statistics for client calls (min, max, avg, sum)
request_bytes Size statistics for requests sent by clients (min, max, avg, sum)
response Total size of responses received by clients (with human-readable format)
response_bytes Total size of responses received by clients (in bytes)

Client Batch Metrics(客户端批次指标)

记录客户端侧流式批次(batch)的接收与处理:

Metric Description
requested Number of batches requested by clients
received Number of batches received by clients
received_bytes Size statistics for batches received by clients (min, max, avg, sum)
processing_time Time statistics for processing received batches (min, max, avg, sum)

Server Call Metrics(服务端调用指标)

记录服务端侧处理的 Flight 调用:

Metric Description
started Number of server calls started
completed Number of server calls completed
duration Duration statistics for server calls (min, max, avg, sum)
request_bytes Size statistics for requests received by servers (min, max, avg, sum)
response Total size of responses sent by servers (with human-readable format)
response_bytes Total size of responses sent by servers (in bytes)

Server Batch Metrics(服务端批次指标)

记录服务端侧流式批次的发送:

Metric Description
sent Number of batches sent by servers
sent_bytes Size statistics for batches sent by servers (min, max, avg, sum)
processing_time Time statistics for processing and sending batches (min, max, avg, sum)

Status Metrics(状态码指标)

按 gRPC/Arrow Flight 状态码统计调用完成情况:

Metric Description
client.{status} Count of client calls completed with each status code (OK, CANCELLED, UNAVAILABLE, etc.)
server.{status} Count of server calls completed with each status code (OK, CANCELLED, UNAVAILABLE, etc.)

状态码直接取自 Flight 调用结束时的 status 字符串(见 FlightCallTracker.java 的 recordCallEnd(String status)),因此除了 OK 之外,还可能看到 CANCELLED、UNAVAILABLE、RESOURCE_EXHAUSTED、DEADLINE_EXCEEDED 等标准状态码。

Resource Metrics(资源指标)

资源指标是采集时刻的点快照(point-in-time snapshot),并非累加计数器:

Metric Description
arrow_allocated Current Arrow memory allocation (human-readable format)
arrow_allocated_bytes Current Arrow memory allocation in bytes
arrow_peak Peak Arrow memory allocation (human-readable format)
arrow_peak_bytes Peak Arrow memory allocation in bytes
direct_memory Current direct memory usage (human-readable format)
direct_memory_bytes Current direct memory usage in bytes
client_threads_active Number of active client threads
client_threads_total Total number of client threads
server_threads_active Number of active server threads
server_threads_total Total number of server threads
client_channels_active Number of active client channels
server_channels_active Number of active server channels

此外,当 client_threads_total / server_threads_total 大于 0 时,响应还会额外输出 client_thread_utilization_percent 与 server_thread_utilization_percent 两个百分比字段(见 FlightMetrics.java 的 toXContent)。

资源指标的采集逻辑位于 FlightStatsCollector.java:

  • Arrow 内存取自 Arrow BufferAllocator 的 getAllocatedMemory() 与 getPeakMemoryAllocation();
  • direct memory 取自 JVM MemoryMXBean 的 Non-Heap 使用量(异常时回退为 totalMemory() - freeMemory());
  • 线程指标按线程池名称精确匹配 flight-client 与 flight-server(常量定义于 ServerConfig.java);
  • 通道数由 incrementClientChannelsActive/decrementServerChannelsActive 等原子计数器动态维护。

指标的数据形态与底层实现

所有 duration 与 size 字段都同时输出人可读格式(如 "1s"、"24.6kb")与原始数值(纳秒 / 字节)。以时长直方图为例:

"duration": {
  "count": 6,
  "sum": "1s",
  "sum_nanos": 1019,
  "min": "9ms",
  "min_nanos": 9,
  "max": "743.7ms",
  "max_nanos": 743,
  "avg": "169.8ms",
  "avg_nanos": 169
}

字节直方图同理,输出 sum_bytes、min_bytes、max_bytes、avg_bytes 等字段(格式化逻辑见 FlightMetrics.java 中的 addDurationHistogramToXContent 与 addBytesHistogramToXContent)。

从源码结构看,指标系统的设计参考了 gRPC 的 OpenTelemetry 指标模型(类注释明确注明 "inspired by gRPC's OpenTelemetry metrics"),核心构件包括:

  • 计数器:使用 JDK LongAdder(clientCallStarted、serverCallCompleted 等),保证高并发写入下的线性伸缩性;
  • 直方图:内部维护 count、sum、min、max 四个原子字段,avg 由 sum / count 计算得出(见 FlightMetrics.Histogram 内部类);
  • 按状态码计数:ConcurrentHashMap<String, LongAdder> 按状态码字符串分桶累加;
  • 单次调用追踪:每次调用通过 FlightCallTracker 记录起始时间戳,结束时以 System.nanoTime() 计算耗时并上报状态码。客户端与服务端的追踪器分别在 FlightClientChannel.java 与 ArrowFlightProducer.java 中创建,响应阶段则由 MetricsTrackingResponseHandler 包装完成数据点上报。

Cluster-Level Metrics:集群级聚合

GET /_flight/stats 的响应除了 nodes 中各节点的独立指标外,还会包含一个 cluster_stats 段落,由 FlightStatsResponse.java 的 aggregateClusterStats 方法将集群内所有节点的指标求和/求均值后得出:

  • Client calls and batches(跨所有节点聚合)
  • Server calls and batches(跨所有节点聚合)
  • Average durations and throughput

聚合结构示意(client 与 server 两个分组):

"cluster_stats": {
  "client": {
    "calls": {
      "started": 6,
      "completed": 6,
      "duration": "1s",
      "duration_nanos": 1019,
      "avg_duration": "169.8ms",
      "avg_duration_nanos": 169,
      "request": "5.9kb",
      "request_bytes": 6132,
      "response": "24.6kb",
      "response_bytes": 25276
    },
    "batches": {
      "requested": 6,
      "received": 6,
      "received_size": "24.6kb",
      "received_bytes": 25276,
      "avg_processing_time": "2ms",
      "avg_processing_time_nanos": 2
    }
  },
  "server": {
    "calls": {
      "started": 6,
      "completed": 6,
      "duration": "556ms",
      "duration_nanos": 556,
      "avg_duration": "92.6ms",
      "avg_duration_nanos": 92,
      "request": "5.9kb",
      "request_bytes": 6132,
      "response": "24.6kb",
      "response_bytes": 25276
    },
    "batches": {
      "sent": 6,
      "sent_size": "24.6kb",
      "sent_bytes": 25276,
      "avg_processing_time": "34.6ms",
      "avg_processing_time_nanos": 34
    }
  }
}

示例响应:单节点指标全貌

一次完整调用 GET /_flight/stats 的响应示例如下(flight_metrics 节点级部分):

{
  "cluster_name": "opensearch",
  "nodes": {
    "node_id": {
      "name": "node_name",
      "streamAddress": "localhost:9400",
      "flight_metrics": {
        "client_calls": {
          "started": 6,
          "completed": 6,
          "duration": {
            "count": 6,
            "sum": "1s",
            "sum_nanos": 1019,
            "min": "9ms",
            "min_nanos": 9,
            "max": "743.7ms",
            "max_nanos": 743,
            "avg": "169.8ms",
            "avg_nanos": 169
          },
          "request_bytes": {
            "count": 6,
            "sum": "5.9kb",
            "sum_bytes": 6132,
            "min": "1022b",
            "min_bytes": 1022,
            "max": "1022b",
            "max_bytes": 1022,
            "avg": "1022b",
            "avg_bytes": 1022
          },
          "response": "24.6kb",
          "response_bytes": 25276
        },
        "client_batches": {
          "requested": 6,
          "received": 6,
          "received_bytes": {
            "count": 6,
            "sum": "24.6kb",
            "sum_bytes": 25276,
            "min": "3.3kb",
            "min_bytes": 3477,
            "max": "4.2kb",
            "max_bytes": 4361,
            "avg": "4.1kb",
            "avg_bytes": 4212
          },
          "processing_time": {
            "count": 6,
            "sum": "12.1ms",
            "sum_nanos": 12,
            "min": "352micros",
            "min_nanos": 0,
            "max": "9.5ms",
            "max_nanos": 9,
            "avg": "2ms",
            "avg_nanos": 2
          }
        },
        "server_calls": {
          "started": 3,
          "completed": 3,
          "duration": {
            "count": 3,
            "sum": "147.9ms",
            "sum_nanos": 147,
            "min": "6ms",
            "min_nanos": 6,
            "max": "135.7ms",
            "max_nanos": 135,
            "avg": "49.3ms",
            "avg_nanos": 49
          },
          "request_bytes": {
            "count": 3,
            "sum": "2.9kb",
            "sum_bytes": 3066,
            "min": "1022b",
            "min_bytes": 1022,
            "max": "1022b",
            "max_bytes": 1022,
            "avg": "1022b",
            "avg_bytes": 1022
          },
          "response": "12.7kb",
          "response_bytes": 13083
        },
        "server_batches": {
          "sent": 3,
          "sent_bytes": {
            "count": 3,
            "sum": "12.7kb",
            "sum_bytes": 13083,
            "min": "4.2kb",
            "min_bytes": 4361,
            "max": "4.2kb",
            "max_bytes": 4361,
            "avg": "4.2kb",
            "avg_bytes": 4361
          },
          "processing_time": {
            "count": 3,
            "sum": "6.4ms",
            "sum_nanos": 6,
            "min": "525.4micros",
            "min_nanos": 0,
            "max": "5.3ms",
            "max_nanos": 5,
            "avg": "2.1ms",
            "avg_nanos": 2
          }
        },
        "status": {
          "client": {
            "OK": 6
          },
          "server": {
            "OK": 3
          }
        },
        "resources": {
          "arrow_allocated": "0b",
          "arrow_allocated_bytes": 0,
          "arrow_peak": "48kb",
          "arrow_peak_bytes": 49152,
          "direct_memory": "120.7mb",
          "direct_memory_bytes": 126642920,
          "client_threads_active": 0,
          "client_threads_total": 0,
          "server_threads_active": 0,
          "server_threads_total": 0,
          "client_channels_active": 2,
          "server_channels_active": 1
        }
      }
    }
  },
  "cluster_stats": {
    "client": {
      "calls": {
        "started": 6,
        "completed": 6,
        "duration": "1s",
        "duration_nanos": 1019,
        "avg_duration": "169.8ms",
        "avg_duration_nanos": 169,
        "request": "5.9kb",
        "request_bytes": 6132,
        "response": "24.6kb",
        "response_bytes": 25276
      },
      "batches": {
        "requested": 6,
        "received": 6,
        "received_size": "24.6kb",
        "received_bytes": 25276,
        "avg_processing_time": "2ms",
        "avg_processing_time_nanos": 2
      }
    },
    "server": {
      "calls": {
        "started": 6,
        "completed": 6,
        "duration": "556ms",
        "duration_nanos": 556,
        "avg_duration": "92.6ms",
        "avg_duration_nanos": 92,
        "request": "5.9kb",
        "request_bytes": 6132,
        "response": "24.6kb",
        "response_bytes": 25276
      },
      "batches": {
        "sent": 6,
        "sent_size": "24.6kb",
        "sent_bytes": 25276,
        "avg_processing_time": "34.6ms",
        "avg_processing_time_nanos": 34
      }
    }
  }
}

值得注意的细节:response/response_bytes 与 client_batches.received_bytes 的 sum 一致,因为客户端收到的响应总字节数正是各批次的字节和(见 FlightMetrics.java 中 response_bytes 取自 clientBatchReceivedBytes 的快照和);同理服务端 response_bytes 取自 serverBatchSentBytes 的快照和。理解这一点有助于跨字段交叉校验数据一致性。

指标解读:从数据到诊断

Performance Monitoring(性能监控)

  • 高延迟(High latency):关注 client/server 的 duration 指标。max 反映极端情况,avg 反映整体水平,count 与 sum 结合可估算总体负载。若 max 与 avg 差距过大,说明存在明显的长尾请求。
  • 内存压力(Memory pressure):监控 arrow_allocated_bytes 与 arrow_peak_bytes。arrow_peak_bytes 代表历史峰值,可用于评估缓冲池容量上限是否需要调整;direct_memory 反映 Netty 与 Arrow 共用的堆外内存水位,持续逼近 JVM 堆外内存上限时需警惕 OutOfMemoryError。
  • 线程池饱和(Thread pool saturation):查看 client_thread_utilization_percent 与 server_thread_utilization_percent。接近 100% 说明 flight-client / flight-server 线程池已满负荷,对应线程池的最小/最大线程数与 keep-alive 可通过 thread_pool.flight-server.min、thread_pool.flight-server.max、thread_pool.flight-server.keep_alive 等设置调整(见 ServerConfig.java)。

Error Detection(错误检测)

  • 调用失败(Failed calls):监控 status.client 与 status.server 中非 OK 状态码的计数。任何非零的非 OK 计数都对应一次未正常完成的调用。
  • 取消操作(Cancelled operations):关注 CANCELLED 状态码计数。客户端主动取消(如超时或用户中断)与服务端拒绝都可能产生该状态码。
  • 资源耗尽(Resource exhaustion):警惕 RESOURCE_EXHAUSTED 状态码计数,这通常与背压触发、分配器内存耗尽或并发流超限相关,可配合 backpressure.md 文档理解产生背压的机制。

Throughput Analysis(吞吐分析)

  • 请求吞吐(Request throughput):观察 client_calls.started 与 server_calls.started 的增速(rate)。两次采样间 started 的差值除以时间间隔即为吞吐速率。
  • 数据吞吐(Data throughput):跟踪 client_calls.request_bytes 与 server_batches.sent_bytes 的增速。这些直方图的 sum_bytes 字段最适合做字节级吞吐统计。
  • 批次效率(Batch efficiency):对比 client_batches.received 与 client_batches.requested。两者长期不一致说明存在批次请求被中断或未完成的情况;同时对比 received_bytes 的平均批次大小(avg_bytes)可以评估流式传输的批大小配置是否合理。

参考资料与深入阅读

登录后查看全文
OpenSearch