Seldon Core 2 收入分类器生产级部署实战:漂移检测、异常检测与 Anchor 可解释 Pipeline

原创2026-10-14 10:45:161,818 阅读
文章标签:人工智能大模型MLOpsLLMOps模型推理服务云原生后端

Seldon Core 2 收入分类器生产级部署实战:漂移检测、异常检测与 Anchor 可解释 Pipeline

本文基于 Seldon Core 2 的表格型收入(Adult Census Income)分类器示例,完整演示如何将预处理、分类、漂移检测、异常检测与 Anchor 可解释模型编排为一条生产级推理 Pipeline,并通过 V2 推理协议与 seldon CLI 验证其监控输出。读完本文,你将掌握 Seldon Core 2 中 Model 资源定义、Pipeline 多步编排(含异步监控分支)、漂移/异常检测结果的读取方法,以及黑盒模型的可解释部署流程。

一、示例整体架构与数据准备

该示例围绕 Adult Census Income 数据集构建了一组配套模型:原始特征经 income-preprocess 预处理后,交由 income 分类器预测收入等级;同时,income-outlier(基于 OutlierVAE 的重构误差)与 income-drift(基于 TabularDrift 的统计漂移检测)分别在主路径的侧翼运行,形成对线上数据质量的持续监控。完整训练流程见仓库脚本 samples/examples/income_classifier/train.py,其中使用了 Alibi-Detect 的 TabularDrift 与 OutlierVAE、Alibi-Explain 的 AnchorTabular,并基于 sklearn 的 ColumnTransformer + RandomForestClassifier 训练分类器与独立保存的 preprocessor。

运行推理 Notebook 需要先准备推理数据,两种方式任选其一:

  • 运行训练脚本产出数据:make train;
  • 或直接从模型仓库下载推理数据:
    gsutil cp -R gs://seldon-models/scv2/examples/income/infer-data .
    

数据文件 infer-data/test.npy 中按顺序保存了四份数据:参考集 x_ref(正常数据)、漂移数据 x_h1(低教育程度样本)、参考标签 y_ref、异常数据 x_outlier(注入离群特征后的样本)。这与训练脚本末尾的保存逻辑一致(见 samples/examples/income_classifier/train.py)。

import numpy as np
import json
import requests

with open('./infer-data/test.npy', 'rb') as f:
    x_ref = np.load(f)
    x_h1 = np.load(f)
    y_ref = np.load(f)
    x_outlier = np.load(f)

二、定义四个监控与分类模型

示例将漂移检测模型、异常检测模型均视作普通 Model 资源部署,关键区别在于 requirements 字段中声明 mlserver 与 alibi-detect 运行时依赖(官方说明见 docs-gb/drift.md 与 docs-gb/outlier.md)。四个模型的定义如下(对应仓库文件 samples/models/income-preprocess.yaml、samples/models/income.yaml、samples/models/income-drift.yaml、samples/models/income-outlier.yaml):

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: income-preprocess
spec:
  storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/preprocessor"
  requirements:
  - sklearn
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: income
spec:
  storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/classifier"
  requirements:
  - sklearn
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: income-drift
spec:
  storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/drift-detector"
  requirements:
    - mlserver
    - alibi-detect
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: income-outlier
spec:
  storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/outlier-detector"
  requirements:
    - mlserver
    - alibi-detect

部署方式一:kubectl

在 Kubernetes 集群中(${NAMESPACE} 为部署命名空间)逐个应用并等待就绪:

kubectl apply -f samples/models/income-preprocess.yaml -n ${NAMESPACE}
kubectl apply -f samples/models/income.yaml -n ${NAMESPACE}
kubectl apply -f samples/models/income-drift.yaml -n ${NAMESPACE}
kubectl apply -f samples/models/income-outlier.yaml -n ${NAMESPACE}

应用成功后依次输出 model.mlops.seldon.io/income-preprocess created 等确认信息。随后使用 kubectl wait 等待 ready 条件:

kubectl wait --for condition=ready --timeout=300s model income-preprocess -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model income -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model income-drift -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model income-outlier -n ${NAMESPACE}

部署方式二:seldon CLI

Seldon Core 2 的 seldon CLI 支持从 YAML 直接加载模型(-f 指定文件),底层通过 Scheduler API 完成注册:

seldon model load -f samples/models/income-preprocess.yaml
seldon model load -f samples/models/income.yaml
seldon model load -f samples/models/income-drift.yaml
seldon model load -f samples/models/income-outlier.yaml

加载成功后各返回 {}。随后用 -w ModelAvailable 条件等待模型可用(seldon model status 的 -w/--wait 参数指定等待条件,详见 docs-gb/cli/seldon_model_status.md):

seldon model status income-preprocess -w ModelAvailable | jq .
seldon model status income -w ModelAvailable | jq .
seldon model status income-drift -w ModelAvailable | jq .
seldon model status income-outlier -w ModelAvailable | jq .

三、编排 income-production Pipeline

Pipeline 将模型连接成数据流图,支持模型链、tensor 映射、条件逻辑以及监控侧翼(参见 docs-gb/pipelines.md)。本示例的 Pipeline 定义见 samples/pipelines/income.yaml:

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: income-production
spec:
  steps:
    - name: income
    - name: income-preprocess
    - name: income-outlier
      inputs:
      - income-preprocess
    - name: income-drift
      batch:
        size: 20
  output:
    steps:
    - income
    - income-outlier.outputs.is_outlier

各字段语义:

  • steps:Pipeline 中的模型步骤,步骤名与已部署模型同名(模型可先于或与 Pipeline 同时部署);
  • inputs:步骤的数据来源,income-outlier 显式声明输入为 income-preprocess 的输出,而 income 与 income-drift 隐式接收 Pipeline 的原始输入;
  • batch.size: 20:income-drift 以每 20 条请求批量聚合后再执行漂移检测,属于异步监控路径,不参与同步推理输出的组合;
  • output.steps:声明 Pipeline 的同步输出,包含分类结果 income 与异常检测标志 income-outlier.outputs.is_outlier,因此该 Pipeline 可被 REST/gRPC 同步调用。

加载 Pipeline

kubectl 方式:

kubectl apply -f samples/pipelines/income.yaml -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s pipelines income -n ${NAMESPACE}

CLI 方式(seldon pipeline load 加载,seldon pipeline status -w PipelineReady 等待就绪):

seldon pipeline load -f samples/pipelines/income.yaml
seldon pipeline status income-production -w PipelineReady | jq -M .

就绪后返回的状态结构包含 versions[].pipeline.steps(步骤列表及 batch、inputs 明细)、output.steps 以及 state.status: "PipelineReady"、modelsReady: true 等字段,其中步骤顺序由 Scheduler 拓扑排序决定(例如 income-drift 的 batch 与 income-outlier 的 inputs 均被完整回显):

{
  "pipelineName": "income-production",
  "versions": [
    {
      "pipeline": {
        "name": "income-production",
        "uid": "cifej8iufmbc73e5int0",
        "version": 1,
        "steps": [
          { "name": "income" },
          { "name": "income-drift", "batch": { "size": 20 } },
          { "name": "income-outlier", "inputs": ["income-preprocess.outputs"] },
          { "name": "income-preprocess" }
        ],
        "output": {
          "steps": ["income.outputs", "income-outlier.outputs.is_outlier"]
        },
        "kubernetesMeta": {}
      },
      "state": {
        "pipelineVersion": 1,
        "status": "PipelineReady",
        "reason": "created pipeline",
        "lastChangeTimestamp": "2023-06-30T14:41:38.343754921Z",
        "modelsReady": true
      }
    }
  ]
}

四、通过 V2 推理协议发起预测

Seldon Core 2 的推理接口遵循 V2 协议,请求体需携带命名张量(name、data、datatype、shape),并通过 HTTP 头 seldon-model 指定目标资源(模型名或 Pipeline名.pipeline)。示例中的辅助函数支持三种请求类型:normal(参考集)、drift(漂移集)、outlier(异常集):

reqJson = json.loads('{"inputs":[{"name":"input_1","data":[],"datatype":"FP32","shape":[]}]}')
url = "http://0.0.0.0:9000/v2/models/model/infer"

def infer(resourceName: str, batchSz: int, requestType: str):
    if requestType == "outlier":
        rows = x_outlier[0:0+batchSz]
    elif requestType == "drift":
        rows = x_h1[0:0+batchSz]
    else:
        rows = x_ref[0:0+batchSz]
    reqJson["inputs"][0]["data"] = rows.flatten().tolist()
    reqJson["inputs"][0]["shape"] = [batchSz, rows.shape[1]]
    headers = {"Content-Type": "application/json", "seldon-model": resourceName}
    response_raw = requests.post(url, json=reqJson, headers=headers)
    print(response_raw)
    print(response_raw.json())

场景一:参考集(正常数据)

调用 infer("income-production.pipeline", 20, "normal"),参考标签为 [0 0 1 1 0 1 0 0 1 0 0 0 0 0 1 1 0 0 0 1]。响应包含两个输出:predict(20 个分类结果,与参考标签完全一致)与 is_outlier(20 个 0,即无异常),符合“参考集不应产生漂移或异常”的预期:

{'model_name': '', 'outputs': [
  {'data': [0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1], 'name': 'predict', 'shape': [20, 1], 'datatype': 'INT64', 'parameters': {'content_type': 'np'}},
  {'data': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'name': 'is_outlier', 'shape': [1, 20], 'datatype': 'INT64', 'parameters': {'content_type': 'np'}}
]}

由于漂移检测是异步侧翼,其结论不进入同步响应,需通过 seldon pipeline inspect 直接读取 Kafka 上该步骤的输出 tensor。inspect 的表达式语法支持 pipelineName.stepName.(inputs|outputs).tensorName(详见 docs-gb/cli/seldon_pipeline_inspect.md):

seldon pipeline inspect income-production.income-drift.outputs.is_drift

输出示例(int64Contents: ["0"] 表示未检测到漂移):

seldon.default.model.income-drift.outputs	cifej9gfh5ss738i5br0	{"name":"is_drift", "datatype":"INT64", "shape":["1", "1"], "parameters":{"content_type":{"stringParam":"np"}}, "contents":{"int64Contents":["0"]}}

场景二:漂移数据

调用 infer("income-production.pipeline", 20, "drift")。predict 输出出现与参考标签明显不同的分布([0 0 0 1 1 0 1 1 1 0 0 0 0 0 1 0 0 1 0 1]),is_outlier 仍为全 0;而 is_drift 变为 1,证明漂移检测器成功捕获了数据分布偏移:

seldon pipeline inspect income-production.income-drift.outputs.is_drift
seldon.default.model.income-drift.outputs	cifejaofh5ss738i5brg	{"name":"is_drift", "datatype":"INT64", "shape":["1", "1"], "parameters":{"content_type":{"stringParam":"np"}}, "contents":{"int64Contents":["1"]}}

场景三:异常数据

调用 infer("income-production.pipeline", 20, "outlier")。predict 结果与参考集基本一致,但 is_outlier 中大量样本被标记为 1([1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1]),且 is_drift 为 0,符合“异常数据应触发异常检测而非漂移检测”的设计:

seldon pipeline inspect income-production.income-drift.outputs.is_drift
seldon.default.model.income-drift.outputs	cifejb8fh5ss738i5bs0	{"name":"is_drift", "datatype":"INT64", "shape":["1", "1"], "parameters":{"content_type":{"stringParam":"np"}}, "contents":{"int64Contents":["0"]}}

五、为模型添加 Anchor 黑盒可解释性

可解释模型同样是 Model 资源,只是在 spec.explainer 下额外声明 type 与黑盒解释目标。官方说明见 docs-gb/explainers.md。本示例的 AnchorTabular 解释器定义见 samples/models/income-explainer.yaml:

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: income-explainer
spec:
  storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/explainer"
  explainer:
    type: anchor_tabular
    modelRef: income

关键字段:

  • type:必须是 MLServer 中 Alibi Explain 运行时支持的解释器类型,此处为 anchor_tabular;
  • modelRef:被解释的黑盒模型名(income);
  • pipelineRef:若要解释整个 Pipeline 而非单模型,可改为引用 Pipeline 名称,且 modelRef 与 pipelineRef 二者只能声明其一。仓库另提供了基于 pipelineRef 的变体 samples/models/income-explainer-pipeline.yaml。

部署与调用

# kubectl
kubectl apply -f samples/models/income-explainer.yaml -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s pipelines income-explainer -n ${NAMESPACE}

# seldon CLI
seldon model load -f samples/models/income-explainer.yaml
seldon model status income-explainer -w ModelAvailable | jq .

以单条参考样本发起解释请求 infer("income-explainer", 1, "normal"),响应中的 explanation 输出(datatype: BYTES,content_type: str)内嵌完整的 AnchorTabular JSON:

  • meta:解释器参数(seed: 1、disc_perc: [25, 50, 75]、threshold: 0.95、delta: 0.1、tau: 0.15、batch_size: 100 等)与 Alibi 版本号;
  • data.anchor:锚点规则,例如 ["Marital Status = Never-Married", "Relationship = Own-child", "Capital Gain <= 0.00"];
  • data.precision / data.coverage:锚点的精度(0.994)与覆盖率(0.0657);
  • data.raw:每个特征的逐级统计(mean、precision、coverage、examples 的 covered_true/covered_false 样本);
  • data.instance:被解释的原始输入实例([47.0, 4.0, 1.0, 1.0, 1.0, 3.0, 4.0, 1.0, 0.0, 0.0, 40.0, 9.0])与预测结果。

响应片段(节选):

{'model_name': 'income-explainer_1', 'model_version': '1', 'id': 'cdd68ba5-c569-4930-886f-fbdc26e24866', 'outputs': [
  {'name': 'explanation', 'shape': [1, 1], 'datatype': 'BYTES', 'parameters': {'content_type': 'str'},
   'data': ['{"meta": {"name": "AnchorTabular", ..., "version": "0.9.1"}, "data": {"anchor": ["Marital Status = Never-Married", "Relationship = Own-child", "Capital Gain <= 0.00"], "precision": 0.9942028985507246, "coverage": 0.0657, ...}}']}
]}

六、资源清理

实验结束后按部署顺序卸载全部资源。kubectl 方式:

kubectl delete -f samples/pipelines/income.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-preprocess.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-drift.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-outlier.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-explainer.yaml -n ${NAMESPACE}

CLI 方式(seldon pipeline unload / seldon model unload,均返回 {}):

seldon pipeline unload income-production
seldon model unload income-preprocess
seldon model unload income
seldon model unload income-drift
seldon model unload income-outlier
seldon model unload income-explainer

七、进阶参考

登录后查看全文
seldon-core