Seldon Core 2 收入分类器生产级部署实战:漂移检测、异常检测与 Anchor 可解释 Pipeline
Seldon Core 2 收入分类器生产级部署实战:漂移检测、异常检测与 Anchor 可解释 Pipeline
本文基于 Seldon Core 2 的表格型收入(Adult Census Income)分类器示例,完整演示如何将预处理、分类、漂移检测、异常检测与 Anchor 可解释模型编排为一条生产级推理 Pipeline,并通过 V2 推理协议与 seldon CLI 验证其监控输出。读完本文,你将掌握 Seldon Core 2 中 Model 资源定义、Pipeline 多步编排(含异步监控分支)、漂移/异常检测结果的读取方法,以及黑盒模型的可解释部署流程。
一、示例整体架构与数据准备
该示例围绕 Adult Census Income 数据集构建了一组配套模型:原始特征经 income-preprocess 预处理后,交由 income 分类器预测收入等级;同时,income-outlier(基于 OutlierVAE 的重构误差)与 income-drift(基于 TabularDrift 的统计漂移检测)分别在主路径的侧翼运行,形成对线上数据质量的持续监控。完整训练流程见仓库脚本 samples/examples/income_classifier/train.py,其中使用了 Alibi-Detect 的 TabularDrift 与 OutlierVAE、Alibi-Explain 的 AnchorTabular,并基于 sklearn 的 ColumnTransformer + RandomForestClassifier 训练分类器与独立保存的 preprocessor。
运行推理 Notebook 需要先准备推理数据,两种方式任选其一:
- 运行训练脚本产出数据:
make train; - 或直接从模型仓库下载推理数据:
gsutil cp -R gs://seldon-models/scv2/examples/income/infer-data .
数据文件 infer-data/test.npy 中按顺序保存了四份数据:参考集 x_ref(正常数据)、漂移数据 x_h1(低教育程度样本)、参考标签 y_ref、异常数据 x_outlier(注入离群特征后的样本)。这与训练脚本末尾的保存逻辑一致(见 samples/examples/income_classifier/train.py)。
import numpy as np
import json
import requests
with open('./infer-data/test.npy', 'rb') as f:
x_ref = np.load(f)
x_h1 = np.load(f)
y_ref = np.load(f)
x_outlier = np.load(f)
二、定义四个监控与分类模型
示例将漂移检测模型、异常检测模型均视作普通 Model 资源部署,关键区别在于 requirements 字段中声明 mlserver 与 alibi-detect 运行时依赖(官方说明见 docs-gb/drift.md 与 docs-gb/outlier.md)。四个模型的定义如下(对应仓库文件 samples/models/income-preprocess.yaml、samples/models/income.yaml、samples/models/income-drift.yaml、samples/models/income-outlier.yaml):
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: income-preprocess
spec:
storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/preprocessor"
requirements:
- sklearn
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: income
spec:
storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/classifier"
requirements:
- sklearn
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: income-drift
spec:
storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/drift-detector"
requirements:
- mlserver
- alibi-detect
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: income-outlier
spec:
storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/outlier-detector"
requirements:
- mlserver
- alibi-detect
部署方式一:kubectl
在 Kubernetes 集群中(${NAMESPACE} 为部署命名空间)逐个应用并等待就绪:
kubectl apply -f samples/models/income-preprocess.yaml -n ${NAMESPACE}
kubectl apply -f samples/models/income.yaml -n ${NAMESPACE}
kubectl apply -f samples/models/income-drift.yaml -n ${NAMESPACE}
kubectl apply -f samples/models/income-outlier.yaml -n ${NAMESPACE}
应用成功后依次输出 model.mlops.seldon.io/income-preprocess created 等确认信息。随后使用 kubectl wait 等待 ready 条件:
kubectl wait --for condition=ready --timeout=300s model income-preprocess -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model income -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model income-drift -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model income-outlier -n ${NAMESPACE}
部署方式二:seldon CLI
Seldon Core 2 的 seldon CLI 支持从 YAML 直接加载模型(-f 指定文件),底层通过 Scheduler API 完成注册:
seldon model load -f samples/models/income-preprocess.yaml
seldon model load -f samples/models/income.yaml
seldon model load -f samples/models/income-drift.yaml
seldon model load -f samples/models/income-outlier.yaml
加载成功后各返回 {}。随后用 -w ModelAvailable 条件等待模型可用(seldon model status 的 -w/--wait 参数指定等待条件,详见 docs-gb/cli/seldon_model_status.md):
seldon model status income-preprocess -w ModelAvailable | jq .
seldon model status income -w ModelAvailable | jq .
seldon model status income-drift -w ModelAvailable | jq .
seldon model status income-outlier -w ModelAvailable | jq .
三、编排 income-production Pipeline
Pipeline 将模型连接成数据流图,支持模型链、tensor 映射、条件逻辑以及监控侧翼(参见 docs-gb/pipelines.md)。本示例的 Pipeline 定义见 samples/pipelines/income.yaml:
apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
name: income-production
spec:
steps:
- name: income
- name: income-preprocess
- name: income-outlier
inputs:
- income-preprocess
- name: income-drift
batch:
size: 20
output:
steps:
- income
- income-outlier.outputs.is_outlier
各字段语义:
steps:Pipeline 中的模型步骤,步骤名与已部署模型同名(模型可先于或与 Pipeline 同时部署);inputs:步骤的数据来源,income-outlier显式声明输入为income-preprocess的输出,而income与income-drift隐式接收 Pipeline 的原始输入;batch.size: 20:income-drift以每 20 条请求批量聚合后再执行漂移检测,属于异步监控路径,不参与同步推理输出的组合;output.steps:声明 Pipeline 的同步输出,包含分类结果income与异常检测标志income-outlier.outputs.is_outlier,因此该 Pipeline 可被 REST/gRPC 同步调用。
加载 Pipeline
kubectl 方式:
kubectl apply -f samples/pipelines/income.yaml -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s pipelines income -n ${NAMESPACE}
CLI 方式(seldon pipeline load 加载,seldon pipeline status -w PipelineReady 等待就绪):
seldon pipeline load -f samples/pipelines/income.yaml
seldon pipeline status income-production -w PipelineReady | jq -M .
就绪后返回的状态结构包含 versions[].pipeline.steps(步骤列表及 batch、inputs 明细)、output.steps 以及 state.status: "PipelineReady"、modelsReady: true 等字段,其中步骤顺序由 Scheduler 拓扑排序决定(例如 income-drift 的 batch 与 income-outlier 的 inputs 均被完整回显):
{
"pipelineName": "income-production",
"versions": [
{
"pipeline": {
"name": "income-production",
"uid": "cifej8iufmbc73e5int0",
"version": 1,
"steps": [
{ "name": "income" },
{ "name": "income-drift", "batch": { "size": 20 } },
{ "name": "income-outlier", "inputs": ["income-preprocess.outputs"] },
{ "name": "income-preprocess" }
],
"output": {
"steps": ["income.outputs", "income-outlier.outputs.is_outlier"]
},
"kubernetesMeta": {}
},
"state": {
"pipelineVersion": 1,
"status": "PipelineReady",
"reason": "created pipeline",
"lastChangeTimestamp": "2023-06-30T14:41:38.343754921Z",
"modelsReady": true
}
}
]
}
四、通过 V2 推理协议发起预测
Seldon Core 2 的推理接口遵循 V2 协议,请求体需携带命名张量(name、data、datatype、shape),并通过 HTTP 头 seldon-model 指定目标资源(模型名或 Pipeline名.pipeline)。示例中的辅助函数支持三种请求类型:normal(参考集)、drift(漂移集)、outlier(异常集):
reqJson = json.loads('{"inputs":[{"name":"input_1","data":[],"datatype":"FP32","shape":[]}]}')
url = "http://0.0.0.0:9000/v2/models/model/infer"
def infer(resourceName: str, batchSz: int, requestType: str):
if requestType == "outlier":
rows = x_outlier[0:0+batchSz]
elif requestType == "drift":
rows = x_h1[0:0+batchSz]
else:
rows = x_ref[0:0+batchSz]
reqJson["inputs"][0]["data"] = rows.flatten().tolist()
reqJson["inputs"][0]["shape"] = [batchSz, rows.shape[1]]
headers = {"Content-Type": "application/json", "seldon-model": resourceName}
response_raw = requests.post(url, json=reqJson, headers=headers)
print(response_raw)
print(response_raw.json())
场景一:参考集(正常数据)
调用 infer("income-production.pipeline", 20, "normal"),参考标签为 [0 0 1 1 0 1 0 0 1 0 0 0 0 0 1 1 0 0 0 1]。响应包含两个输出:predict(20 个分类结果,与参考标签完全一致)与 is_outlier(20 个 0,即无异常),符合“参考集不应产生漂移或异常”的预期:
{'model_name': '', 'outputs': [
{'data': [0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1], 'name': 'predict', 'shape': [20, 1], 'datatype': 'INT64', 'parameters': {'content_type': 'np'}},
{'data': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'name': 'is_outlier', 'shape': [1, 20], 'datatype': 'INT64', 'parameters': {'content_type': 'np'}}
]}
由于漂移检测是异步侧翼,其结论不进入同步响应,需通过 seldon pipeline inspect 直接读取 Kafka 上该步骤的输出 tensor。inspect 的表达式语法支持 pipelineName.stepName.(inputs|outputs).tensorName(详见 docs-gb/cli/seldon_pipeline_inspect.md):
seldon pipeline inspect income-production.income-drift.outputs.is_drift
输出示例(int64Contents: ["0"] 表示未检测到漂移):
seldon.default.model.income-drift.outputs cifej9gfh5ss738i5br0 {"name":"is_drift", "datatype":"INT64", "shape":["1", "1"], "parameters":{"content_type":{"stringParam":"np"}}, "contents":{"int64Contents":["0"]}}
场景二:漂移数据
调用 infer("income-production.pipeline", 20, "drift")。predict 输出出现与参考标签明显不同的分布([0 0 0 1 1 0 1 1 1 0 0 0 0 0 1 0 0 1 0 1]),is_outlier 仍为全 0;而 is_drift 变为 1,证明漂移检测器成功捕获了数据分布偏移:
seldon pipeline inspect income-production.income-drift.outputs.is_drift
seldon.default.model.income-drift.outputs cifejaofh5ss738i5brg {"name":"is_drift", "datatype":"INT64", "shape":["1", "1"], "parameters":{"content_type":{"stringParam":"np"}}, "contents":{"int64Contents":["1"]}}
场景三:异常数据
调用 infer("income-production.pipeline", 20, "outlier")。predict 结果与参考集基本一致,但 is_outlier 中大量样本被标记为 1([1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1]),且 is_drift 为 0,符合“异常数据应触发异常检测而非漂移检测”的设计:
seldon pipeline inspect income-production.income-drift.outputs.is_drift
seldon.default.model.income-drift.outputs cifejb8fh5ss738i5bs0 {"name":"is_drift", "datatype":"INT64", "shape":["1", "1"], "parameters":{"content_type":{"stringParam":"np"}}, "contents":{"int64Contents":["0"]}}
五、为模型添加 Anchor 黑盒可解释性
可解释模型同样是 Model 资源,只是在 spec.explainer 下额外声明 type 与黑盒解释目标。官方说明见 docs-gb/explainers.md。本示例的 AnchorTabular 解释器定义见 samples/models/income-explainer.yaml:
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: income-explainer
spec:
storageUri: "gs://seldon-models/scv2/examples/mlserver_1.3.5/income/explainer"
explainer:
type: anchor_tabular
modelRef: income
关键字段:
type:必须是 MLServer 中 Alibi Explain 运行时支持的解释器类型,此处为anchor_tabular;modelRef:被解释的黑盒模型名(income);pipelineRef:若要解释整个 Pipeline 而非单模型,可改为引用 Pipeline 名称,且modelRef与pipelineRef二者只能声明其一。仓库另提供了基于pipelineRef的变体 samples/models/income-explainer-pipeline.yaml。
部署与调用
# kubectl
kubectl apply -f samples/models/income-explainer.yaml -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s pipelines income-explainer -n ${NAMESPACE}
# seldon CLI
seldon model load -f samples/models/income-explainer.yaml
seldon model status income-explainer -w ModelAvailable | jq .
以单条参考样本发起解释请求 infer("income-explainer", 1, "normal"),响应中的 explanation 输出(datatype: BYTES,content_type: str)内嵌完整的 AnchorTabular JSON:
meta:解释器参数(seed: 1、disc_perc: [25, 50, 75]、threshold: 0.95、delta: 0.1、tau: 0.15、batch_size: 100等)与 Alibi 版本号;data.anchor:锚点规则,例如["Marital Status = Never-Married", "Relationship = Own-child", "Capital Gain <= 0.00"];data.precision/data.coverage:锚点的精度(0.994)与覆盖率(0.0657);data.raw:每个特征的逐级统计(mean、precision、coverage、examples的 covered_true/covered_false 样本);data.instance:被解释的原始输入实例([47.0, 4.0, 1.0, 1.0, 1.0, 3.0, 4.0, 1.0, 0.0, 0.0, 40.0, 9.0])与预测结果。
响应片段(节选):
{'model_name': 'income-explainer_1', 'model_version': '1', 'id': 'cdd68ba5-c569-4930-886f-fbdc26e24866', 'outputs': [
{'name': 'explanation', 'shape': [1, 1], 'datatype': 'BYTES', 'parameters': {'content_type': 'str'},
'data': ['{"meta": {"name": "AnchorTabular", ..., "version": "0.9.1"}, "data": {"anchor": ["Marital Status = Never-Married", "Relationship = Own-child", "Capital Gain <= 0.00"], "precision": 0.9942028985507246, "coverage": 0.0657, ...}}']}
]}
六、资源清理
实验结束后按部署顺序卸载全部资源。kubectl 方式:
kubectl delete -f samples/pipelines/income.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-preprocess.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-drift.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-outlier.yaml -n ${NAMESPACE}
kubectl delete -f samples/models/income-explainer.yaml -n ${NAMESPACE}
CLI 方式(seldon pipeline unload / seldon model unload,均返回 {}):
seldon pipeline unload income-production
seldon model unload income-preprocess
seldon model unload income
seldon model unload income-drift
seldon model unload income-outlier
seldon model unload income-explainer
七、进阶参考
- Pipeline 演进过程:仓库提供了三个递增版本,从仅含分类器(samples/pipelines/income-v1.yaml)到增加异步漂移检测(samples/pipelines/income-v2.yaml),再到本文所述的完整监控形态(samples/pipelines/income-v3.yaml),可对照理解 Pipeline 配置的演化;
- 本地运行:若以
LOCAL_MODEL_FOLDER启动本地 Seldon,可参考本地版 Notebook 说明 samples/examples/income_classifier/infer_local.md,其命令与本文一致但模型存储指向本地目录; - 同类示例:漂移与异常检测的通用接入方法见 docs-gb/drift.md 与 docs-gb/outlier.md;图像分类的 CIFAR-10 完整示例见 docs-gb/examples/cifar10.md;
- 底层实现:Pipeline 内部基于 Kafka 消息驱动,每个步骤的输入/输出均对应独立 Kafka topic,因此
seldon pipeline inspect可以直接回放查看任意步骤(含漂移检测等异步步骤)的数据;该数据面设计在 docs-gb/pipelines.md 中有详细说明。