基于 Ragas 对比 Vertex AI Gemini 模型:RAG 问答任务的评估与选型实战
基于 Ragas 对比 Vertex AI Gemini 模型:RAG 问答任务的评估与选型实战
本教程以 Ragas 评估库为核心,演示如何用统一指标(上下文精确度、忠实度、ROUGE 分数、自定义 Rubrics 评分)对 Vertex AI 上不同 Gemini 模型的 RAG 问答输出进行打分与对比,并通过雷达图、柱状图辅助选型。读完本文,你将掌握:在 Google Vertex AI 环境中初始化 Ragas 评估器、构造
EvaluationDataset、运行批量评估,以及用 Plotly 可视化多模型对比结果的完整链路。
本文是「Ragas × Vertex AI」三篇系列教程的第二篇(模型对比篇),建议先完成系列第一篇 Ragas with Vertex AI(了解 Vertex AI 模型与 Ragas 的接入方式),即使跳过也不影响理解;同系列第三篇 Align LLM Metrics 讲解如何对齐 LLM 评估器以贴近人类判断。
教程目标与整体思路
本教程解决一个非常具体的工程问题:在 RAG 问答(QA)场景中,如何用同一套 Ragas 指标客观地横向对比多个候选生成模型,并据此做出选型决策。
整体流程如下:
- 在 Vertex AI 上初始化 Ragas 的评估器 LLM 与 Embedding;
- 选定一组 Ragas 指标(模型型 + 计算型 + 自定义 Rubrics);
- 用两个候选 Gemini 模型(
gemini-1.5-pro与gemini-1.0-pro)在相同的检索上下文与标准答案上分别生成回答; - 构造两个
EvaluationDataset并运行evaluate(); - 汇总、对比并可视化两个模型的评估结果。
环境准备
安装依赖
在 Jupyter 环境中安装以下依赖包(Ragas 本体、LangChain 及其 Vertex AI 集成、ROUGE 评分库):
%pip install --upgrade --user --quiet langchain-core langchain-google-vertexai langchain ragas rouge_score
其中 rouge_score 是 Ragas 中 RougeScore 指标(计算型指标)的运行时依赖——从 RougeScore 实现 可以看到,该指标在实例化时会惰性导入 rouge_score.rouge_scorer,缺失时会抛出带有安装提示的 ImportError。
重启运行时
新装包后需要重启 Jupyter 内核才能生效:
import IPython
app = IPython.Application.instance()
app.kernel.do_shutdown(True)
重启可能耗时一分钟以上,重启完成后继续后续步骤。
认证(仅 Colab 需要)
在 Google Colab 上运行时,先执行用户认证:
import sys
if "google.colab" in sys.modules:
from google.colab import auth
auth.authenticate_user()
初始化 Vertex AI SDK
设置 Google Cloud 项目信息并初始化 Vertex AI(请将 PROJECT_ID 替换为你的真实项目 ID):
PROJECT_ID = "[your-project-id]" # @param {type:"string"}
LOCATION = "us-central1" # @param {type:"string"}
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
raise ValueError("Please set your PROJECT_ID")
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)
辅助函数:评估报告展示与可视化
教程提供三个辅助函数,用于统一展示评估报告并绘制对比图。它们不是 Ragas 库的一部分,而是为了把评估结果组织成 (标题, summary_metrics, report_df) 三元组结构,便于后续在多个模型之间对比。
display_eval_report(eval_result, metrics=None):以 Markdown 渲染报告标题,并分「Summary Metrics」(各指标均值)与「Report Metrics」(逐条样本明细)两个 DataFrame 展示;可选metrics参数按子串过滤只关心的指标列。plot_radar_plot(eval_results, max_score=5, metrics=None):为每个模型绘制一条极坐标雷达图迹线,max_score控制径向轴上限(示例中 Rubrics 评分为 1~5 分制,故设为 5)。plot_bar_plot(eval_results, metrics=None):以分组柱状图并排对比各模型的每个指标得分。
import pandas as pd
import plotly.graph_objects as go
from IPython.display import HTML, Markdown, display
def display_eval_report(eval_result, metrics=None):
"""Display the evaluation results."""
title, summary_metrics, report_df = eval_result
metrics_df = pd.DataFrame.from_dict(summary_metrics, orient="index").T
if metrics:
metrics_df = metrics_df.filter(
[
metric
for metric in metrics_df.columns
if any(selected_metric in metric for selected_metric in metrics)
]
)
report_df = report_df.filter(
[
metric
for metric in report_df.columns
if any(selected_metric in metric for selected_metric in metrics)
]
)
# Display the title with Markdown for emphasis
display(Markdown(f"## {title}"))
# Display the metrics DataFrame
display(Markdown("### Summary Metrics"))
display(metrics_df)
# Display the detailed report DataFrame
display(Markdown("### Report Metrics"))
display(report_df)
def plot_radar_plot(eval_results, max_score=5, metrics=None):
fig = go.Figure()
for eval_result in eval_results:
title, summary_metrics, report_df = eval_result
if metrics:
summary_metrics = {
k: summary_metrics[k]
for k, v in summary_metrics.items()
if any(selected_metric in k for selected_metric in metrics)
}
fig.add_trace(
go.Scatterpolar(
r=list(summary_metrics.values()),
theta=list(summary_metrics.keys()),
fill="toself",
name=title,
)
)
fig.update_layout(
polar=dict(radialaxis=dict(visible=True, range=[0, max_score])), showlegend=True
)
fig.show()
def plot_bar_plot(eval_results, metrics=None):
fig = go.Figure()
data = []
for eval_result in eval_results:
title, summary_metrics, _ = eval_result
if metrics:
summary_metrics = {
k: summary_metrics[k]
for k, v in summary_metrics.items()
if any(selected_metric in k for selected_metric in metrics)
}
data.append(
go.Bar(
x=list(summary_metrics.keys()),
y=list(summary_metrics.values()),
name=title,
)
)
fig = go.Figure(data=data)
# Change the bar mode
fig.update_layout(barmode="group")
fig.show()
使用 Ragas 指标搭建评估器
定义 evaluator_llm
Ragas 的模型型指标(model-based metrics)依赖 LLM 充当评判者(judge)。这里用 LangChain 的 Vertex AI 封装,再包上 Ragas 的适配层,得到 Ragas 侧的评估 LLM 与 Embedding:
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_google_vertexai import VertexAI, VertexAIEmbeddings
evaluator_llm = LangchainLLMWrapper(VertexAI(model_name="gemini-pro"))
evaluator_embeddings = LangchainEmbeddingsWrapper(VertexAIEmbeddings(model_name="text-embedding-004"))
LangchainLLMWrapper / LangchainEmbeddingsWrapper 是 Ragas 提供的 LangChain 适配器(分别位于 src/ragas/llms/ 与 src/ragas/embeddings/),作用是把任意 LangChain LLM / Embedding 实例统一为 Ragas 内部的 BaseRagasLLM / BaseRagasEmbeddings 接口。在本教程中,评估器使用 gemini-pro 作为评判模型,被评估的候选模型则是 gemini-1.5-pro 与 gemini-1.0-pro——即「评估者」与「被评估者」解耦,这正是模型对比任务的关键设计。
选择并定义 Ragas 指标
教程选取了四类指标,覆盖 Ragas 的三大指标家族:
from ragas import evaluate
from ragas.metrics import ContextPrecision, Faithfulness, RubricsScore, RougeScore
rouge_score = RougeScore()
helpfulness_rubrics = {
"score1_description": "Response is useless/irrelevant, contains inaccurate/deceptive/misleading information, and/or contains harmful/offensive content. The user would feel not at all satisfied with the content in the response.",
"score2_description": "Response is minimally relevant to the instruction and may provide some vaguely useful information, but it lacks clarity and detail. It might contain minor inaccuracies. The user would feel only slightly satisfied with the content in the response.",
"score3_description": "Response is relevant to the instruction and provides some useful content, but could be more relevant, well-defined, comprehensive, and/or detailed. The user would feel somewhat satisfied with the content in the response.",
"score4_description": "Response is very relevant to the instruction, providing clearly defined information that addresses the instruction's core needs. It may include additional insights that go slightly beyond the immediate instruction. The user would feel quite satisfied with the content in the response.",
"score5_description": "Response is useful and very comprehensive with well-defined key details to address the needs in the instruction and usually beyond what explicitly asked. The user would feel very satisfied with the content in the response.",
}
rubrics_score = RubricsScore(name="helpfulness", rubrics=helpfulness_rubrics)
context_precision = ContextPrecision(llm=evaluator_llm)
faithfulness = Faithfulness(llm=evaluator_llm)
四个指标的分工如下:
| 指标 | 家族 | 含义 | 需要的样本字段 |
|---|---|---|---|
ContextPrecision |
模型型 | 判断检索到的上下文对生成答案是否有用,衡量检索质量(带参考版) | user_input、retrieved_contexts、reference(见 src/ragas/metrics/_context_precision.py) |
Faithfulness |
模型型 | 衡量回答中的每个论断是否都能由检索上下文支撑,衡量幻觉程度 | user_input、response、retrieved_contexts(见 src/ragas/metrics/_faithfulness.py) |
RougeScore |
计算型 | 基于 n-gram 重合的文本相似度,默认 rougeL + fmeasure 模式,无需 LLM 调用 |
reference、response(见 src/ragas/metrics/_rouge_score.py) |
RubricsScore |
模型型(自定义) | 用 1~5 分制自定义评分细则,让 LLM 按 rubric 打分(这里是 helpfulness 有用性) | user_input、response(参考无关评分) |
几点源码级说明:
RougeScore支持rouge_type(rouge1/rougeL)与mode(fmeasure/precision/recall)两个可调参数,默认rougeL+fmeasure;实现上使用rouge_scorer.RougeScorer并开启use_stemmer=True(词干化)后取对应模式分数(src/ragas/metrics/_rouge_score.py)。这也解释了结果表中指标列名显示为rouge_score(mode=fmeasure)——Ragas 在汇总时会为带 mode 的指标拼接(mode=...)后缀(见 src/ragas/evaluation.py)。RubricsScore的构造参数为name与rubrics(一个scoreN_description到描述文本的字典),内部会把 rubric 文本拼接到评分 prompt 的 instruction 中(src/ragas/metrics/_domain_specific_rubrics.py),因此你可以自由定制任意维度的评分标准。它同时实现了单轮与多轮评分(SingleTurnMetric、MultiTurnMetric)。context_precision与faithfulness在这里显式传入llm=evaluator_llm;而evaluate()调用时还会再次传入llm=——从 evaluate 实现 看,只有当指标自身llm为None时才会被全局llm覆盖,因此两种传参方式都合法,显式传入可以让指标独立复用。
准备数据集:让两个候选模型分别作答
定义问题、检索上下文与标准答案
# questions or query from user
user_inputs = [
"Which part of the brain does short-term memory seem to rely on?",
"What provided the Roman senate with exuberance?",
"What area did the Hasan-jalalians command?",
]
# retrieved data used in answer generation
retrieved_contexts = [
["Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Long-term memory, on the other hand, is maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, as learned from patient Henry Molaison after removal of both his hippocampi, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three months or more after the initial learning."],
["In 62 BC, Pompey returned victorious from Asia. The Senate, elated by its successes against Catiline, refused to ratify the arrangements that Pompey had made. Pompey, in effect, became powerless. Thus, when Julius Caesar returned from a governorship in Spain in 61 BC, he found it easy to make an arrangement with Pompey. Caesar and Pompey, along with Crassus, established a private agreement, now known as the First Triumvirate. Under the agreement, Pompey's arrangements would be ratified. Caesar would be elected consul in 59 BC, and would then serve as governor of Gaul for five years. Crassus was promised a future consulship."],
["The Seljuk Empire soon started to collapse. In the early 12th century, Armenian princes of the Zakarid noble family drove out the Seljuk Turks and established a semi-independent Armenian principality in Northern and Eastern Armenia, known as Zakarid Armenia, which lasted under the patronage of the Georgian Kingdom. The noble family of Orbelians shared control with the Zakarids in various parts of the country, especially in Syunik and Vayots Dzor, while the Armenian family of Hasan-Jalalians controlled provinces of Artsakh and Utik as the Kingdom of Artsakh."],
]
# expected responses or ground truth
references = [
"frontal lobe and the parietal lobe",
"Due to successes against Catiline.",
"The Hasan-Jalalians commanded the area of Artsakh and Utik.",
]
注意这里的「检索上下文」对两个模型完全相同——只有生成模型不同,才能把分数差异归因于模型本身,这是对比实验保持变量可控的关键。
实例化候选模型并生成回答
from vertexai.generative_models import GenerativeModel
generation_config = {
"max_output_tokens": 128,
"temperature": 0.1,
}
model_a_name = "gemini-1.5-pro"
model_b_name = "gemini-1.0-pro"
gemini_model_15 = GenerativeModel(
model_a_name,
generation_config=generation_config,
)
gemini_model_1 = GenerativeModel(
model_b_name,
generation_config=generation_config,
)
generation_config 统一了 max_output_tokens=128 与低温 temperature=0.1,降低生成随机性对评分的影响。随后用「仅依据给定上下文作答」的模板逐条生成回答:
responses_a = []
responses_b = []
# Template for creating the prompt
template = """Answer the question based only on the following context:
{context}
Question: {query}
"""
# Iterate through each user input and corresponding context
for i in range(len(user_inputs)):
# Join the list of retrieved contexts into a single string
context_str = "\n".join(retrieved_contexts[i])
# Create prompt Generate response for Gemini 1.5 pro model
gemini_15_prompt = template.format(context=context_str, query=user_inputs[i])
gemini_15_response = gemini_model_15.generate_content(gemini_15_prompt)
responses_a.append(gemini_15_response.text)
# Create prompt Generate response for Gemini 1 pro model
gemini_1_prompt = template.format(context=context_str, query=user_inputs[i])
gemini_1_response = gemini_model_1.generate_content(gemini_1_prompt)
responses_b.append(gemini_1_response.text)
转换为 Ragas EvaluationDataset
Ragas 评估的输入是 EvaluationDataset,其最小单元是 SingleTurnSample。SingleTurnSample 的字段定义见 src/ragas/dataset_schema.py,本教程用到 user_input、retrieved_contexts、response、reference 四个字段(reference 即标准答案/ground truth):
from ragas.dataset_schema import SingleTurnSample, EvaluationDataset
n = len(user_inputs)
samples_a = []
samples_b = []
for i in range(n):
sample_a = SingleTurnSample(
user_input=user_inputs[i],
retrieved_contexts=retrieved_contexts[i],
response=responses_a[i],
reference=references[i],
)
sample_b = SingleTurnSample(
user_input=user_inputs[i],
retrieved_contexts=retrieved_contexts[i],
response=responses_b[i],
reference=references[i],
)
samples_a.append(sample_a)
samples_b.append(sample_b)
ragas_eval_dataset_a = EvaluationDataset(samples=samples_a)
ragas_eval_dataset_b = EvaluationDataset(samples=samples_b)
两个数据集在 user_input / retrieved_contexts / reference 上完全一致,只有 response 分别来自两个候选模型。调用 to_pandas() 可以预览数据集内容:
ragas_eval_dataset_a.to_pandas()
| user_input | retrieved_contexts | response | reference | |
|---|---|---|---|---|
| 0 | Which part of the brain does short-term memory... | [Short-term memory is supported by transient p... | Short-term memory relies on regions of the **f... | frontal lobe and the parietal lobe |
| 1 | What provided the Roman senate with exuberance? | [In 62 BC, Pompey returned victorious from Asi... | The Roman Senate was elated by its successes a... | Due to successes against Catiline. |
| 2 | What area did the Hasan-jalalians command? | [The Seljuk Empire soon started to collapse. I... | The Hasan-Jalalians controlled the provinces o... | The Hasan-Jalalians commanded the area of Arts... |
ragas_eval_dataset_b.to_pandas()
| user_input | retrieved_contexts | response | reference | |
|---|---|---|---|---|
| 0 | Which part of the brain does short-term memory... | [Short-term memory is supported by transient p... | The frontal lobe, especially the dorsolateral ... | frontal lobe and the parietal lobe |
| 1 | What provided the Roman senate with exuberance? | [In 62 BC, Pompey returned victorious from Asi... | The Roman Senate's exuberance stemmed from its... | Due to successes against Catiline. |
| 2 | What area did the Hasan-jalalians command? | [The Seljuk Empire soon started to collapse. I... | The Hasan-Jalalians controlled the provinces o... | The Hasan-Jalalians commanded the area of Arts... |
运行评估
将数据集与指标列表传入 Ragas 的 evaluate() 函数,两个模型分别评估一次:
from ragas import evaluate
ragas_metrics = [
context_precision,
faithfulness,
rouge_score,
rubrics_score,
]
ragas_result_rag_a = evaluate(
dataset=ragas_eval_dataset_a, metrics=ragas_metrics, llm=evaluator_llm
)
ragas_result_rag_b = evaluate(
dataset=ragas_eval_dataset_b, metrics=ragas_metrics, llm=evaluator_llm
)
Evaluating: 100%|██████████| 12/12 [00:00<?, ?it/s]
Evaluating: 100%|██████████| 12/12 [00:00<?, ?it/s]
进度条显示 12 个任务(3 个样本 × 4 个指标)。从 evaluate 源码 可以看到,Ragas 会为每个 (样本, 指标) 组合提交一个异步评分任务到 Executor 中并行执行,并通过 RagasTracer 记录每个样本的评估链路,这也是「3 × 4 = 12」的由来。
包装为 Vertex AI 的 EvalResult 结构
为了让结果能直接对接 Vertex AI 生态的展示与后续处理,将 Ragas 结果包装成 vertexai.evaluation.EvalResult:
from vertexai.evaluation import EvalResult
result_rag_a = EvalResult(
summary_metrics=ragas_result_rag_a._repr_dict,
metrics_table=ragas_result_rag_a.to_pandas(),
)
result_rag_b = EvalResult(
summary_metrics=ragas_result_rag_b._repr_dict,
metrics_table=ragas_result_rag_b.to_pandas(),
)
这里用到的两个 Ragas API 说明如下:
to_pandas():EvaluationResult的公开方法,返回逐样本明细表(包含原始字段 + 各指标列),见 src/ragas/dataset_schema.py。_repr_dict:EvaluationResult内部属性,在初始化时对每个指标的逐样本分数取均值(safe_nanmean,NaN 安全)得到汇总分数(src/ragas/dataset_schema.py)。它也是EvaluationResult.__repr__显示内容的数据来源。教程直接使用这一内部属性以复用 Vertex AI 的EvalResult结构——如果你的代码希望更稳妥,可以自行对to_pandas()的指标列取均值来构造summary_metrics。
对比评估结果
查看汇总结果
调用 display_eval_report() 可以在一张表中纵览全部指标:
display_eval_report(
eval_result=(
f"{model_a_name} Eval Result",
result_rag_a.summary_metrics,
result_rag_a.metrics_table,
),
)
gemini-1.5-pro Eval Result —— Summary Metrics
| context_precision | faithfulness | rouge_score(mode=fmeasure) | helpfulness | |
|---|---|---|---|---|
| 0 | 0.666667 | 1.0 | 0.56 | 4.333333 |
Report Metrics
| user_input | retrieved_contexts | response | reference | context_precision | faithfulness | rouge_score(mode=fmeasure) | helpfulness | |
|---|---|---|---|---|---|---|---|---|
| 0 | Which part of the brain does short-term memory... | [Short-term memory is supported by transient p... | Short-term memory relies on regions of the **f... | frontal lobe and the parietal lobe | 1.0 | 1.0 | 0.48 | 5 |
| 1 | What provided the Roman senate with exuberance? | [In 62 BC, Pompey returned victorious from Asi... | The Roman Senate was elated by its successes a... | Due to successes against Catiline. | 0.0 | 1.0 | 0.40 | 4 |
| 2 | What area did the Hasan-jalalians command? | [The Seljuk Empire soon started to collapse. I... | The Hasan-Jalalians controlled the provinces o... | The Hasan-Jalalians commanded the area of Arts... | 1.0 | 1.0 | 0.80 | 4 |
display_eval_report(
(
f"{model_b_name} Eval Result",
result_rag_b.summary_metrics,
result_rag_b.metrics_table,
)
)
gemini-1.0-pro Eval Result —— Summary Metrics
| context_precision | faithfulness | rouge_score(mode=fmeasure) | helpfulness | |
|---|---|---|---|---|
| 0 | 1.0 | 0.916667 | 0.479034 | 4.0 |
Report Metrics
| user_input | retrieved_contexts | response | reference | context_precision | faithfulness | rouge_score(mode=fmeasure) | helpfulness | |
|---|---|---|---|---|---|---|---|---|
| 0 | Which part of the brain does short-term memory... | [Short-term memory is supported by transient p... | The frontal lobe, especially the dorsolateral ... | frontal lobe and the parietal lobe | 1.0 | 1.00 | 0.666667 | 4 |
| 1 | What provided the Roman senate with exuberance? | [In 62 BC, Pompey returned victorious from Asi... | The Roman Senate's exuberance stemmed from its... | Due to successes against Catiline. | 1.0 | 0.75 | 0.130435 | 4 |
| 2 | What area did the Hasan-jalalians command? | [The Seljuk Empire soon started to collapse. I... | The Hasan-Jalalians controlled the provinces o... | The Hasan-Jalalians commanded the area of Arts... | 1.0 | 1.00 | 0.640000 | 4 |
如何解读这份结果
结合两个模型的汇总与明细,可以得出如下分析(基于本教程示例数据):
- Context Precision(检索质量):
gemini-1.5-pro在第 2 个样本上为 0.0(其回答引用了上下文未覆盖的信息),gemini-1.0-pro三个样本均为 1.0。该指标反映的是「检索到的上下文是否被有效利用」,由于两个模型共享同一组retrieved_contexts,这里的分差主要来自模型是否只依据上下文作答。 - Faithfulness(忠实度):
gemini-1.5-pro满分 1.0,gemini-1.0-pro为 0.9167(第 2 个样本 0.75,说明有部分论断未能由上下文支撑)。 - Rouge Score(词面重合度):两者接近(0.56 vs 0.479),说明与标准答案的文本重合程度相当;从明细看,两者在不同样本上各有高低(如样本 0 中 1.0-pro 的 0.667 高于 1.5-pro 的 0.48,而样本 1 中 1.5-pro 的 0.40 更高)。
- Helpfulness(Rubrics 评分):
gemini-1.5-pro均值 4.33 高于gemini-1.0-pro的 4.0,且有一个样本拿到满分 5 分。
因此,在本示例的 3 个样本上,gemini-1.5-pro 在忠实度与有用性评分上占优,而 gemini-1.0-pro 在上下文利用率(context precision)上表现更稳。真实选型时应扩大样本量(例如使用 Ragas 的 Testset 生成器构造 50+ 条测试样本,参见 Ragas 测试集生成),并结合成本、延迟等非质量因素综合决策。
可视化对比
将两个结果组装成 eval_results 列表,分别绘制雷达图与柱状图:
eval_results = []
eval_results.append(
(model_a_name, result_rag_a.summary_metrics, result_rag_a.metrics_table)
)
eval_results.append(
(model_b_name, result_rag_b.summary_metrics, result_rag_b.metrics_table)
)
plot_radar_plot(eval_results, max_score=5)
plot_bar_plot(eval_results)
雷达图适合一眼看出各模型的「能力轮廓」,柱状图则适合逐指标精确比对数值。max_score=5 是因为本组指标中 helpfulness 为 5 分制;若引入其他量纲的指标,可相应调整径向轴范围或对指标做归一化。
源码视角:Ragas 评估管线是如何工作的
为了让读者对评估过程有更深的理解,这里把教程中用到的关键机制与仓库源码对应起来:
- 指标自动注入与校验:
evaluate()在开始评分前会做两件事:一是对每个MetricWithLLM/MetricWithEmbeddings指标,若其自身未绑定 LLM/Embedding 则注入全局传入的llm/embeddings(src/ragas/evaluation.py);二是调用validate_required_columns()与validate_supported_metrics()校验数据集是否包含指标所需的字段(src/ragas/validation.py)。这也是为什么数据集少一个字段(如reference)时评估会直接报错而非静默出错。 - 异步并行执行:每个
(样本, 指标)作为一个任务提交给Executor,evaluate()内部基于aevaluate的异步实现(src/ragas/evaluation.py),show_progress=True时用 tqdm 显示进度。本教程 3 样本 × 4 指标即 12 个任务。 - 结果聚合:
EvaluationResult.__post_init__把逐样本分数转成「指标名 → 分数列表」的字典,并用safe_nanmean求均值得到_repr_dict(汇总分数),to_pandas()则把分数与原始样本合并为明细表(src/ragas/dataset_schema.py)。 - 指标实现可读:
RougeScore是纯计算型(rougeL/fmeasure,见 src/ragas/metrics/_rouge_score.py);ContextPrecision的带参考版要求user_input+retrieved_contexts+reference(src/ragas/metrics/_context_precision.py);Faithfulness要求user_input+response+retrieved_contexts(src/ragas/metrics/_faithfulness.py);RubricsScore则把 rubrics 文本动态拼入评分 prompt(src/ragas/metrics/_domain_specific_rubrics.py)。理解这些字段依赖,有助于你为自己的数据集裁剪指标组合。
延伸阅读
本系列还有两篇配套教程,可继续深入:
- Ragas with Vertex AI:了解如何在 Vertex AI 上使用 Ragas 评估 LLM 工作流,包括自定义指标(
AspectCritic)、模型型指标与计算型指标的分类与定义。 - Align LLM Metrics:训练并对齐你的 LLM 评估器,使其判断更贴近人类专家。
此外,Ragas 的 EvaluationDataset 结构说明 与 可用指标总览 可以帮助你在更大规模的对比实验中扩展指标与数据集。

