Claude 文档摘要实战指南:从基础 Prompt 到摘要索引 RAG 与 Promptfoo 评估(claude-cookbooks)

原创2026-10-09 21:46:111,879 阅读
文章标签:示例工程人工智能大模型

Claude 文档摘要实战指南:从基础 Prompt 到摘要索引 RAG 与 Promptfoo 评估(claude-cookbooks)

导读

本文基于 claude-cookbooks 仓库的 Summarization 指南 与配套的 Promptfoo 评估说明,系统讲解如何用 Claude 对长文档(尤其法律文件)进行高质量摘要。你将掌握从 PDF 文本抽取、基础/多示例/引导式摘要、长文档分块元摘要,到基于摘要索引的 RAG 检索,以及用 BLEU、ROUGE、LLM 打分构建自动化评估与迭代改进的完整链路。


为什么选择摘要任务:场景与挑战

摘要是 NLP 中最核心也最具挑战的任务之一:把大量文本压缩为更易读的短内容,同时保留关键信息。在法律、金融等领域,文档往往冗长且充满专业术语,人工阅读耗时巨大,摘要需求强烈。

但评估摘要质量是公认的难题:

  • 缺乏客观、统一的度量指标,不同读者对"好摘要"的判断高度主观;
  • 传统经验方法(如 ROUGE)难以捕捉连贯性、事实准确性和相关性等细微维度;
  • "最佳"摘要取决于具体用途、目标受众和期望的详细程度。

因此,指南给出的策略是:结合自动化指标、正则表达式与任务特定标准,组合多种技术来逼近可靠评估。这正是本仓库整套方案的设计初衷——从提示词工程到评估再到迭代改进,形成闭环。


环境准备:安装依赖与初始化客户端

完整运行本指南需要以下 Python 包,以及 promptfoo(用于评估,见后文):

%pip install anthropic pypdf pandas matplotlib numpy rouge-score nltk seaborn --quiet

导入并初始化 Anthropic 客户端。API Key 建议通过 .env 文件配合 python-dotenv 管理,而不是硬编码在代码中:

import re
import anthropic
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

# from dotenv import load_dotenv
# load_dotenv()
api_key = "ANTHROPIC_API_KEY"  # 替换为真实 Key
client = anthropic.Anthropic(api_key=api_key)
print("Setup complete!")

本指南全部示例围绕同一份公开的 Sublease Agreement(转租协议)样本展开,仓库中的 PDF 位于 capabilities/summarization/data/Sample Sublease Agreement.pdf。你也可以替换为任意 PDF,或直接用文本字符串跳过抽取步骤。


数据准备:从 PDF 到可送入模型的文本

PDF 是法律文档最常见的载体。仓库通过 pypdf 抽取文本,并做清洗与截断:

import re
import pypdf

pdf_path = "capabilities/summarization/data/Sample Sublease Agreement.pdf"

def extract_text_from_pdf(pdf_path):
    with open(pdf_path, "rb") as file:
        reader = pypdf.PdfReader(file)
        text = ""
        for page in reader.pages:
            text += page.extract_text() + "\n"
    return text

def clean_text(text):
    # 去除多余空白
    text = re.sub(r"\s+", " ", text)
    # 去除页码
    text = re.sub(r"\n\s*\d+\s*\n", "\n", text)
    return text.strip()

def prepare_for_llm(text, max_tokens=180000):
    # 近似截断到 token 上限(假设平均每 token 4 字符)
    return text[: max_tokens * 4]

def get_llm_text(path):
    extracted_text = extract_text_from_pdf(path)
    cleaned_text = clean_text(extracted_text)
    llm_ready_text = prepare_for_llm(cleaned_text)
    return llm_ready_text

text = get_llm_text(pdf_path)
print(text[:500])

这段流水线将 PDF 逐页抽取文本 → 压缩空白并去页码 → 按 max_tokens*4 字符近似截断,得到可直接放入 prompt 的干净文本。值得注意的是 prepare_for_llm 中 max_tokens=180000 的默认值,说明该方案面向 Claude 的大上下文能力,允许将整份合同一次性送入。


基础摘要:assistant 角色与 stop sequence 的巧妙配合

第一个版本非常简单,但它已经用上了两个重要机制:

  1. assistant 角色引导:在 messages 中预置 {"role": "assistant", "content": "Here is the summary of the legal document: <summary>"},让 Claude 直接从 <summary> 之后续写,避免前置废话;
  2. stop_sequences 截断:设置 stop_sequences=["</summary>"],模型在输出闭合标签后即停止生成。
def basic_summarize(text, max_tokens=1000):
    prompt = f"""Summarize the following text in bullet points. Focus on the main ideas and key details:
    {text}
    """

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=max_tokens,
        system="You are a legal analyst known for highly accurate and detailed summaries of legal documents.",
        messages=[
            {"role": "user", "content": prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the legal document: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text

basic_response = basic_summarize(text, max_tokens=1000)
print(basic_response)

运行结果会得到一个按要点组织的转租协议概述(双方主体、主要条款、租期与租金、关键义务、特殊条款等)。但指南明确指出其局限:输出没有标准化结构,难以机器解析,也难以验证"是否遗漏了什么",在真实业务场景中难以直接信任和使用。


Multi-Shot 基础摘要:少样本学习带来的结构化输出

在基础版本之上,只需追加两样东西,输出质量就会显著提升:

  1. "Do not preamble":直接约束模型只输出答案本体,不做寒暄式开场;
  2. 3 组"原文 → 参考摘要"示例:这就是 few-shot(多示例)学习,Claude 会从示例中自动归纳出期望的格式。

仓库把示例文档和参考摘要存放在 capabilities/summarization/data/multiple_subleases.py 中(三个 document* 与对应的 sample*),避免占用 notebook 空间:

from data.multiple_subleases import document1, document2, document3, sample1, sample2, sample3

def basic_summarize_multishot(text, max_tokens=1000):
    prompt = f"""Summarize the following text in bullet points. Focus on the main ideas and key details:
        {text}

    Do not preamble.

    Use these examples for guidance in summarizing:

    <example1>
        <original1>
            {document1}
        </original1>

        <summary1>
            {sample1}
        </summary1>
    </example1>

    <example2>
        <original2>
            {document2}
        </original2>

        <summary2>
            {sample2}
        </summary2>
    </example2>

    <example3>
        <original3>
            {document3}
        </original3>

        <summary3>
            {sample3}
        </summary3>
    </example3>
    """

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=max_tokens,
        system="You are a legal analyst known for highly accurate and detailed summaries of legal documents.",
        messages=[
            {"role": "user", "content": prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the legal document: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text

basic_multishot_response = basic_summarize_multishot(text, max_tokens=1000)
print(basic_multishot_response)

对比输出可以发现:Claude 自动模仿了示例中的 <parties involved>、<property details>、<term and rent> 等 XML 结构——我们并没有显式要求它遵循示例格式,这就是少样本学习的泛化能力。参考摘要文件位于 capabilities/summarization/data/ 下的 sample-lease*.txt。


高级技术一:引导式摘要(Guided Summarization)

引导式摘要通过显式定义框架,让模型按指定维度组织输出。可以在 prompt 中灵活调节详细程度、专业术语密度或摘要层级。对法律文档,我们引导 Claude 聚焦五个方面:

def guided_legal_summary(text, max_tokens=1000):
    prompt = f"""Summarize the following legal document. Focus on these key aspects:

    1. Parties involved
    2. Main subject matter
    3. Key terms and conditions
    4. Important dates or deadlines
    5. Any unusual or notable clauses

    Provide the summary in bullet points under each category.

    Document text:
    {text}

    """

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=max_tokens,
        system="You are a legal analyst known for highly accurate and detailed summaries of legal documents.",
        messages=[
            {"role": "user", "content": prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the legal document: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text

legal_summary = guided_legal_summary(text)
print(legal_summary)

输出被组织成 "Parties Involved / Main Subject Matter / Key Terms and Conditions / Important Dates/Deadlines / Notable Clauses" 五个板块,显著提升了对关键条款的定位和可读性。指南还指出:这一格式化输出也可以通过 few-shot 示例实现,两种路线可互为替代。


高级技术二:领域定制引导摘要(Domain-Specific Guided Summarization)

通用引导摘要适用于任意文档,但如果已知文档类型(例如确定是转租协议),就可以进一步定制,让模型聚焦该类型最相关的法律术语与关键值。同时把 model 提升为函数参数,便于按任务选择不同模型:

def guided_sublease_summary(text, model="claude-sonnet-4-6", max_tokens=1000):
    prompt = f"""Summarize the following sublease agreement. Focus on these key aspects:

    1. Parties involved (sublessor, sublessee, original lessor)
    2. Property details (address, description, permitted use)
    3. Term and rent (start date, end date, monthly rent, security deposit)
    4. Responsibilities (utilities, maintenance, repairs)
    5. Consent and notices (landlord's consent, notice requirements)
    6. Special provisions (furniture, parking, subletting restrictions)

    Provide the summary in bullet points nested within the XML header for each section. For example:

    <parties involved>
    - Sublessor: [Name]
    // Add more details as needed
    </parties involved>

    If any information is not explicitly stated in the document, note it as "Not specified". Do not preamble.

    Sublease agreement text:
    {text}

    """

    response = client.messages.create(
        model=model,
        max_tokens=max_tokens,
        system="You are a legal analyst specializing in real estate law, known for highly accurate and detailed summaries of sublease agreements.",
        messages=[
            {"role": "user", "content": prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the sublease agreement: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text

sublease_summary = guided_sublease_summary(text)
print(sublease_summary)

关键设计点:

  • XML 标签作为输出容器:每个板块用 <parties involved> 等标签包裹,便于程序化解析(也可用 JSON 等其他格式);
  • "Not specified" 兜底:要求模型对文档未明确的信息如实标注,防止臆造;
  • system prompt 专业化:指定"房地产法律专家",提升领域术语的准确性。

用正则解析 XML 分段输出

因为输出是带 XML 标签的,可以写一个通用正则解析器,按标签提取每个板块:

import re

def parse_sections_regex(text):
    pattern = r"<(.*?)>(.*?)</\1>"
    matches = re.findall(pattern, text, re.DOTALL)

    parsed_sections = {}
    for tag, content in matches:
        items = [item.strip("- ").strip() for item in content.strip().split("\n") if item.strip()]
        parsed_sections[tag] = items

    return parsed_sections

parsed_sections = parse_sections_regex(sublease_summary)

if isinstance(parsed_sections, dict) and "parties involved" in parsed_sections:
    print("Parties involved:")
    for item in parsed_sections["parties involved"]:
        print(f"- {item}")
else:
    print("Error: Parsing failed or 'parties involved' section not found.")
    print("Parsed result:", parsed_sections)

这样,摘要就从"给人看的长文"变成了"可被程序消费的结构化数据",是进入自动化流程(如入库、检索)的前提。


高级技术三:多文档元摘要(Meta-Summarization)

当需要处理同一客户的多个相关文档,或单个超长文档时,可以采用**分块(chunking)**策略:将文本切成小块分别摘要,再把各块摘要合并成一份"关于整批文档"的元摘要。

from data.multiple_subleases import document1, document2, document3

def chunk_text(text, chunk_size=2000):
    return [text[i : i + chunk_size] for i in range(0, len(text), chunk_size)]

def summarize_long_document(text, max_tokens=2000):
    chunks = chunk_text(text)

    # 每个 chunk 先用轻量模型(Haiku)产出临时摘要
    chunk_summaries = [
        guided_sublease_summary(chunk, model="claude-haiku-4-5", max_tokens=max_tokens)
        for chunk in chunks
    ]

    final_summary_prompt = f"""

    You are looking at the chunked summaries of multiple documents that are all related. Combine the following summaries of the document from different truthful sources into a coherent overall summary:

    {"".join(chunk_summaries)}

    1. Parties involved (sublessor, sublessee, original lessor)
    2. Property details (address, description, permitted use)
    3. Term and rent (start date, end date, monthly rent, security deposit)
    4. Responsibilities (utilities, maintenance, repairs)
    5. Consent and notices (landlord's consent, notice requirements)
    6. Special provisions (furniture, parking, subletting restrictions)

    Provide the summary in bullet points nested within the XML header for each section. For example:

    <parties involved>
    - Sublessor: [Name]
    // Add more details as needed
    </parties involved>

    If any information is not explicitly stated in the document, note it as "Not specified".

    Summary:
    """

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=max_tokens,
        system="You are a legal expert that summarizes notes on one document.",
        messages=[
            {"role": "user", "content": final_summary_prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the legal document: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text

text = document1 + document2 + document3
long_summary = summarize_long_document(text)
print(long_summary)

这是一个经典的两级模型分工:

阶段 模型 职责
块级摘要 claude-haiku-4-5(更快更便宜) 为每个 chunk 产出局部结构化摘要
最终合成 claude-sonnet-4-6(更强推理) 将各 chunk 摘要融合为整体元摘要

示例中三个相关转租协议被合并后,元摘要不仅涵盖各自的当事人、物业、租金,还呈现出跨文档的共性与差异(例如两家 sublessee、可选的 3-5 年续租期权等),这是单文档摘要无法提供的信息维度。


Summary Indexed Documents:一种文档级的高级 RAG 方法

传统 RAG 通常在块级别做向量检索,而 Summary Indexed Documents(摘要索引文档) 把操作粒度提升到文档级,尤其适合"大文档集合 + 精确信息定位"的场景。

工作原理

  1. 文档摘要:为语料库中每份文档生成简洁摘要(可先对文本子集快速摘要);
  2. 上下文窗口优化:确保所有摘要都能放进模型的上下文窗口;
  3. 相关性打分:让模型针对查询给每份摘要的 0-10 相关性打分;
  4. 重排序(可选):进一步压缩 top-K 结果;
  5. 回答查询。

优势

  • 更省上下文:用摘要而非全文做排序,比传统 RAG 更高效;
  • 特定任务表现更优:能稳定地把正确文档排到第一;
  • 信息检索更精准:重排序压缩结果,把最精炼相关的信息呈现给模型。

仓库实现:LegalSummaryIndexedDocuments

本仓库用 LegalSummaryIndexedDocuments 类落地了这套方法(完整代码见 guide.ipynb):

class LegalSummaryIndexedDocuments:
    def __init__(self, client):
        self.client = client  # Claude client
        self.documents: list[dict[str, str]] = []  # List of docs to store
        self.summaries: list[str] = []

    def add_document(self, doc_id: str, content: str):
        # Adds a document to the index
        self.documents.append({"id": doc_id, "content": content})

    def generate_summaries(self):
        # Generates summaries for all documents in the index
        for doc in self.documents:
            summary = self._generate_legal_summary(doc["content"])
            self.summaries.append(summary)

    def _generate_legal_summary(self, content: str) -> str:
        # 注意:内容截断到前 2000 字——排序阶段不需要过多信息
        prompt = f"""
        Summarize the following sublease agreement. Focus on these key aspects:

        1. Parties involved (sublessor, sublessee, original lessor)
        2. Property details (address, description, permitted use)
        3. Term and rent (start date, end date, monthly rent, security deposit)
        4. Responsibilities (utilities, maintenance, repairs)
        5. Consent and notices (landlord's consent, notice requirements)
        6. Special provisions (furniture, parking, subletting restrictions)

        Provide the summary in bullet points nested within the XML header for each section. For example:

        <parties involved>
        - Sublessor: [Name]
        // Add more details as needed
        </parties involved>

        If any information is not explicitly stated in the document, note it as "Not specified".

        Sublease agreement text:
        {content[:2000]}...

        Summary:
        """

        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=500,
            temperature=0.2,
            messages=[
                {"role": "user", "content": prompt},
                {
                    "role": "assistant",
                    "content": "Here is the summary of the legal document: <summary>",
                },
            ],
            stop_sequences=["</summary>"],
        )
        return response.content[0].text

    def rank_documents(self, query: str, top_k: int = 3) -> list[tuple[str, float]]:
        """
        Rank documents based on their relevance to the given query.
        We use Haiku here as a cheaper, faster model for ranking.
        """
        ranked_scores = []
        for summary in self.summaries:
            prompt = f"Legal document summary: {summary}\n\nLegal query: {query}\n\nRate the relevance of this legal document to the query on a scale of 0 to 10. Only output the numeric value:"

            response = client.messages.create(
                model="claude-haiku-4-5",
                max_tokens=2,
                temperature=0,
                messages=[{"role": "user", "content": prompt}],
            )
            ranked_score = float(response.content[0].text)
            ranked_scores.append(ranked_score)

        ranked_indices = np.argsort(ranked_scores)[::-1][:top_k]
        return [(self.documents[i]["id"], ranked_scores[i]) for i in ranked_indices]

    def extract_relevant_clauses(self, doc_id: str, query: str) -> list[str]:
        """
        Extracts relevant clauses from a document based on a query.
        """
        doc_content = next(doc["content"] for doc in self.documents if doc["id"] == doc_id)

        prompt = f"""
        Given the following legal query and document content, extract the most relevant clauses or sections and write the answer to the query.
        Provide each relevant clause or section separately, preserving the original legal language:

        Legal query: {query}

        Document content: {doc_content}...

        Relevant clauses or sections (separated by '---'):"""

        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1000,
            temperature=0,
            messages=[{"role": "user", "content": prompt}],
        )

        clauses = re.split(r"\n\s*---\s*\n", response.content[0].text.strip())
        return [clause.strip() for clause in clauses if clause.strip()]

调用链路非常清晰:

from data.multiple_subleases import document1, document2, document3

lsid = LegalSummaryIndexedDocuments(client=client)

# 入库
lsid.add_document("doc1", document1)
lsid.add_document("doc2", document2)
lsid.add_document("doc3", document3)

# 入库时生成摘要(ingestion 阶段完成)
lsid.generate_summaries()

# 针对查询做相关性排序
legal_query = "What contract is for the sublessor Apex Innovations, LLC?"
ranked_results = lsid.rank_documents(legal_query)

print("Initial ranking:", ranked_results)

# 从排名第一的文档中抽取相关条款
top_doc_id = ranked_results[0][0]
relevant_clauses = lsid.extract_relevant_clauses(top_doc_id, legal_query)

print("\nRelevant clauses from the top-ranked document:")
for i, clause in enumerate(relevant_clauses[1:], 1):
    print(f"Clause {i}: {clause}")

实现中的几个值得借鉴的工程决策:

  • 模型分工再细化:摘要与条款抽取用 claude-sonnet-4-6,而打分排序用 claude-haiku-4-5(更便宜更快),且 max_tokens=2、temperature=0 只输出数值;
  • 摘要截断 2000 字:排序阶段不需要全文信息,控制输入规模;
  • 条款用 --- 分隔:配合正则 re.split 切分抽取结果,保留原始法律语言。

运行示例中,针对 "Apex Innovations, LLC" 的查询,模型正确识别出实际主体是 "Apex Innovations, Inc." 并返回对应合同与相关条款——这正是摘要索引在"精准信息定位"上的价值体现。

Summarization RAG 最佳实践

  • 最优摘要长度:实验不同摘要长度,在简洁性与信息量之间取平衡;
  • 迭代重排序:文档集较大时,可进行多轮重排序以获得更精确结果;
  • 缓存:对摘要和初始排序结果做缓存,加速重复查询。

评估:用 Promptfoo 建立自动化摘要评估

摘要评估主观且无统一标准,随数据集增大和 prompt 增多,必须借助可扩展的工具。本仓库选用开源的 Promptfoo,评估配置集中在 capabilities/summarization/evaluation/ 目录。

前置条件

  • 安装 Node.js 与 npm(Promptfoo 运行依赖),使用 npx 直接运行;
  • 额外 Python 依赖(自定义评估脚本需要):
pip install nltk rouge-score
  • 设置 API Key:export ANTHROPIC_API_KEY=YOUR_API_KEY。

注意:本目录已初始化好 promptfooconfig.yaml,无需再执行 npx promptfoo@latest init。

运行评估与查看结果

cd capabilities/summarization/evaluation
npx promptfoo@latest eval -c promptfooconfig.yaml --output ../data/results.csv
npx promptfoo@latest view   # 动态查看结果

评估结果会写入 capabilities/summarization/data/results.csv。

配置解剖:promptfooconfig.yaml

promptfooconfig.yaml 是整个评估的心脏,定义了四大板块:

description: 'Summarization Evaluation'

prompts:
  - prompts.py:basic_summarize
  - prompts.py:guided_legal_summary
  - prompts.py:summarize_long_document

providers:
  - id: anthropic:messages:claude-haiku-4-5
    label: "3.0 Haiku"
    config:
      max_tokens: 4096
      temperature: 0
  - id: anthropic:messages:claude-sonnet-4-6
    label: "3.5 Sonnet"
    config:
      max_tokens: 4096
      temperature: 0

defaultTest:
  assert:
    - type: not-contains-any
      value:
        - "I am an AI"
        - "As an AI language model"
        - "I'm an AI assistant"
        - "I am a chatbot"

tests: tests.yaml

outputPath: ../data/results.csv

evaluateOptions.showProgressBar: true
配置块 作用
prompts 从 prompts.py 导入三个摘要函数(基础、引导式、长文档元摘要),对应上文三种方法
providers 配置两个模型:claude-haiku-4-5(标签 "3.0 Haiku")与 claude-sonnet-4-6(标签 "3.5 Sonnet"),均设 max_tokens: 4096、temperature: 0,用于跨模型对比
defaultTest.assert 内置断言:所有输出不得包含 "I am an AI" 等 AI 身份声明式前缀
tests 测试用例从 tests.yaml 导入
outputPath 结果输出到 ../data/results.csv

temperature: 0 保证评估可复现;max_tokens: 4096 给足摘要空间。

测试用例与四种断言

tests.yaml 定义了 9 组用例(sample-lease1.txt 到 sample-lease9.txt),每组包含输入 input(源文档)与 ground_truth(人工/参考摘要),分别指向 capabilities/summarization/data/ 下的 sample-lease*.txt 与 sample-lease*-summary.txt。每组用例挂 4 个断言:

  1. icontains-all(内置):输出必须包含全部 6 个 XML 板块标签(parties involved、property details、term and rent、responsibilities、consent and notices、special provisions)——验证结构化输出;
  2. python → llm_eval.py:LLM 评估;
  3. python → rouge_eval.py:ROUGE 评估;
  4. python → bleu_eval.py:BLEU 评估。

三个自定义评估器

Promptfoo 允许通过 type: python + file:// 挂载任意 Python 评估函数,每个文件都导出统一的 get_assert(output, context) 接口。

BLEU(bleu_eval.py):用 NLTK 的 sentence_bleu,将生成摘要与参考摘要分词后计算 4-gram 权重均匀的 BLEU,默认阈值 0.3。BLEU 衡量机器文本与参考文本的重叠度。

ROUGE(rouge_eval.py):用 rouge_scorer 计算 rouge1、rouge2、rougeL(开启 stemmer),取三者 F-measure 均值与阈值 0.3 比较。ROUGE 更侧重"摘要是否覆盖了参考摘要中的关键内容",是摘要评估的经典指标。

LLM 评估(llm_eval.py):让 Claude(claude-sonnet-4-6)从四个维度对摘要打分:

  1. 简洁性(Conciseness,1-5)
  2. 准确性(Accuracy,1-5)
  3. 完整性(Completeness,1-5)
  4. 清晰性(Clarity,1-5)

外加一段自由文本"Explanation"。prompt 还给出了 7 条评审准则(如"是否准确捕捉关键条款""是否遗漏重要细节""是否公正代表整份文件"),并要求以 JSON 输出、用 <json> 与 </json> 做 stop sequence。最终:

# 过滤掉非数值字段,求四个维度平均分
numeric_values = [value for key, value in evaluation.items() if isinstance(value, (int, float))]
avg_score = sum(numeric_values) / len(numeric_values)
# 归一化:score / 4,阈值默认 0.5

这套"规则化指标(BLEU/ROUGE)+ 语义化指标(LLM 评分)"的组合,正是本指南开头强调的"自动化指标 + 任务特定标准"的组合思路落地。

结果解读

评估完成后,回到 guide.ipynb 用 pandas + matplotlib + seaborn 读取 results.csv,解析每列 [PASS|FAIL] (score) 格式,绘制各模型 × 各 prompt 的通过率柱状图与分数分布箱线图。指南示例中,claude-sonnet-4-6("3.5 Sonnet")整体通过率约 66%,45 个测试中仅 3 个失败,优于 claude-haiku-4-5。同时要清醒认识到:这里用的是概念性数据(Claude 生成或来自 SEC 官网),真实数据下表现会更好,因为我们对真实问题域有更多先验知识。


迭代改进:把评估结果变成优化循环

摘要 prompt 工程是"艺术 + 工程"的迭代过程。指南给出的改进路径:

  1. 分析 Promptfoo 结果定位强弱项:例如发现 icontains-all(contains eval)频繁失败,往往是文档本身不含要求的信息——此时应优化评估器本身,而非一味改 prompt;
  2. 针对具体问题精炼 prompt:改善简洁性或完整性。少样本(multi-shot)是很好的起点,可将其与高级技术结合进一步提升;
  3. 实验不同的分块策略:针对长文档,调整 chunk_size 与块间重叠;
  4. 调优 temperature 与 max_tokens:例如评估场景固定 temperature=0,生产场景按需调整;
  5. 增加后处理步骤:如对输出做正则清洗、结构校验、去重等。

仓库示例恰好演示了这一点:基础摘要(无结构)→ 少样本(隐式获得结构)→ 引导式(显式框架)→ 领域定制(XML 标签 + 正则解析)→ 元摘要(分块 + 两级模型)——每一步都建立在前一步的不足之上。


结论与最佳实践

构建"完美"的摘要系统与评估框架是一门艺术,需要多种方法组合。评估摘要时不要追求 100% 准确率,而是与自己手工完成该任务的能力对标——Claude 的速度与效率使这种方法论式流程真正可行,让人把时间花在最终决策上。

综合全篇,核心最佳实践:

  1. 写清晰具体的 prompt,善用 "Do not preamble" 约束输出形态;
  2. 至少提供 2 个示例(few-shot),让模型自动归纳期望格式;
  3. 对领域文档使用引导式摘要,可配合 XML/JSON 标签输出 + 正则解析实现程序化消费;
  4. 长文档采用高级策略:分块 + 轻量模型(如 Haiku)做块级摘要、强模型做最终合成,或使用摘要索引 RAG;
  5. 定期评估并精炼:用 Promptfoo 的 BLEU/ROUGE/LLM 打分形成量化反馈,驱动迭代;
  6. 正视 AI 生成摘要的伦理与局限:涉及事实敏感场景时保留人工复核环节。

通过本仓库的完整代码(guide.ipynb、prompts.py、promptfooconfig.yaml、tests.yaml)与数据(capabilities/summarization/data/),你可以将这套"摘要 → 结构化 → 检索 → 评估 → 迭代"的完整方法论直接迁移到自己的文档处理业务中。

登录后查看全文
claude-cookbooks