文本摘要技术全景:NLP-progress 英文摘要基准数据集与 SOTA 追踪指南
文本摘要技术全景:NLP-progress 英文摘要基准数据集与 SOTA 追踪指南
文本摘要(Summarization)是自然语言处理中一项核心的生成式任务:给定一篇或多篇文档,模型需要产出一段更短、但保留输入绝大部分语义的文本。本文以 NLP-progress 仓库 中的 english/summarization.md 为骨架,系统梳理英文摘要评测领域的主流基准数据集(CNN/Daily Mail、Gigaword、X-Sum、DUC 2004、Webis-TLDR-17、Webis-Snippet-20、句子压缩 Google 数据集)及其上历年的 state-of-the-art(SOTA)结果,并解释这些指标与数据表的正确读法。读完本文,你将掌握:如何理解与比较 ROUGE/METEOR 分数的局限、各数据集的任务设定与规模、各模型在每一基准上的精确得分,以及如何利用仓库配套的 结构化导出工具 将 SOTA 表格转成机器可读的 JSON。
任务定义:什么是文本摘要
文本摘要(Summarization)是指从一篇或多篇文档中生成一个较短的版本,同时保留输入的大部分含义。它通常被划分为两条技术路线:
- 抽取式(Extractive):直接从原文中挑选句子或片段拼接成摘要,内容与原文逐字重叠,典型代表如 Lead-3、SummaRuNNer、BERT-ext 等;
- 生成式 / 抽象式(Abstractive):由模型重新组织语言生成新句子,允许使用原文中没有的词与句法,典型代表如 Pointer-Generator、BART、PEGASUS、T5、BRIO 等;
- 混合式(Mixed):先抽取后改写,例如
rnn-ext + abs + RL + rerank、BertSumExtAbs 等两阶段方案。
NLP-progress 项目的定位,正是「追踪 NLP 各常见任务与数据集的 SOTA 进展」——这一目标在 README.md 中有明确说明:它既覆盖依赖解析、词性标注等传统核心任务,也覆盖阅读理解、自然语言推理等较新任务,目的就是为研究者提供一个「某个任务该看哪个基准、当前最好的模型是谁」的快速入口。english/summarization.md 正是其中英文摘要任务的章节。
评测警示:ROUGE 与 METEOR 的已知局限
在查看任何一张 SOTA 表之前,必须先了解一个重要的前提。文档开头给出了明确的警告:对于摘要任务,ROUGE、METEOR 这类自动指标存在严重局限,具体有三点:
- 只评估内容选择,不评估其他质量维度。这些指标不会考虑流利度(fluency)、语法正确性(grammaticality)、连贯性(coherence)等摘要质量的重要方面;
- 内容选择也主要依赖词面重叠(lexical overlap)。而一条抽象式摘要完全可能在没有与参考摘要产生任何词面重叠的情况下表达同样的内容,此时指标会给出错误(偏低)的评价;
- 单参考摘要与指标设计初衷不匹配。由于摘要任务本身的主观性,标注者之间的一致性很低,指标本应配合「每个输入多条参考摘要」使用;然而 CNN/Daily Mail、Gigaword 等近期数据集每个输入只提供一条参考摘要。
因此,仅凭这些指标来追踪进展、宣称 SOTA 是有争议的。多数论文会额外进行人工对比不同候选摘要的实验,但这类实验难以跨论文公平比较。这正是阅读本页所有结果表时必须保持的批判性视角:ROUGE 分数是重要参考,但绝不是质量的全部。
英文摘要基准数据集全景
english/summarization.md 覆盖了以下英文摘要评测资源,本文后续逐节展开:
| 数据集 | 任务类型 | 规模特点 | 评测指标 |
|---|---|---|---|
| CNN / Daily Mail | 文档摘要(多句摘要) | 287,226 训练对 | ROUGE-1/2/L、METEOR(可选) |
| Gigaword | 句子摘要 / 标题生成 | 3.8M 训练实例 | ROUGE-1/2/L(F1) |
| X-Sum | 极端摘要(单句新闻摘要) | 204,045 训练样本 | ROUGE-1/2/L |
| DUC 2004 Task 1 | 句子摘要 | 500 篇文档 | ROUGE-1/2/L(recall @ 75 bytes) |
| Webis-TLDR-17 | 社交媒体摘要 | 约 300 万对 | ROUGE-1/2/L |
| Webis-Snippet-20 | (查询偏置)摘要片段生成 | 约 1,000 万对 | ROUGE-1/2/L、Usefulness |
| Sentence Compression(Google) | 句子压缩(删除式) | 21 万对 | F1、压缩率 CR |
CNN / Daily Mail
CNN / Daily Mail 是文本摘要领域使用最广泛的基准之一。该数据集由 [Nallapati et al. (2016)] 处理后用于摘要评测:它包含在线新闻文章(平均 781 个 token)以及与之配对的多句人工摘要(平均 3.75 个句子、56 个 token)。处理后的版本共有 287,226 个训练对、13,368 个验证对、11,490 个测试对。模型统一使用全长度(full-length)F1 分数的 ROUGE-1、ROUGE-2、ROUGE-L 进行评估,METEOR 为可选项。
CNN / Daily Mail 还存在多语言版本,覆盖五种语言,分别对应仓库中的独立章节:法语、德语、西班牙语、俄语 和土耳其语。
实体匿名版本(Anonymized Version)
以下模型在 Nallapati et al. (2016) 提出的实体匿名化版本上进行了评测(将文档中的命名实体替换为匿名占位符):
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | METEOR | Paper / Source | Code |
|---|---|---|---|---|---|---|
| RNES w/o coherence (Wu and Hu, 2018) | 41.25 | 18.87 | 37.75 | - | Learning to Extract Coherent Summary via Deep Reinforcement Learning | |
| SWAP-NET (Jadhav and Rajan, 2018) | 41.6 | 18.3 | 37.7 | - | Extractive Summarization with SWAP-NET: Sentences and Words from Alternating Pointer Networks | |
| HSASS (Al-Sabahi et al., 2018) | 42.3 | 17.8 | 37.6 | - | A Hierarchical Structured Self-Attentive Model for Extractive Document Summarization (HSSAS) | |
| GAN (Liu et al., 2018) | 39.92 | 17.65 | 36.71 | - | Generative Adversarial Network for Abstractive Text Summarization | |
| KIGN+Prediction-guide (Li et al., 2018) | 38.95 | 17.12 | 35.68 | - | Guiding Generation for Abstractive Text Summarization based on Key Information Guide Network | |
| SummaRuNNer (Nallapati et al., 2017) | 39.6 | 16.2 | 35.3 | - | SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents | |
| rnn-ext + abs + RL + rerank (Chen and Bansal, 2018) | 39.66 | 15.85 | 37.34 | - | Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting | 官方实现 |
| ML+RL, with intra-attention (Paulus et al., 2018) | 39.87 | 15.82 | 36.90 | - | A Deep Reinforced Model for Abstractive Summarization | |
| Lead-3 baseline (Nallapati et al., 2017) | 39.2 | 15.7 | 35.5 | - | SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents | |
| ML+RL ROUGE+Novel, with LM (Kryscinski et al., 2018) | 40.02 | 15.53 | 37.44 | - | Improving Abstraction in Text Summarization | |
| (Tan et al., 2017) | 38.1 | 13.9 | 34.0 | - | Abstractive Document Summarization with a Graph-Based Attentional Neural Model | |
| words-lvt2k-temp-att (Nallapti et al., 2016) | 35.46 | 13.30 | 32.65 | - | Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond |
该表揭示了一个清晰的趋势:即使是最强的抽象式模型(如带强化学习的 ML+RL、rnn-ext+abs),在这张匿名版本榜单上也长期难以拉开与 Lead-3 强基线(ROUGE-1 39.2)的差距,这也是「摘要任务用 ROUGE 衡量时抽取式优势明显」这一现象的早期证据。
非匿名版本:抽取式模型(Extractive Models)
以下模型在 See et al. (2017) 引入的非匿名版本上进行评测。第一张表覆盖抽取式模型,第二张表覆盖生成式与混合式方案:
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | METEOR | Paper / Source | Code |
|---|---|---|---|---|---|---|
| MatchSum (Zhong et al., 2020) | 44.41 | 20.86 | 40.55 | - | Extractive Summarization as Text Matching | 官方实现 |
| DiscoBERT w.G_R & G_C (Xu et al. 2019) | 43.77 | 20.85 | 40.67 | - | A Discourse-Aware Neural Extractive Model for Text Summarization | 官方实现 |
| BertSumExt (Liu and Lapata 2019) | 43.85 | 20.34 | 39.90 | - | Text Summarization with Pretrained Encoders | 官方实现 |
| BERT-ext + RL (Bae et al., 2019) | 42.76 | 19.87 | 39.11 | - | Summary Level Training of Sentence Rewriting for Abstractive Summarization | |
| PNBERT (Zhong et al., 2019) | 42.69 | 19.60 | 38.85 | - | Searching for Effective Neural Extractive Summarization: What Works and What's Next | 官方实现 |
| HIBERT (Zhang et al., 2019) | 42.37 | 19.95 | 38.83 | - | HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization | |
| NeuSUM (Zhou et al., 2018) | 41.59 | 19.01 | 37.98 | - | Neural Document Summarization by Jointly Learning to Score and Select Sentences | 官方实现 |
| Latent (Zhang et al., 2018) | 41.05 | 18.77 | 37.54 | - | Neural Latent Extractive Document Summarization | |
| BanditSum (Dong et al., 2018) | 41.5 | 18.7 | 37.6 | - | BANDITSUM: Extractive Summarization as a Contextual Bandit | 官方实现 |
| REFRESH (Narayan et al., 2018) | 40.0 | 18.2 | 36.6 | - | Ranking Sentences for Extractive Summarization with Reinforcement Learning | 官方实现 |
| Lead-3 baseline (See et al., 2017) | 40.34 | 17.70 | 36.57 | 22.21 | Get To The Point: Summarization with Pointer-Generator Networks | 官方实现 |
抽取式赛道在 2020 年前后的分水岭是预训练编码器的引入:从 Lead-3(40.34)到基于 BERT 的 BertSumExt(43.85)、引入篇章级预训练的 HIBERT,再到把抽取视为「文本匹配」问题的 MatchSum(44.41),ROUGE-1 被持续推高,且这一进步主要来自更好的句子表征而非复杂的解码结构。
非匿名版本:生成式模型与混合模型(Abstractive & Mixed Models)
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | METEOR | Paper / Source | Code |
|---|---|---|---|---|---|---|
| BRIO (Liu et al., 2022) | 47.78 | 23.55 | 44.57 | - | BRIO: Bringing Order to Abstractive Summarization | 官方实现 |
| SimCLS (Liu et al., 2021) | 46.67 | 22.15 | 43.54 | - | SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization | 官方实现 |
| GSum (Dou et al., 2020) | 45.94 | 22.32 | 42.48 | - | GSum: A General Framework for Guided Neural Abstractive Summarization | 官方实现 |
| ProphetNet (Yan, Qi, Gong, Liu et al., 2020) | 44.20 | 21.17 | 41.30 | - | ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training | 官方实现 |
| PEGASUS (Zhang et al., 2019) | 44.17 | 21.47 | 41.11 | - | PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization | 官方实现 |
| BART (Lewis et al., 2019) | 44.16 | 21.28 | 40.90 | - | BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension | 官方实现 |
| T5 (Raffel et al., 2019) | 43.52 | 21.55 | 40.69 | - | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | 官方实现 |
| UniLM (Dong et al., 2019) | 43.33 | 20.21 | 40.51 | - | Unified Language Model Pre-training for Natural Language Understanding and Generation | 官方实现 |
| CNN-2sent-hieco-RBM (Zhang et al., 2019) | 42.04 | 19.77 | 39.42 | - | Abstract Text Summarization with a Convolutional Seq2Seq Model | |
| BertSumExtAbs (Liu and Lapata 2019) | 42.13 | 19.60 | 39.18 | - | Text Summarization with Pretrained Encoders | 官方实现 |
| BERT-ext + abs + RL + rerank (Bae et al., 2019) | 41.90 | 19.08 | 39.64 | - | Summary Level Training of Sentence Rewriting for Abstractive Summarization | |
| Two-Stage + RL (Zhang et al., 2019) | 41.71 | 19.49 | 38.79 | - | Pretraining-Based Natural Language Generation for Text Summarization | |
| DCA (Celikyilmaz et al., 2018) | 41.69 | 19.47 | 37.92 | - | Deep Communicating Agents for Abstractive Summarization | |
| EditNet (Moroshko et al., 2018) | 41.42 | 19.03 | 38.36 | - | An Editorial Network for Enhanced Document Summarization | |
| rnn-ext + RL (Chen and Bansal, 2018) | 41.47 | 18.72 | 37.76 | 22.35 | Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting | 官方实现 |
| Bottom-Up Summarization (Gehrmann et al., 2018) | 41.22 | 18.68 | 38.34 | - | Bottom-Up Abstractive Summarization | 官方实现 |
| (Li et al., 2018a) | 41.54 | 18.18 | 36.47 | - | Improving Neural Abstractive Document Summarization with Explicit Information Selection Modeling | |
| (Li et al., 2018b) | 40.30 | 18.02 | 37.36 | - | Improving Neural Abstractive Document Summarization with Structural Regularization | |
| ROUGESal+Ent RL (Pasunuru and Bansal, 2018) | 40.43 | 18.00 | 37.10 | 20.02 | Multi-Reward Reinforced Summarization with Saliency and Entailment | |
| end2end w/ inconsistency loss (Hsu et al., 2018) | 40.68 | 17.97 | 37.13 | - | A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss | |
| RL + pg + cbdec (Jiang and Bansal, 2018) | 40.66 | 17.87 | 37.06 | 20.51 | Closed-Book Training to Improve Summarization Encoder Memory | |
| rnn-ext + abs + RL + rerank (Chen and Bansal, 2018) | 40.88 | 17.80 | 38.54 | 20.38 | Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting | 官方实现 |
| Pointer + Coverage + EntailmentGen + QuestionGen (Guo et al., 2018) | 39.81 | 17.64 | 36.54 | 18.54 | Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation | |
| ML+RL ROUGE+Novel, with LM (Kryscinski et al., 2018) | 40.19 | 17.38 | 37.52 | - | Improving Abstraction in Text Summarization | |
| Pointer-generator + coverage (See et al., 2017) | 39.53 | 17.28 | 36.38 | 18.72 | Get To The Point: Summarization with Pointer-Generator Networks | 官方实现 |
从这张表可以读出一条完整的技术演进线:
- 2017–2018:指针网络与强化学习时代。Pointer-Generator + coverage(39.53)通过指针机制缓解 OOV 与重复生成问题;ML+RL、ROUGESal+Ent、Two-Stage + RL 等方案用策略梯度直接在 ROUGE 上做优化;
- 2019–2020:预训练 Seq2Seq 时代。BART、PEGASUS、T5、ProphetNet 等以去噪/断句掩码预训练的生成模型大幅刷新榜单,把 ROUGE-1 从约 42 推到 44 以上;
- 2021–2022:排序与对比学习时代。SimCLS 提出「先生成后排序」的对比学习两阶段框架,BRIO 将摘要训练统一到「对候选摘要排序」的目标上,以 47.78 的 ROUGE-1 成为该表榜首。
Gigaword
Gigaword 摘要数据集由 Rush et al. (2015) 首次使用,代表句子摘要 / 标题生成任务:输入文档非常短(平均 31.4 个 token),摘要更短(平均 8.3 个 token)。数据集包含 380 万训练实例、18.9 万开发实例和 1,951 个测试实例。模型统一用全长度 F1 分数的 ROUGE-1、ROUGE-2、ROUGE-L 评估。下表按 ROUGE-2 分数降序排列:
| Model | ROUGE-1 | ROUGE-2* | ROUGE-L | Paper / Source | Code |
|---|---|---|---|---|---|
| ControlCopying (Song et al., 2020) | 39.08 | 20.47 | 36.69 | Controlling the Amount of Verbatim Copying in Abstractive Summarization | 官方实现 |
| ProphetNet (Yan, Qi, Gong, Liu et al., 2020) | 39.51 | 20.42 | 36.69 | ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training | 官方实现 |
| UniLM (Dong et al., 2019) | 38.90 | 20.05 | 36.00 | Unified Language Model Pre-training for Natural Language Understanding and Generation | 官方实现 |
| PEGASUS (Zhang et al., 2019) | 39.12 | 19.86 | 36.24 | PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization | 官方实现 |
| BiSET (Wang et al., 2019) | 39.11 | 19.78 | 36.87 | BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization | 官方实现 |
| MASS (Song et al., 2019) | 38.73 | 19.71 | 35.96 | MASS: Masked Sequence to Sequence Pre-training for Language Generation | 官方实现 |
| Re^3 Sum (Cao et al., 2018) | 37.04 | 19.03 | 34.46 | Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization | |
| JointParsing (Song el at., 2020) | 36.61 | 18.85 | 34.33 | Joint Parsing and Generation for Abstractive Summarization | 官方实现 |
| CNN-2sent-hieco-RBM (Zhang et al., 2019) | 37.95 | 18.64 | 35.11 | Abstract Text Summarization with a Convolutional Seq2Seq Model | |
| Reinforced-Topic-ConvS2S (Wang et al., 2018) | 36.92 | 18.29 | 34.58 | A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization | |
| CGU (Lin et al., 2018) | 36.3 | 18.0 | 33.8 | Global Encoding for Abstractive Summarization | 官方实现 |
| Pointer + Coverage + EntailmentGen + QuestionGen (Guo et al., 2018) | 35.98 | 17.76 | 33.63 | Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation | |
| Struct+2Way+Word (Song et al., 2018) | 35.47 | 17.66 | 33.52 | Structure-Infused Copy Mechanisms for Abstractive Summarization | 官方实现 |
| FTSum_g (Cao et al., 2018) | 37.27 | 17.65 | 34.24 | Faithful to the Original: Fact Aware Neural Abstractive Summarization | |
| DRGD (Li et al., 2017) | 36.27 | 17.57 | 33.62 | Deep Recurrent Generative Decoder for Abstractive Text Summarization | |
| SEASS (Zhou et al., 2017) | 36.15 | 17.54 | 33.63 | Selective Encoding for Abstractive Sentence Summarization | 官方实现 |
| EndDec+WFE (Suzuki and Nagata, 2017) | 36.30 | 17.31 | 33.88 | Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization | |
| Seq2seq + selective + MTL + ERAM (Li et al., 2018) | 35.33 | 17.27 | 33.19 | Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization | |
| Seq2seq + E2T_cnn (Amplayo et al., 2018) | 37.04 | 16.66 | 34.93 | Entity Commonsense Representation for Neural Abstractive Summarization | |
| RAS-Elman (Chopra et al., 2016) | 33.78 | 15.97 | 31.15 | Abstractive Sentence Summarization with Attentive Recurrent Neural Networks | |
| words-lvt5k-1sent (Nallapti et al., 2016) | 32.67 | 15.59 | 30.64 | Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond | |
| ABS+ (Rush et al., 2015) | 29.76 | 11.88 | 26.96 | A Neural Attention Model for Sentence Summarization * | |
| ABS (Rush et al., 2015) | 29.55 | 11.32 | 26.42 | A Neural Attention Model for Sentence Summarization * |
注:Rush et al. (2015) 原论文报告的是 ROUGE recall,上表中 Rush 模型的数值是 Chopra et al. (2016) 重新报告的 ROUGE F1 分数,直接对比时务必注意口径一致。
X-Sum(极端摘要)
X-Sum(Extreme Summarization)由 Narayan et al. (2018) 提出,是一个天然不利于抽取式策略、呼吁抽象式建模的数据集。它的设计思想是生成一句话的新闻摘要,数据通过抓取 BBC 在线文章构建。数据集包含 204,045 个训练样本、11,332 个验证样本、11,334 个测试样本;文章平均长度为 431 词(约 20 个句子),摘要平均长度为 23 词。评测指标为 ROUGE-1、ROUGE-2、ROUGE-L。
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | Paper / Source | Code |
|---|---|---|---|---|---|
| BRIO (Liu et al., 2022) | 49.07 | 25.59 | 40.40 | BRIO: Bringing Order to Abstractive Summarization | 官方实现 |
| PEGASUS (Zhang et al., 2019) | 47.21 | 24.56 | 39.25 | PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization | 官方实现 |
| BART (Lewis et al., 2019) | 45.14 | 22.27 | 37.25 | BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension | 官方实现 |
| BertSumExtAbs (Liu et al., 2019) | 38.81 | 16.50 | 31.27 | Text Summarization with Pretrained Encoders | 官方实现 |
| T-ConvS2S | 31.89 | 11.54 | 25.75 | Don't Give Me the Details, Just the Summary! | 官方实现 |
| PtGen | 29.70 | 9.21 | 23.24 | Don't Give Me the Details, Just the Summary! | 官方实现 |
| Seq2Seq | 28.42 | 8.77 | 22.48 | Don't Give Me the Details, Just the Summary! | 官方实现 |
| PtGen-Covg | 28.10 | 8.02 | 21.72 | Don't Give Me the Details, Just the Summary! | 官方实现 |
| Baseline: Extractive Oracle | 29.79 | 8.81 | 22.66 | Don't Give Me the Details, Just the Summary! | 官方实现 |
| Baseline: Lead-3 | 16.30 | 1.60 | 11.95 | Don't Give Me the Details, Just the Summary! | 官方实现 |
| Baseline: Random | 15.16 | 1.78 | 11.27 | Don't Give Me the Details, Just the Summary! | 官方实现 |
X-Sum 的对照格外能说明问题:在 CNN/Daily Mail 上极具竞争力的抽取式基线与 Lead-3 策略,在这里的 ROUGE-L 只有 11.95(甚至低于随机基线 11.27 之外的意义不大),而 Extractive Oracle 也只有 29.79。这说明「取前几句」或「挑选句子」根本无法完成一句话新闻摘要——该数据集强制要求模型具备真正的抽象式改写能力,因此成为衡量生成式摘要模型(PEGASUS、BART、BRIO)的核心战场。
DUC 2004 Task 1
与 Gigaword 类似,DUC 2004 的 Task 1 是一个句子摘要任务。数据集包含 500 篇文档,平均每篇 35.6 个 token,摘要平均 10.4 个 token。由于规模较小,神经模型通常在其他数据集上训练、只在 DUC 2004 上测试。评测指标为 ROUGE-1、ROUGE-2、ROUGE-L 的 recall @ 75 bytes。
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | Paper / Source | Code |
|---|---|---|---|---|---|
| Transformer + LRPE + PE + ALONE + RE-ranking (Takase and Kobayashi, 2020) | 32.57 | 11.63 | 28.24 | All Word Embeddings from One Embedding | 官方实现 |
| Transformer + LRPE + PE + Re-ranking (Takase and Okazaki, 2019) | 32.29 | 11.49 | 28.03 | Positional Encoding to Control Output Sequence Length | 官方实现 |
| DRGD (Li et al., 2017) | 31.79 | 10.75 | 27.48 | Deep Recurrent Generative Decoder for Abstractive Text Summarization | |
| EndDec+WFE (Suzuki and Nagata, 2017) | 32.28 | 10.54 | 27.8 | Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization | |
| Reinforced-Topic-ConvS2S (Wang et al., 2018) | 31.15 | 10.85 | 27.68 | A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization | |
| CNN-2sent-hieco-RBM (Zhang et al., 2019) | 29.74 | 9.85 | 25.81 | Abstract Text Summarization with a Convolutional Seq2Seq Model | |
| Seq2seq + selective + MTL + ERAM (Li et al., 2018) | 29.33 | 10.24 | 25.24 | Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization | |
| SEASS (Zhou et al., 2017) | 29.21 | 9.56 | 25.51 | Selective Encoding for Abstractive Sentence Summarization | |
| words-lvt5k-1sent (Nallapti et al., 2016) | 28.61 | 9.42 | 25.24 | Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond | |
| ABS+ (Rush et al., 2015) | 28.18 | 8.49 | 23.81 | A Neural Attention Model for Sentence Summarization | |
| RAS-Elman (Chopra et al., 2016) | 28.97 | 8.26 | 24.06 | Abstractive Sentence Summarization with Attentive Recurrent Neural Networks | |
| ABS (Rush et al., 2015) | 26.55 | 7.06 | 22.05 | A Neural Attention Model for Sentence Summarization |
注意此表的评测口径与其他数据集不同(recall @ 75 bytes 的截断设置),因此不应跨数据集直接比较分数;它更适合作为跨模型迁移能力的参照:榜单头部由引入可学习相对位置编码(LRPE)并配合输出长度控制与重排序的 Transformer 变体占据。
Webis-TLDR-17 Corpus
Webis-TLDR-17 是来自社交媒体领域的首批大规模摘要数据集之一:它包含约 300 万对「内容 + 作者自写摘要」,数据来自 Reddit。详细背景参见论文 TL;DR: Mining Reddit to Learn Automatic Summarization。
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | Paper/Source | Code |
|---|---|---|---|---|---|
| Transformer + Copy (Gehrmann et al., 2019) | 22 | 6 | 17 | Generating Summaries with Finetuned Language Models | |
| Unified VAE + PGN (Choi et al., 2019) | 19 | 4 | 15 | VAE-PGN based Abstractive Model in Multi-stage Architecture for Text Summarization |
社交媒体的口语化、非正式表达与新闻摘要差异巨大,这也解释了为何该基准上模型的绝对分数明显低于新闻域数据集——领域差异本身就是评测的一部分。
Webis-Snippet-20 Corpus
Webis-Snippet-20 面向一个较新的任务:(查询偏置的)网页摘要片段生成(Abstractive Snippet Generation)。数据包含约 1,000 万对「网页内容 + 抽象式片段」,以及 350 万个「查询词 + 网页内容 + 抽象式片段」三元组。语料由 ClueWeb09、ClueWeb12 与 DMOZ Open Directory Project 编译而成。
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | Usefulness | Paper/Source | Code |
|---|---|---|---|---|---|---|
| Anchor-context + Query biased (Chen et al., 2020) | 25.7 | 5.2 | 20.1 | 66.18 | Abstractive Snippet Generation |
该表在 ROUGE 之外额外引入 Usefulness(有用性)这一人工评测维度,呼应了文档开头的警告——对于片段生成这类开放任务,单靠词面重叠指标远远不够。
句子压缩(Sentence Compression)
句子压缩的目标是通过删除冗余信息得到一个更短的句子,同时保持语法正确性与原文的重要信息,本质上是一种删除式(deletion-based)任务:压缩结果必须是原句的子序列。
Google 数据集
Google 数据集由 Filippova et al. (2013) 构建:首批发布 10,000 对句对压缩对,随后又追加了 200,000 对。评测时使用 eval 部分(共 10,000 对)中的前 1,000 句进行自动评测,用 200,000 对进行训练。
压缩对示例:
Sentence: Floyd Mayweather is open to fighting Amir Khan in the future, despite snubbing the Bolton-born boxer in favour of a May bout with Argentine Marcos Maidana, according to promoters Golden Boy
Compression: Floyd Mayweather is open to fighting Amir Khan in the future.
评测指标有两个:
- F1:按「金标准压缩与生成压缩中保留的 token」计算召回率与精确率;
- 压缩率(Compression Rate, CR):压缩后字符串字符数除以原句字符数。
| Model | F1 | CR | Paper / Source | Code |
|---|---|---|---|---|
| SLAHAN with syntactic information (Kamigaito et al. 2020) | 0.855 | 0.407 | Syntactically Look-Ahead Attention Network for Sentence Compression | 公开实现 |
| BiRNN + LM Evaluator (Zhao et al. 2018) | 0.851 | 0.39 | A Language Model based Evaluator for Sentence Compression | 公开实现 |
| LSTM (Filippova et al., 2015) | 0.82 | 0.38 | Sentence Compression by Deletion with LSTMs | |
| BiLSTM (Wang et al., 2017) | 0.8 | 0.43 | Can Syntax Help? Improving an LSTM-based Sentence Compression Model for New Domains |
从表中可以看到,句子压缩的进展更多体现在 F1(内容保留更精准)而非压缩率上——最新的 SLAHAN 在保持较高压缩率的同时把 F1 提升到 0.855,说明引入句法前瞻注意力有助于在删除时保留关键成分。
仓库配套:把 SOTA 表格导出为结构化数据
english/summarization.md 这类任务页面以 Markdown 表格承载了大量 SOTA 记录。为了让这些数据可被程序复用,NLP-progress 仓库在 structured/ 目录下提供了一个解析导出工具 export.py,能够把散落在 Markdown 中的任务、描述和 SOTA 表格解析为结构化 JSON。
使用方法(来自 structured/README.md):
# 在仓库根目录(LICENSE 文件所在处)执行
python structured/export.py english
export.py 接受一个或多个目录或文件作为位置参数,默认将结果写入 structured.json,也可用 --output 覆盖输出文件名。例如导出整个 english/ 目录后,就能拿到一个按「任务 → 子任务 → 数据集 → SOTA 行」组织的 JSON。
从 export.py 的 parse_markdown_file 可以看出其解析约定:
- H1 标题 = 任务(task),标题后到下一个标题前的内容为任务描述;
- H2 = 子任务(subtask),H3 = 数据集(dataset),H4 = 子数据集(subdataset);
- 每张 SOTA 表至少包含一列 "Model",其余列会被自动识别为指标(如 ROUGE-1、ROUGE-2、ROUGE-L、METEOR),并抽取 "Paper / Source" 列中的论文链接与 "Code" 列中的实现链接;
- 当一个数据集下有多张表(如 CNN/Daily Mail 的匿名 / 非匿名、抽取式 / 生成式分表)时,
handle_multiple_sota_table_exceptions会结合表前的说明行做子数据集划分。
另外,从 export.py 的导入语句看,它仅依赖 argparse、os、pprint、typing、re、sys、json 等 Python 标准库(仓库中的 structured/requirements.txt 目前为空文件),配合 structured/README.md 要求的 Python 3.6+ 即可直接运行,无需额外安装第三方依赖。
本站点如何渲染这些结果表
NLP-progress 同时以 Jekyll 站点形式发布这些任务页面。仓库中的 _includes/table.html 定义了一个可复用的表格渲染片段:它接收 scores(指标列名,逗号分隔)与 results(结果对象列表),对每条结果渲染出「模型名 + 作者(年份)」、各指标分数、论文链接与代码链接单元格,从而保证所有任务页的 SOTA 表样式一致。
若要在本地构建并预览该站点,可参考 jekyll_instructions.md:依次完成 Ruby(2.1.0 以上)、Bundler 安装、克隆仓库、bundle install 安装依赖后,运行 bundle exec jekyll serve,即可在本地浏览器预览所有任务页面(包括本摘要页面的表格渲染效果)。
如何为本页贡献新的 SOTA 结果
阅读 english/summarization.md 时,任何读者都可以按 README.md 的贡献指南把新结果补充进来。规则要点包括:
- 结果:优先收录已发表论文中报告的结果,有影响力的预印本可作为例外;
- 数据集:数据集应至少被除引入论文以外的一篇论文用于评测;
- 代码:有实现时建议在
Code列给出链接,官方实现标记为[Official],非官方实现标记为[Link]; - 新结果:直接在该任务文件的对应表格中按相同格式增加一行,并保持表格按分数降序(最优结果在最上方);
- 新增数据集或任务:为全新任务新建文件并加入 README 目录树;已有任务则在对应文件的对应章节(按字母序)补充数据集说明、评测设置、评测指标、标注示例、下载链接,并至少填入两个结果(含 SOTA)。
这套流程保证了 english/summarization.md 这样的页面能够持续被社区维护,追踪内容始终跟上领域进展。
小结
english/summarization.md 为英文文本摘要研究提供了一份高度浓缩的「地图」:从 CNN/Daily Mail 的抽取式与生成式两条赛道的完整历年成绩,到 Gigaword、X-Sum、DUC 2004 等句子级摘要基准,再到社交网络(Webis-TLDR-17)、网页片段(Webis-Snippet-20)与句子压缩(Google 数据集)等专门场景。阅读时应始终牢记文档开头的警告——ROUGE/METEOR 只衡量内容选择与词面重叠,无法覆盖流利度、语法与连贯性,因此任何「SOTA 榜首」都应当结合人工评测与任务设定来综合判断。配合仓库的 结构化导出工具,这些表格数据还可一键转成 JSON 供进一步的元研究或基准分析使用。