A Structured Review of the Validity of BLEU
Bibliographic Data
| ID | 14844385 |
|---|---|
| Authors | Ella Reiter (0000-0002-7548-9504, University of Aberdeen, corresponding author), Ehud Reiter (University of Aberdeen) |
| Year | 2018 |
| Volume | 44 |
| Issue | 3 |
| Pages | 393-401 |
| Publication date | 2018-09-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Computational Linguistics (JOURNAL) |
| Journal identifiers | ISSN: 0891-2017 • E-ISSN: 1530-9312 |
| Publisher | MIT Press - Journals (PUBLISHER) |
| DOI | 10.1162/coli_a_00322 |
| OpenAlex | W2806532810 |
| Language | EN |
| Citations received | 10 |
| References cited | 10 |
The BLEU metric has been widely used in NLP for over 15 years to evaluate NLP systems, especially in machine translation and natural language generation. I present a structured review of the evidence on whether BLEU is a valid evaluation technique—in other words, whether BLEU scores correlate with real-world utility and user-satisfaction of NLP systems; this review covers 284 correlations reported in 34 papers. Overall, the evidence supports using BLEU for diagnostic evaluation of MT systems (which is what it was originally proposed for), but does not support using BLEU outside of MT, for evaluation of individual texts, or for scientific hypothesis testing
BLEU · Machine translation · Natural language processing · Biomedical Text Mining and Ontologies · Computer Science · Natural Language Processing Techniques · Topic Modeling · Artificial Intelligence
AnoTex
An Evaluation of ChatGPT's Translation Accuracy Using BLEU Score
Data-Driven Sentence Simplification
Evaluating Synthetic Data Generation from User Generated Text
Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization
The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification
Varieties of specification
A rapid evidence review of evaluation techniques for large language models in legal use cases
Automatic assessment of spoken-language interpreting based on machine-translation evaluation metrics
Assessing the role of AI in advancing construction sector industrial symbiosis research
| Citations per year | 1.25 |
|---|---|
| Citation span | 2018 - 2026 (9) |
| Highly cited | No |