Deduplicating corpora

Introducing the Document Similarity tool

Bibliographic Data

ID21701241
AuthorsMonika Bednarek (0000-0001-7819-5424, The University of Sydney), Kelvin K H Lee (0000-0002-2375-7474, The University of Sydney)
Year2026
Publication date2026-06-16
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueInternational Journal of Corpus Linguistics (JOURNAL)
Journal identifiersISSN: 1384-6655 • E-ISSN: 1569-9811
PublisherJohn Benjamins Publishing Company (PUBLISHER • NL)
DOI10.1075/ijcl.24197.bed
OpenAlexW7164881107
LanguageEN
References cited22

Corpora containing newspaper articles are widely used in corpus-based discourse research. However, these corpora often contain duplicate texts due to issues such as syndication, the existence of agency copy, and largely similar print and online versions of the same article. While there are several tools that have automated deduplication functions, only a limited number of these tools allow corpus compilers to inspect duplicates before removal. In this paper, we will introduce the Document Similarity tool, which was developed to automatically remove identical texts and then provides near-identical candidates for users to examine side-by-side, giving corpus compilers more control of the deduplication process. We will describe several experiments using corpora that have been deduplicated using the Document Similarity tool in order to demonstrate its utility for corpus creation in corpus-based discourse analysis of newspapers and potentially other text types

Compiler · Corpus linguistics · Data deduplication · Newspaper · Text corpus · Computational and Text Analysis Methods · Natural Language Processing Techniques · Text Readability and Simplification

Tools

Open DOIOpen Access