Pre-Training With Whole Word Masking for Chinese Bert

Bibliographic Data

ID23319672
AuthorsYiming Cui (0000-0002-2452-375X, Harbin Institute of Technology), Wanxiang Che (0000-0002-3907-0335, Harbin Institute of Technology), Ting Liu (0000-0001-5479-7228, Harbin Institute of Technology), Bing Qin (0000-0003-2842-9637, Harbin Institute of Technology), Ziqing Yang (0009-0002-1144-3408, Institute of Animal Sciences)
Year2021
Volume29
Pages3504-3514
Publication date2021-01-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueIEEE/ACM Transactions on Audio, Speech, and Language Processing (JOURNAL)
Journal identifiersISSN: 2329-9290 • E-ISSN: 2329-9304
PublisherInstitute of Electrical and Electronics Engineers (IEEE) (PUBLISHER)
DOI10.1109/taslp.2021.3124365
OpenAlexW2952370363
LanguageEN
Citations received51
References cited16

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we aim to first introduce the whole word masking (wwm) strategy for Chinese BERT, along with a series of Chinese pre-trained language models. Then we also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways. Especially, we propose a new masking strategy called MLM as correction (Mac). To demonstrate the effectiveness of these models, we create a series of Chinese pre-trained language models as our baselines, including BERT, RoBERTa, ELECTRA, RBT, etc. We carried out extensive experiments on ten Chinese NLP tasks to evaluate the created Chinese pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. We open-source our pre-trained language models for further facilitating our research community.

Chinese language · Encoder · Language model · Masking (illustration) · Transformer · Word (group theory) · Generative Adversarial Networks and Image Synthesis · Multimodal Machine Learning Applications · Topic Modeling

Tools

Open DOIOpen Access