Accessor Variety Criteria for Chinese Word Extraction

Bibliographic Data

ID12156458
AuthorsHaodi Feng (0000-0002-0656-5905, Shandong University Tsinghua University City University of Hong Kong, School of Computer Science and Technology, Jinan, PRC; Department of Computer Science, Tat Chee Avenue, Kowloon, Hong Kong.or, corresponding author), Kang Chen (0000-0002-2376-2898, Department of Computer Science and Technology, Peking, PR China., corresponding author), Xiaotie Deng (0000-0002-5282-6467, City University of Hong Kong Tsinghua University, Department of Computer Science, Tat Chee Avenue, Kowloon, Hong Kong., corresponding author), Weimin Zheng (0000-0002-4450-5428, Department of Computer Science and Technology, Peking, PR China., corresponding author)
Year2004
Volume30
Issue1
Pages75-93
Publication date2004-03-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueComputational Linguistics (JOURNAL)
Journal identifiersISSN: 0891-2017 • E-ISSN: 1530-9312
PublisherAssociation for Computational Linguistics (PUBLISHER • US)
DOI10.1162/089120104773633394
OpenAlexW2154756108
LanguageEN
Citations received3
References cited14

We are interested in the problem of word extraction from Chinese text collections. We define a word to be a meaningful string composed of several Chinese characters. For example, ‘percent’, and, ‘more and more’, are not recognized as traditional Chinese words from the viewpoint of some people. However, in our work, they are words because they are very widely used and have specific meanings. We start with the viewpoint that a word is a distinguished linguistic entity that can be used in many different language environments. We consider the characters that are directly before a string (predecessors) and the characters that are directly after a string (successors) as important factors for determining the independence of the string. We call such characters accessors of the string, consider the number of distinct predecessors and successors of a string in a large corpus (TREC 5 and TREC 6 documents), and use them as the measurement of the context independency of a string from the rest of the sentences in the document. Our experiments confirm our hypothesis and show that this simple rule gives quite good results for Chinese word extraction and is comparable to, and for long words outperforms, other iterative methods

Chinese characters · Context (archaeology · Independence (probability theory · Linguistics · Natural language processing · Simple (philosophy · String (physics · Variety (cybernetics · Word (group theory · Advanced Text Analysis Techniques · Computer Science · History · Mathematics · Natural Language Processing Techniques · Topic Modeling · Artificial Intelligence

Tools

Open DOISci-HubOpen Access