Accessor Variety Criteria for Chinese Word Extraction

We are interested in the problem of word extraction from Chinese text collections. We define a word to be a meaningful string composed of several Chinese characters. For example, ‘percent’, and, ‘more and more’, are not recognized as traditional Chinese words from the viewpoint of some people. However, in our work, they are words because they are very widely used and have specific meanings. We start with the viewpoint that a word is a distinguished linguistic entity that can be used in many different language environments. We consider the characters that are directly before a string (predecessors) and the characters that are directly after a string (successors) as important factors for determining the independence of the string. We call such characters accessors of the string, consider the number of distinct predecessors and successors of a string in a large corpus (TREC 5 and TREC 6 documents), and use them as the measurement of the context independency of a string from the rest of the sentences in the document. Our experiments confirm our hypothesis and show that this simple rule gives quite good results for Chinese word extraction and is comparable to, and for long words outperforms, other iterative methods

Chinese characters · Context (archaeology · Independence (probability theory · Linguistics · Natural language processing · Simple (philosophy · String (physics · Variety (cybernetics · Word (group theory · Advanced Text Analysis Techniques · Computer Science · History · Mathematics · Natural Language Processing Techniques · Topic Modeling · Artificial Intelligence

• Word segmentation from transcriptions of child-directed speech using lexical and sub-lexical cues

  Open Access•Zébulon GORIELY, Andrew Caines et al.•Journal of Child Language•2025

  Incluir item

• Punctuation as Implicit Annotations for Chinese Word Segmentation

  Open Access•Zhongguo Li, Maosong Sun•Computational Linguistics•2009

  Incluir item

• A New Unsupervised Approach to Word Segmentation

  Open Access•Hanshi Wang, Jian Zhu et al.•Computational Linguistics•2011

  Incluir item

• A grammar of spoken Chinese.

  Yuen Ren Chao•A grammar of spoken Chinese.•1965

  Incluir item

• Using Suffix Arrays to Compute Term Frequency and Document Frequency for All Substrings in a Corpus

  Open Access•Mikio Yamamoto, Kathryn Church et al.•Computational Linguistics•2001

  Incluir item

• Automatic Verb Classification Based on Statistical Distributions of Argument Structure

  Open Access•Paola Merlo, Suzanne Stevenson•Computational Linguistics•2001

  Incluir item

• Extracting the Lowest-Frequency Words

  Open Access•Marc Weeber, Rob Vos et al.•Computational Linguistics•2000

  Incluir item

• A Compression-based Algorithm for Chinese Word Segmentation

  Open Access•William J Teahan, Yingying Wen et al.•Computational Linguistics•2000

  Incluir item

• The Computational Analysis of English

  W Nelson Francis, Roger Garside et al.•Language•1989

  Incluir item

• Human Rights and the World Response To the 1989 Crackdown in China

  Open Access•James D Seymour•China Information•1990

  Incluir item