Abstract: Given the lack of word delimiters in written Japanese, word segmentation is generally considered a crucial first step in processing Japanese texts. Typical Japanese segmentation algorithms rely either on a lexicon and syntactic analysis or on pre-segmented data; but these are labor-intensive, and the lexico-syntactic techniques are vulnerable to the unknown word problem. In contrast, we introduce a novel, more robust statistical method utilizing unsegmented training data. Despite its simplicity, the algorithm yields performance on long kanji sequences comparable to and sometimes surpassing that of state-of-the-art morphological analyzers over a variety of error metrics. The algorithm also outperforms another mostly-unsupervised statistical algorithm previously proposed for Chinese.

Additionally, we present a two-level annotation scheme for Japanese to incorporate multiple segmentation granularities, and introduce two novel evaluation metrics, both based on the notion of a {\em compatible bracket}, that can account for multiple granularities simultaneously.

Paper formats: ps, pdf, other

BibTeX entry:

@Article{Ando+Lee:03a,
  author =       {Rie Kubota Ando and Lillian Lee},
  title =        {Mostly-Unsupervised Statistical Segmentation of {Japanese} Kanji Sequences},
  journal =      {Journal of Natural Language Engineering},
  volume={9},
  issue={2},
  pages={127--149},
  year =         2003
}