## org.apache.lucene.analysis.cn
Class ChineseTokenizer

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.Tokenizer
                  org.apache.lucene.analysis.cn.ChineseTokenizer

    public final class ChineseTokenizer

extends org.apache.lucene.analysis.Tokenizer

Tokenize Chinese text as individual chinese characters.

The difference between ChineseTokenizer and CJKTokenizer is that they have different token parsing logic.

For example, if the Chinese text "C1C2C3C4" is to be indexed:

- The tokens returned from ChineseTokenizer are C1, C2, C3, C4.
- The tokens returned from the CJKTokenizer are C1C2, C2C3, C3C4.

Therefore the index created by CJKTokenizer is much larger.

The problem is that when searching for C1, C1C2, C1C3, C4C2, C1C2C3 ... the ChineseTokenizer works, but the CJKTokenizer will not work.

Version:
  1.0

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State`

Field Summary

Fields inherited from class org.apache.lucene.analysis.Tokenizer
`input`

Constructor Summary
`ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, java.io.Reader in)`
`ChineseTokenizer(org.apache.lucene.util.AttributeSource source, java.io.Reader in)`
`ChineseTokenizer(java.io.Reader in)`

Method Summary
`void` `end()`
`boolean` `incrementToken()`
`void` `reset()`
`void` `reset(java.io.Reader input)`

Methods inherited from class org.apache.lucene.analysis.Tokenizer
`close, correctOffset`

Methods inherited from class org.apache.lucene.analysis.TokenStream
`getOnlyUseNewAPI, next, next, setOnlyUseNewAPI`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Constructor Detail

### ChineseTokenizer

    public ChineseTokenizer(java.io.Reader in)

### ChineseTokenizer

    public ChineseTokenizer(org.apache.lucene.util.AttributeSource source,
                            java.io.Reader in)

### ChineseTokenizer

    public ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
                            java.io.Reader in)

Method Detail

### incrementToken

    public boolean incrementToken()
                           throws java.io.IOException

  Overrides:
    `incrementToken` in class `org.apache.lucene.analysis.TokenStream`
  Throws:
    `java.io.IOException`

### end

    public final void end()

  Overrides:
    `end` in class `org.apache.lucene.analysis.TokenStream`

### reset

    public void reset()
               throws java.io.IOException

  Overrides:
    `reset` in class `org.apache.lucene.analysis.TokenStream`
  Throws:
    `java.io.IOException`

### reset

    public void reset(java.io.Reader input)
               throws java.io.IOException

  Overrides:
    `reset` in class `org.apache.lucene.analysis.Tokenizer`
  Throws:
    `java.io.IOException`

Copyright © 2000-2011 Apache Software Foundation. All Rights Reserved.