org.apache.lucene.analysis.cn

## Class ChineseTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.cn.ChineseTokenizer

-     public final class ChineseTokenizer
      extends org.apache.lucene.analysis.Tokenizer

  Tokenize Chinese text as individual chinese characters.

  The difference between ChineseTokenizer and CJKTokenizer is that they have different token parsing logic.

  For example, if the Chinese text "C1C2C3C4" is to be indexed:

  - The tokens returned from ChineseTokenizer are C1, C2, C3, C4.
  - The tokens returned from the CJKTokenizer are C1C2, C2C3, C3C4.

  Therefore the index created by CJKTokenizer is much larger.

  The problem is that when searching for C1, C1C2, C1C3, C4C2, C1C2C3 ... the ChineseTokenizer works, but the CJKTokenizer will not work.

  Version:
    1.0

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

      `org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State`

  - ### Field Summary

    - ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

      `input`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, java.io.Reader in)`
    `ChineseTokenizer(org.apache.lucene.util.AttributeSource source, java.io.Reader in)`
    `ChineseTokenizer(java.io.Reader in)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `end()`
    `boolean` | `incrementToken()`
    `void` | `reset()`
    `void` | `reset(java.io.Reader input)`

    - ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

      `close, correctOffset`

    - ### Methods inherited from class org.apache.lucene.analysis.TokenStream

      `getOnlyUseNewAPI, next, next, setOnlyUseNewAPI`

    - ### Methods inherited from class org.apache.lucene.util.AttributeSource

      `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### ChineseTokenizer

          public ChineseTokenizer(java.io.Reader in)

    - #### ChineseTokenizer

          public ChineseTokenizer(org.apache.lucene.util.AttributeSource source,
                                  java.io.Reader in)

    - #### ChineseTokenizer

          public ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
                                  java.io.Reader in)

  - ### Method Detail

    - #### incrementToken

          public boolean incrementToken()
                                 throws java.io.IOException

      Overrides:
        `incrementToken` in class `org.apache.lucene.analysis.TokenStream`
      Throws:
        `java.io.IOException`

    - #### end

          public final void end()

      Overrides:
        `end` in class `org.apache.lucene.analysis.TokenStream`

    - #### reset

          public void reset()
                     throws java.io.IOException

      Overrides:
        `reset` in class `org.apache.lucene.analysis.TokenStream`
      Throws:
        `java.io.IOException`

    - #### reset

          public void reset(java.io.Reader input)
                     throws java.io.IOException

      Overrides:
        `reset` in class `org.apache.lucene.analysis.Tokenizer`
      Throws:
        `java.io.IOException`

Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.