org.apache.lucene.analysis.cn.smart

## Class SmartChineseAnalyzer

- java.lang.Object
- - org.apache.lucene.analysis.Analyzer
  - - org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer

-     public class SmartChineseAnalyzer
      extends Analyzer

  SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

  Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

  This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

  The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!

  WARNING: The status of the analyzers/smartcn analysis.cn.smart package is experimental. The APIs and file formats introduced here might change in the future and will not be supported anymore in such a case.

- - ### Field Summary

    - ### Fields inherited from class org.apache.lucene.analysis.Analyzer

      `overridesTokenStreamMethod`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `SmartChineseAnalyzer()` Deprecated. Use `SmartChineseAnalyzer(Version)` instead
    `SmartChineseAnalyzer(boolean useDefaultStopWords)` Deprecated. Use `SmartChineseAnalyzer(Version, boolean)` instead
    `SmartChineseAnalyzer(java.util.Set stopWords)` Deprecated. Use `SmartChineseAnalyzer(Version, Set)` instead
    `SmartChineseAnalyzer(Version matchVersion)` Create a new SmartChineseAnalyzer, using the default stopword list.
    `SmartChineseAnalyzer(Version matchVersion, boolean useDefaultStopWords)` Create a new SmartChineseAnalyzer, optionally using the default stopword list.
    `SmartChineseAnalyzer(Version matchVersion, java.util.Set stopWords)` Create a new SmartChineseAnalyzer, using the provided `Set` of stopwords.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `TokenStream` | `reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method.
    `TokenStream` | `tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

    - ### Methods inherited from class org.apache.lucene.analysis.Analyzer

      `close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setOverridesTokenStreamMethod, setPreviousTokenStream`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### SmartChineseAnalyzer

          public SmartChineseAnalyzer()

      Deprecated. Use `SmartChineseAnalyzer(Version)` instead
      Create a new SmartChineseAnalyzer, using the default stopword list.

    - #### SmartChineseAnalyzer

          public SmartChineseAnalyzer(Version matchVersion)

      Create a new SmartChineseAnalyzer, using the default stopword list.

    - #### SmartChineseAnalyzer

          public SmartChineseAnalyzer(boolean useDefaultStopWords)

      Deprecated. Use `SmartChineseAnalyzer(Version, boolean)` instead

      Create a new SmartChineseAnalyzer, optionally using the default stopword list.

      The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

      Parameters:
        `useDefaultStopWords` - true to use the default stopword list.

    - #### SmartChineseAnalyzer

          public SmartChineseAnalyzer(Version matchVersion,
                                      boolean useDefaultStopWords)

      Create a new SmartChineseAnalyzer, optionally using the default stopword list.

      The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

      Parameters:
        `useDefaultStopWords` - true to use the default stopword list.

    - #### SmartChineseAnalyzer

          public SmartChineseAnalyzer(java.util.Set stopWords)

      Deprecated. Use `SmartChineseAnalyzer(Version, Set)` instead

      Create a new SmartChineseAnalyzer, using the provided `Set` of stopwords.

      Note: the set should include punctuation, unless you want to index punctuation!

      Parameters:
        `stopWords` - `Set` of stopwords to use.

    - #### SmartChineseAnalyzer

          public SmartChineseAnalyzer(Version matchVersion,
                                      java.util.Set stopWords)

      Create a new SmartChineseAnalyzer, using the provided `Set` of stopwords.

      Note: the set should include punctuation, unless you want to index punctuation!

      Parameters:
        `stopWords` - `Set` of stopwords to use.

  - ### Method Detail

    - #### tokenStream

          public TokenStream tokenStream(java.lang.String fieldName,
                                         java.io.Reader reader)

      Description copied from class: `Analyzer`
      Creates a TokenStream which tokenizes all the text in the provided Reader. Must be able to handle null field name for backward compatibility.
      Specified by:
        `tokenStream` in class `Analyzer`

    - #### reusableTokenStream

          public TokenStream reusableTokenStream(java.lang.String fieldName,
                                                 java.io.Reader reader)
                                          throws java.io.IOException

      Description copied from class: `Analyzer`
      Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method. Callers that do not need to use more than one TokenStream at the same time from this analyzer should use this method for better performance.
      Overrides:
        `reusableTokenStream` in class `Analyzer`
      Throws:
        `java.io.IOException`

Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.