org.apache.lucene.analysis.query

## Class QueryAutoStopWordAnalyzer

- java.lang.Object
- - org.apache.lucene.analysis.Analyzer
  - - org.apache.lucene.analysis.query.QueryAutoStopWordAnalyzer

-     public class QueryAutoStopWordAnalyzer
      extends Analyzer

  An `Analyzer` used primarily at query time to wrap another analyzer and provide a layer of protection which prevents very common words from being passed into queries.

  For very large indexes the cost of reading TermDocs for a very common word can be high. This analyzer was created after experience with a 38 million doc index which had a term in around 50% of docs and was causing TermQueries for this term to take 2 seconds.

  Use the various "addStopWords" methods in this class to automate the identification and addition of stop words found in an already existing index.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static float` | `defaultMaxDocFreqPercent`

    - ### Fields inherited from class org.apache.lucene.analysis.Analyzer

      `overridesTokenStreamMethod`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `QueryAutoStopWordAnalyzer(Analyzer delegate)` Deprecated. Use `QueryAutoStopWordAnalyzer(Version, Analyzer)` instead
    `QueryAutoStopWordAnalyzer(Version matchVersion, Analyzer delegate)` Initializes this analyzer with the Analyzer object that actually produces the tokens

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `int` | `addStopWords(IndexReader reader)` Automatically adds stop words for all fields with terms exceeding the defaultMaxDocFreqPercent
    `int` | `addStopWords(IndexReader reader, float maxPercentDocs)` Automatically adds stop words for all fields with terms exceeding the maxDocFreqPercent
    `int` | `addStopWords(IndexReader reader, int maxDocFreq)` Automatically adds stop words for all fields with terms exceeding the maxDocFreqPercent
    `int` | `addStopWords(IndexReader reader, java.lang.String fieldName, float maxPercentDocs)` Automatically adds stop words for the given field with terms exceeding the maxPercentDocs
    `int` | `addStopWords(IndexReader reader, java.lang.String fieldName, int maxDocFreq)` Automatically adds stop words for the given field with terms exceeding the maxPercentDocs
    `Term[]` | `getStopWords()` Provides information on which stop words have been identified for all fields
    `java.lang.String[]` | `getStopWords(java.lang.String fieldName)` Provides information on which stop words have been identified for a field
    `TokenStream` | `reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method.
    `TokenStream` | `tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

    - ### Methods inherited from class org.apache.lucene.analysis.Analyzer

      `close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setOverridesTokenStreamMethod, setPreviousTokenStream`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### defaultMaxDocFreqPercent

          public static final float defaultMaxDocFreqPercent

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### QueryAutoStopWordAnalyzer

          public QueryAutoStopWordAnalyzer(Analyzer delegate)

      Deprecated. Use `QueryAutoStopWordAnalyzer(Version, Analyzer)` instead
      Initializes this analyzer with the Analyzer object that actually produces the tokens
      Parameters:
        `delegate` - The choice of `Analyzer` that is used to produce the token stream which needs filtering

    - #### QueryAutoStopWordAnalyzer

          public QueryAutoStopWordAnalyzer(Version matchVersion,
                                           Analyzer delegate)

      Initializes this analyzer with the Analyzer object that actually produces the tokens
      Parameters:
        `delegate` - The choice of `Analyzer` that is used to produce the token stream which needs filtering

  - ### Method Detail

    - #### addStopWords

          public int addStopWords(IndexReader reader)
                           throws java.io.IOException

      Automatically adds stop words for all fields with terms exceeding the defaultMaxDocFreqPercent
      Parameters:
        `reader` - The `IndexReader` which will be consulted to identify potential stop words that exceed the required document frequency
      Returns:
        The number of stop words identified.
      Throws:
        `java.io.IOException`

    - #### addStopWords

          public int addStopWords(IndexReader reader,
                                  int maxDocFreq)
                           throws java.io.IOException

      Automatically adds stop words for all fields with terms exceeding the maxDocFreqPercent
      Parameters:
        `reader` - The `IndexReader` which will be consulted to identify potential stop words that exceed the required document frequency
        `maxDocFreq` - The maximum number of index documents which can contain a term, after which the term is considered to be a stop word
      Returns:
        The number of stop words identified.
      Throws:
        `java.io.IOException`

    - #### addStopWords

          public int addStopWords(IndexReader reader,
                                  float maxPercentDocs)
                           throws java.io.IOException

      Automatically adds stop words for all fields with terms exceeding the maxDocFreqPercent
      Parameters:
        `reader` - The `IndexReader` which will be consulted to identify potential stop words that exceed the required document frequency
        `maxPercentDocs` - The maximum percentage (between 0.0 and 1.0) of index documents which contain a term, after which the word is considered to be a stop word.
      Returns:
        The number of stop words identified.
      Throws:
        `java.io.IOException`

    - #### addStopWords

          public int addStopWords(IndexReader reader,
                                  java.lang.String fieldName,
                                  float maxPercentDocs)
                           throws java.io.IOException

      Automatically adds stop words for the given field with terms exceeding the maxPercentDocs
      Parameters:
        `reader` - The `IndexReader` which will be consulted to identify potential stop words that exceed the required document frequency
        `fieldName` - The field for which stopwords will be added
        `maxPercentDocs` - The maximum percentage (between 0.0 and 1.0) of index documents which contain a term, after which the word is considered to be a stop word.
      Returns:
        The number of stop words identified.
      Throws:
        `java.io.IOException`

    - #### addStopWords

          public int addStopWords(IndexReader reader,
                                  java.lang.String fieldName,
                                  int maxDocFreq)
                           throws java.io.IOException

      Automatically adds stop words for the given field with terms exceeding the maxPercentDocs
      Parameters:
        `reader` - The `IndexReader` which will be consulted to identify potential stop words that exceed the required document frequency
        `fieldName` - The field for which stopwords will be added
        `maxDocFreq` - The maximum number of index documents which can contain a term, after which the term is considered to be a stop word.
      Returns:
        The number of stop words identified.
      Throws:
        `java.io.IOException`

    - #### tokenStream

          public TokenStream tokenStream(java.lang.String fieldName,
                                         java.io.Reader reader)

      Description copied from class: `Analyzer`
      Creates a TokenStream which tokenizes all the text in the provided Reader. Must be able to handle null field name for backward compatibility.
      Specified by:
        `tokenStream` in class `Analyzer`

    - #### reusableTokenStream

          public TokenStream reusableTokenStream(java.lang.String fieldName,
                                                 java.io.Reader reader)
                                          throws java.io.IOException

      Description copied from class: `Analyzer`
      Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method. Callers that do not need to use more than one TokenStream at the same time from this analyzer should use this method for better performance.
      Overrides:
        `reusableTokenStream` in class `Analyzer`
      Throws:
        `java.io.IOException`

    - #### getStopWords

          public java.lang.String[] getStopWords(java.lang.String fieldName)

      Provides information on which stop words have been identified for a field
      Parameters:
        `fieldName` - The field for which stop words identified in "addStopWords" method calls will be returned
      Returns:
        the stop words identified for a field

    - #### getStopWords

          public Term[] getStopWords()

      Provides information on which stop words have been identified for all fields
      Returns:
        the stop words (as terms)

Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.