## Uses of Class
org.apache.lucene.analysis.TokenStream

Packages that use TokenStream
org.apache.lucene.analysis: API and code to convert text into indexable/searchable tokens.
org.apache.lucene.analysis.ar: Analyzer for Arabic.
org.apache.lucene.analysis.br: Analyzer for Brazilian Portuguese.
org.apache.lucene.analysis.cjk: Analyzer for Chinese, Japanese, and Korean, which indexes bigrams (overlapping groups of two adjacent Han characters).
org.apache.lucene.analysis.cn: Analyzer for Chinese, which indexes unigrams (individual chinese characters).
org.apache.lucene.analysis.cn.smart: Analyzer for Simplified Chinese, which indexes words.
org.apache.lucene.analysis.compound: A filter that decomposes compound words you find in many Germanic languages into the word parts.
org.apache.lucene.analysis.cz: Analyzer for Czech.
org.apache.lucene.analysis.de: Analyzer for German.
org.apache.lucene.analysis.el: Analyzer for Greek.
org.apache.lucene.analysis.fa: Analyzer for Persian.
org.apache.lucene.analysis.fr: Analyzer for French.
org.apache.lucene.analysis.miscellaneous: Miscellaneous TokenStreams
org.apache.lucene.analysis.ngram: Character n-gram tokenizers and filters.
org.apache.lucene.analysis.nl: Analyzer for Dutch.
org.apache.lucene.analysis.payloads: Provides various convenience classes for creating payloads on Tokens.
org.apache.lucene.analysis.position: Filter for assigning position increments.
org.apache.lucene.analysis.query: Automatically filter high-frequency stopwords.
org.apache.lucene.analysis.reverse: Filter to reverse token text.
org.apache.lucene.analysis.ru: Analyzer for Russian.
org.apache.lucene.analysis.shingle: Word n-gram filters
org.apache.lucene.analysis.sinks: Implementations of the SinkTokenizer that might be useful.
org.apache.lucene.analysis.snowball: `TokenFilter` and `Analyzer` implementations that use Snowball stemmers.
org.apache.lucene.analysis.standard: A fast grammar-based tokenizer constructed with JFlex.
org.apache.lucene.analysis.th: Analyzer for Thai.
org.apache.lucene.collation: `CollationKeyFilter` and `ICUCollationKeyFilter` convert each token into its binary `CollationKey` using the provided `Collator`, and then encode the `CollationKey` as a String using `IndexableBinaryStringTools`, to allow it to be stored as an index term.
org.apache.lucene.document: The logical representation of a `Document` for indexing and searching.
org.apache.lucene.index.memory: High-performance single-document main memory Apache Lucene fulltext search index.
org.apache.lucene.search.highlight: The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages.
org.apache.lucene.wikipedia.analysis: Tokenizer that is aware of Wikipedia syntax.

Uses of TokenStream in org.apache.lucene.analysis

Subclasses of TokenStream in org.apache.lucene.analysis
`class` `ASCIIFoldingFilter` This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists.
`class` `CachingTokenFilter` This class can be used if the token attributes of a TokenStream are intended to be consumed more than once.
`class` `CharTokenizer` An abstract base class for simple, character-oriented tokenizers.
`class` `ISOLatin1AccentFilter` Deprecated. in favor of `ASCIIFoldingFilter` which covers a superset of Latin 1. This class will be removed in Lucene 3.0.
`class` `KeywordTokenizer` Emits the entire input as a single token.
`class` `LengthFilter` Removes words that are too long or too short from the stream.
`class` `LetterTokenizer` A LetterTokenizer is a tokenizer that divides text at non-letters.
`class` `LowerCaseFilter` Normalizes token text to lower case.
`class` `LowerCaseTokenizer` LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together.
`class` `NumericTokenStream` Expert: This class provides a `TokenStream` for indexing numeric values that can be used by `NumericRangeQuery` or `NumericRangeFilter`.
`class` `PorterStemFilter` Transforms the token stream as per the Porter stemming algorithm.
`class` `SinkTokenizer` Deprecated. Use `TeeSinkTokenFilter` instead
`class` `StopFilter` Removes stop words from a token stream.
`class` `TeeSinkTokenFilter` This TokenFilter provides the ability to set aside attribute states that have already been analyzed.
`static class` `TeeSinkTokenFilter.SinkTokenStream`
`class` `TeeTokenFilter` Deprecated. Use `TeeSinkTokenFilter` instead
`class` `TokenFilter` A TokenFilter is a TokenStream whose input is another TokenStream.
`class` `Tokenizer` A Tokenizer is a TokenStream whose input is a Reader.
`class` `WhitespaceTokenizer` A WhitespaceTokenizer is a tokenizer that divides text at whitespace.

Fields in org.apache.lucene.analysis declared as TokenStream
`protected TokenStream` `TokenFilter.input` The source of tokens for this filter.

Methods in org.apache.lucene.analysis that return TokenStream
`TokenStream` `WhitespaceAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `StopAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `SimpleAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `PerFieldAnalyzerWrapper.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `KeywordAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `Analyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method.
`TokenStream` `WhitespaceAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `StopAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Filters LowerCaseTokenizer with StopFilter.
`TokenStream` `SimpleAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `PerFieldAnalyzerWrapper.tokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `KeywordAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`
`abstract TokenStream` `Analyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis with parameters of type TokenStream
`ASCIIFoldingFilter(TokenStream input)`
`CachingTokenFilter(TokenStream input)`
`ISOLatin1AccentFilter(TokenStream input)` Deprecated.
`LengthFilter(TokenStream in, int min, int max)` Build a filter that removes words that are too long or too short from the text.
`LowerCaseFilter(TokenStream in)`
`PorterStemFilter(TokenStream in)`
`StopFilter(boolean enablePositionIncrements, TokenStream in, java.util.Set stopWords)` Constructs a filter which removes words from the input TokenStream that are named in the Set.
`StopFilter(boolean enablePositionIncrements, TokenStream input, java.util.Set stopWords, boolean ignoreCase)` Construct a token stream filtering the given input.
`StopFilter(boolean enablePositionIncrements, TokenStream input, java.lang.String[] stopWords)` Deprecated. Use `StopFilter.StopFilter(boolean, TokenStream, Set)` instead.
`StopFilter(boolean enablePositionIncrements, TokenStream in, java.lang.String[] stopWords, boolean ignoreCase)` Deprecated. Use `StopFilter.StopFilter(boolean, TokenStream, Set, boolean)` instead.
`StopFilter(TokenStream in, java.util.Set stopWords)` Deprecated. Use `StopFilter.StopFilter(boolean, TokenStream, Set)` instead
`StopFilter(TokenStream input, java.util.Set stopWords, boolean ignoreCase)` Deprecated. Use `StopFilter.StopFilter(boolean, TokenStream, Set, boolean)` instead
`StopFilter(TokenStream input, java.lang.String[] stopWords)` Deprecated. Use `StopFilter.StopFilter(boolean, TokenStream, String[])` instead
`StopFilter(TokenStream in, java.lang.String[] stopWords, boolean ignoreCase)` Deprecated. Use `StopFilter.StopFilter(boolean, TokenStream, String[], boolean)` instead
`TeeSinkTokenFilter(TokenStream input)` Instantiates a new TeeSinkTokenFilter.
`TeeTokenFilter(TokenStream input, SinkTokenizer sink)` Deprecated.
`TokenFilter(TokenStream input)` Construct a token stream filtering the given input.

Uses of TokenStream in org.apache.lucene.analysis.ar

Subclasses of TokenStream in org.apache.lucene.analysis.ar
`class` `ArabicLetterTokenizer` Tokenizer that breaks text into runs of letters and diacritics.
`class` `ArabicNormalizationFilter` A `TokenFilter` that applies `ArabicNormalizer` to normalize the orthography.
`class` `ArabicStemFilter` A `TokenFilter` that applies `ArabicStemmer` to stem Arabic words..

Methods in org.apache.lucene.analysis.ar that return TokenStream
`TokenStream` `ArabicAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `ArabicAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.ar with parameters of type TokenStream
`ArabicNormalizationFilter(TokenStream input)`
`ArabicStemFilter(TokenStream input)`

Uses of TokenStream in org.apache.lucene.analysis.br

Subclasses of TokenStream in org.apache.lucene.analysis.br
`class` `BrazilianStemFilter` A `TokenFilter` that applies `BrazilianStemmer`.

Methods in org.apache.lucene.analysis.br that return TokenStream
`TokenStream` `BrazilianAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `BrazilianAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.br with parameters of type TokenStream
`BrazilianStemFilter(TokenStream in)`
`BrazilianStemFilter(TokenStream in, java.util.Set exclusiontable)`

Uses of TokenStream in org.apache.lucene.analysis.cjk

Subclasses of TokenStream in org.apache.lucene.analysis.cjk
`class` `CJKTokenizer` CJKTokenizer is designed for Chinese, Japanese, and Korean languages.

Methods in org.apache.lucene.analysis.cjk that return TokenStream
`TokenStream` `CJKAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `CJKAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Uses of TokenStream in org.apache.lucene.analysis.cn

Subclasses of TokenStream in org.apache.lucene.analysis.cn
`class` `ChineseFilter` A `TokenFilter` with a stop word table.
`class` `ChineseTokenizer` Tokenize Chinese text as individual chinese characters.

Methods in org.apache.lucene.analysis.cn that return TokenStream
`TokenStream` `ChineseAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `ChineseAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.cn with parameters of type TokenStream
`ChineseFilter(TokenStream in)`

Uses of TokenStream in org.apache.lucene.analysis.cn.smart

Subclasses of TokenStream in org.apache.lucene.analysis.cn.smart
`class` `SentenceTokenizer` Tokenizes input text into sentences.
`class` `WordTokenFilter` A `TokenFilter` that breaks sentences into words.

Methods in org.apache.lucene.analysis.cn.smart that return TokenStream
`TokenStream` `SmartChineseAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `SmartChineseAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`

Constructors in org.apache.lucene.analysis.cn.smart with parameters of type TokenStream
`WordTokenFilter(TokenStream in)` Construct a new WordTokenizer.

Uses of TokenStream in org.apache.lucene.analysis.compound

Subclasses of TokenStream in org.apache.lucene.analysis.compound
`class` `CompoundWordTokenFilterBase` Base class for decomposition token filters.
`class` `DictionaryCompoundWordTokenFilter` A `TokenFilter` that decomposes compound words found in many Germanic languages.
`class` `HyphenationCompoundWordTokenFilter` A `TokenFilter` that decomposes compound words found in many Germanic languages.

Constructors in org.apache.lucene.analysis.compound with parameters of type TokenStream
`CompoundWordTokenFilterBase(TokenStream input, java.util.Set dictionary)`
`CompoundWordTokenFilterBase(TokenStream input, java.util.Set dictionary, boolean onlyLongestMatch)`
`CompoundWordTokenFilterBase(TokenStream input, java.util.Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`CompoundWordTokenFilterBase(TokenStream input, java.lang.String[] dictionary)`
`CompoundWordTokenFilterBase(TokenStream input, java.lang.String[] dictionary, boolean onlyLongestMatch)`
`CompoundWordTokenFilterBase(TokenStream input, java.lang.String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`DictionaryCompoundWordTokenFilter(TokenStream input, java.util.Set dictionary)`
`DictionaryCompoundWordTokenFilter(TokenStream input, java.util.Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`DictionaryCompoundWordTokenFilter(TokenStream input, java.lang.String[] dictionary)`
`DictionaryCompoundWordTokenFilter(TokenStream input, java.lang.String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.util.Set dictionary)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.util.Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.lang.String[] dictionary)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.lang.String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`

Uses of TokenStream in org.apache.lucene.analysis.cz

Methods in org.apache.lucene.analysis.cz that return TokenStream
`TokenStream` `CzechAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `CzechAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Uses of TokenStream in org.apache.lucene.analysis.de

Subclasses of TokenStream in org.apache.lucene.analysis.de
`class` `GermanStemFilter` A `TokenFilter` that stems German words.

Methods in org.apache.lucene.analysis.de that return TokenStream
`TokenStream` `GermanAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `GermanAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.de with parameters of type TokenStream
`GermanStemFilter(TokenStream in)`
`GermanStemFilter(TokenStream in, java.util.Set exclusionSet)` Builds a GermanStemFilter that uses an exclusion table.

Uses of TokenStream in org.apache.lucene.analysis.el

Subclasses of TokenStream in org.apache.lucene.analysis.el
`class` `GreekLowerCaseFilter` Normalizes token text to lower case, analyzing given ("greek") charset.

Methods in org.apache.lucene.analysis.el that return TokenStream
`TokenStream` `GreekAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `GreekAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.el with parameters of type TokenStream
`GreekLowerCaseFilter(TokenStream in)`
`GreekLowerCaseFilter(TokenStream in, char[] charset)` Deprecated. Use `GreekLowerCaseFilter.GreekLowerCaseFilter(TokenStream)` instead.

Uses of TokenStream in org.apache.lucene.analysis.fa

Subclasses of TokenStream in org.apache.lucene.analysis.fa
`class` `PersianNormalizationFilter` A `TokenFilter` that applies `PersianNormalizer` to normalize the orthography.

Methods in org.apache.lucene.analysis.fa that return TokenStream
`TokenStream` `PersianAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `PersianAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.fa with parameters of type TokenStream
`PersianNormalizationFilter(TokenStream input)`

Uses of TokenStream in org.apache.lucene.analysis.fr

Subclasses of TokenStream in org.apache.lucene.analysis.fr
`class` `ElisionFilter` Removes elisions from a `TokenStream`.
`class` `FrenchStemFilter` A `TokenFilter` that stems french words.

Methods in org.apache.lucene.analysis.fr that return TokenStream
`TokenStream` `FrenchAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `FrenchAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.fr with parameters of type TokenStream
`ElisionFilter(TokenStream input)` Constructs an elision filter with standard stop words
`ElisionFilter(TokenStream input, java.util.Set articles)` Constructs an elision filter with a Set of stop words
`ElisionFilter(TokenStream input, java.lang.String[] articles)` Constructs an elision filter with an array of stop words
`FrenchStemFilter(TokenStream in)`
`FrenchStemFilter(TokenStream in, java.util.Set exclusiontable)`

Uses of TokenStream in org.apache.lucene.analysis.miscellaneous

Subclasses of TokenStream in org.apache.lucene.analysis.miscellaneous
`class` `EmptyTokenStream` An always exhausted token stream.
`class` `PrefixAndSuffixAwareTokenFilter` Links two `PrefixAwareTokenFilter`.
`class` `PrefixAwareTokenFilter` Joins two token streams and leaves the last token of the first stream available to be used when updating the token values in the second stream based on that token.
`class` `SingleTokenTokenStream` A `TokenStream` containing a single token.

Methods in org.apache.lucene.analysis.miscellaneous that return TokenStream
`TokenStream` `PrefixAwareTokenFilter.getPrefix()`
`TokenStream` `PrefixAwareTokenFilter.getSuffix()`

Methods in org.apache.lucene.analysis.miscellaneous with parameters of type TokenStream
`void` `PrefixAwareTokenFilter.setPrefix(TokenStream prefix)`
`void` `PrefixAwareTokenFilter.setSuffix(TokenStream suffix)`

Constructors in org.apache.lucene.analysis.miscellaneous with parameters of type TokenStream
`PrefixAndSuffixAwareTokenFilter(TokenStream prefix, TokenStream input, TokenStream suffix)`
`PrefixAwareTokenFilter(TokenStream prefix, TokenStream suffix)`

Uses of TokenStream in org.apache.lucene.analysis.ngram

Subclasses of TokenStream in org.apache.lucene.analysis.ngram
`class` `EdgeNGramTokenFilter` Tokenizes the given token into n-grams of given size(s).
`class` `EdgeNGramTokenizer` Tokenizes the input from an edge into n-grams of given size(s).
`class` `NGramTokenFilter` Tokenizes the input into n-grams of the given size(s).
`class` `NGramTokenizer` Tokenizes the input into n-grams of the given size(s).

Constructors in org.apache.lucene.analysis.ngram with parameters of type TokenStream
`EdgeNGramTokenFilter(TokenStream input)`
`EdgeNGramTokenFilter(TokenStream input, EdgeNGramTokenFilter.Side side, int minGram, int maxGram)` Creates EdgeNGramTokenFilter that can generate n-grams in the sizes of the given range
`EdgeNGramTokenFilter(TokenStream input, java.lang.String sideLabel, int minGram, int maxGram)` Creates EdgeNGramTokenFilter that can generate n-grams in the sizes of the given range
`NGramTokenFilter(TokenStream input)` Creates NGramTokenFilter with default min and max n-grams.
`NGramTokenFilter(TokenStream input, int minGram, int maxGram)` Creates NGramTokenFilter with given min and max n-grams.

Uses of TokenStream in org.apache.lucene.analysis.nl

Subclasses of TokenStream in org.apache.lucene.analysis.nl
`class` `DutchStemFilter` A `TokenFilter` that stems Dutch words.

Methods in org.apache.lucene.analysis.nl that return TokenStream
`TokenStream` `DutchAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `DutchAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.nl with parameters of type TokenStream
`DutchStemFilter(TokenStream _in)`
`DutchStemFilter(TokenStream _in, java.util.Set exclusiontable)` Builds a DutchStemFilter that uses an exclusion table.
`DutchStemFilter(TokenStream _in, java.util.Set exclusiontable, java.util.Map stemdictionary)`

Uses of TokenStream in org.apache.lucene.analysis.payloads

Subclasses of TokenStream in org.apache.lucene.analysis.payloads
`class` `DelimitedPayloadTokenFilter` Characters before the delimiter are the "token", those after are the payload.
`class` `NumericPayloadTokenFilter` Assigns a payload to a token based on the `Token.type()`
`class` `TokenOffsetPayloadTokenFilter` Adds the `Token.setStartOffset(int)` and `Token.setEndOffset(int)` First 4 bytes are the start
`class` `TypeAsPayloadTokenFilter` Makes the `Token.type()` a payload.

Constructors in org.apache.lucene.analysis.payloads with parameters of type TokenStream
`DelimitedPayloadTokenFilter(TokenStream input)` Construct a token stream filtering the given input.
`DelimitedPayloadTokenFilter(TokenStream input, char delimiter, PayloadEncoder encoder)`
`NumericPayloadTokenFilter(TokenStream input, float payload, java.lang.String typeMatch)`
`TokenOffsetPayloadTokenFilter(TokenStream input)`
`TypeAsPayloadTokenFilter(TokenStream input)`

Uses of TokenStream in org.apache.lucene.analysis.position

Subclasses of TokenStream in org.apache.lucene.analysis.position
`class` `PositionFilter` Set the positionIncrement of all tokens to the "positionIncrement", except the first return token which retains its original positionIncrement value.

Constructors in org.apache.lucene.analysis.position with parameters of type TokenStream
`PositionFilter(TokenStream input)` Constructs a PositionFilter that assigns a position increment of zero to all but the first token from the given input stream.
`PositionFilter(TokenStream input, int positionIncrement)` Constructs a PositionFilter that assigns the given position increment to all but the first token from the given input stream.

Uses of TokenStream in org.apache.lucene.analysis.query

Methods in org.apache.lucene.analysis.query that return TokenStream
`TokenStream` `QueryAutoStopWordAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `QueryAutoStopWordAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`

Uses of TokenStream in org.apache.lucene.analysis.reverse

Subclasses of TokenStream in org.apache.lucene.analysis.reverse
`class` `ReverseStringFilter` Reverse token string, for example "country" => "yrtnuoc".

Constructors in org.apache.lucene.analysis.reverse with parameters of type TokenStream
`ReverseStringFilter(TokenStream in)` Create a new ReverseStringFilter that reverses all tokens in the supplied `TokenStream`.
`ReverseStringFilter(TokenStream in, char marker)` Create a new ReverseStringFilter that reverses and marks all tokens in the supplied `TokenStream`.

Uses of TokenStream in org.apache.lucene.analysis.ru

Subclasses of TokenStream in org.apache.lucene.analysis.ru
`class` `RussianLetterTokenizer` A RussianLetterTokenizer is a `Tokenizer` that extends `LetterTokenizer` by additionally looking up letters in a given "russian charset".
`class` `RussianLowerCaseFilter` Normalizes token text to lower case, analyzing given ("russian") charset.
`class` `RussianStemFilter` A `TokenFilter` that stems Russian words.

Methods in org.apache.lucene.analysis.ru that return TokenStream
`TokenStream` `RussianAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `TokenStream` which tokenizes all the text in the provided `Reader`.
`TokenStream` `RussianAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a `TokenStream` which tokenizes all the text in the provided `Reader`.

Constructors in org.apache.lucene.analysis.ru with parameters of type TokenStream
`RussianLowerCaseFilter(TokenStream in)`
`RussianLowerCaseFilter(TokenStream in, char[] charset)` Deprecated. Use `RussianLowerCaseFilter.RussianLowerCaseFilter(TokenStream)` instead.
`RussianStemFilter(TokenStream in)`
`RussianStemFilter(TokenStream in, char[] charset)` Deprecated. Use `RussianStemFilter.RussianStemFilter(TokenStream)` instead.

Uses of TokenStream in org.apache.lucene.analysis.shingle

Subclasses of TokenStream in org.apache.lucene.analysis.shingle
`class` `ShingleFilter` A ShingleFilter constructs shingles (token n-grams) from a token stream.
`class` `ShingleMatrixFilter` A ShingleMatrixFilter constructs shingles (token n-grams) from a token stream.

Methods in org.apache.lucene.analysis.shingle that return TokenStream
`TokenStream` `ShingleAnalyzerWrapper.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `ShingleAnalyzerWrapper.tokenStream(java.lang.String fieldName, java.io.Reader reader)`

Constructors in org.apache.lucene.analysis.shingle with parameters of type TokenStream
`ShingleFilter(TokenStream input)` Construct a ShingleFilter with default shingle size.
`ShingleFilter(TokenStream input, int maxShingleSize)` Constructs a ShingleFilter with the specified single size from the `TokenStream` `input`
`ShingleFilter(TokenStream input, java.lang.String tokenType)` Construct a ShingleFilter with the specified token type for shingle tokens.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize)` Creates a shingle filter using default settings.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, java.lang.Character spacerCharacter)` Creates a shingle filter using default settings.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, java.lang.Character spacerCharacter, boolean ignoringSinglePrefixOrSuffixShingle)` Creates a shingle filter using the default `ShingleMatrixFilter.TokenSettingsCodec`.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, java.lang.Character spacerCharacter, boolean ignoringSinglePrefixOrSuffixShingle, ShingleMatrixFilter.TokenSettingsCodec settingsCodec)` Creates a shingle filter with ad hoc parameter settings.

Uses of TokenStream in org.apache.lucene.analysis.sinks

Subclasses of TokenStream in org.apache.lucene.analysis.sinks
`class` `DateRecognizerSinkTokenizer` Deprecated. Use `DateRecognizerSinkFilter` and `TeeSinkTokenFilter` instead.
`class` `TokenRangeSinkTokenizer` Deprecated. Use `TokenRangeSinkFilter` and `TeeSinkTokenFilter` instead.
`class` `TokenTypeSinkTokenizer` Deprecated. Use `TokenTypeSinkFilter` and `TeeSinkTokenFilter` instead.

Uses of TokenStream in org.apache.lucene.analysis.snowball

Subclasses of TokenStream in org.apache.lucene.analysis.snowball
`class` `SnowballFilter` A filter that stems words using a Snowball-generated stemmer.

Methods in org.apache.lucene.analysis.snowball that return TokenStream
`TokenStream` `SnowballAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Returns a (possibly reused) `StandardTokenizer` filtered by a `StandardFilter`, a `LowerCaseFilter`, a `StopFilter`, and a `SnowballFilter`
`TokenStream` `SnowballAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Constructs a `StandardTokenizer` filtered by a `StandardFilter`, a `LowerCaseFilter`, a `StopFilter`, and a `SnowballFilter`

Constructors in org.apache.lucene.analysis.snowball with parameters of type TokenStream
`SnowballFilter(TokenStream input, SnowballProgram stemmer)`
`SnowballFilter(TokenStream in, java.lang.String name)` Construct the named stemming filter.

Uses of TokenStream in org.apache.lucene.analysis.standard

Subclasses of TokenStream in org.apache.lucene.analysis.standard
`class` `StandardFilter` Normalizes tokens extracted with `StandardTokenizer`.
`class` `StandardTokenizer` A grammar-based tokenizer constructed with JFlex

Methods in org.apache.lucene.analysis.standard that return TokenStream
`TokenStream` `StandardAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)` Deprecated. Use `StandardAnalyzer.tokenStream(java.lang.String, java.io.Reader)` instead
`TokenStream` `StandardAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Constructs a `StandardTokenizer` filtered by a `StandardFilter`, a `LowerCaseFilter` and a `StopFilter`.

Constructors in org.apache.lucene.analysis.standard with parameters of type TokenStream
`StandardFilter(TokenStream in)` Construct filtering in.

Uses of TokenStream in org.apache.lucene.analysis.th

Subclasses of TokenStream in org.apache.lucene.analysis.th
`class` `ThaiWordFilter` `TokenFilter` that use `BreakIterator` to break each Token that is Thai into separate Token(s) for each Thai word.

Methods in org.apache.lucene.analysis.th that return TokenStream
`TokenStream` `ThaiAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `ThaiAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`

Constructors in org.apache.lucene.analysis.th with parameters of type TokenStream
`ThaiWordFilter(TokenStream input)`

Uses of TokenStream in org.apache.lucene.collation

Subclasses of TokenStream in org.apache.lucene.collation
`class` `CollationKeyFilter` Converts each token into its `CollationKey`, and then encodes the CollationKey with `IndexableBinaryStringTools`, to allow it to be stored as an index term.
`class` `ICUCollationKeyFilter` Converts each token into its `com.ibm.icu.text.CollationKey`, and then encodes the CollationKey with `IndexableBinaryStringTools`, to allow it to be stored as an index term.

Methods in org.apache.lucene.collation that return TokenStream
`TokenStream` `ICUCollationKeyAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `CollationKeyAnalyzer.reusableTokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `ICUCollationKeyAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`
`TokenStream` `CollationKeyAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)`

Constructors in org.apache.lucene.collation with parameters of type TokenStream
`CollationKeyFilter(TokenStream input, java.text.Collator collator)`
`ICUCollationKeyFilter(TokenStream input, Collator collator)`

Uses of TokenStream in org.apache.lucene.document

Fields in org.apache.lucene.document declared as TokenStream
`protected TokenStream` `AbstractField.tokenStream`

Methods in org.apache.lucene.document that return TokenStream
`TokenStream` `NumericField.tokenStreamValue()` Returns a `NumericTokenStream` for indexing the numeric value.
`TokenStream` `Field.tokenStreamValue()` The TokesStream for this field to be used when indexing, or null.
`TokenStream` `Fieldable.tokenStreamValue()` The TokenStream for this field to be used when indexing, or null.

Methods in org.apache.lucene.document with parameters of type TokenStream
`void` `Field.setTokenStream(TokenStream tokenStream)` Expert: sets the token stream to be used for indexing and causes isIndexed() and isTokenized() to return true.
`void` `Field.setValue(TokenStream value)` Deprecated. use `Field.setTokenStream(org.apache.lucene.analysis.TokenStream)`

Constructors in org.apache.lucene.document with parameters of type TokenStream
`Field(java.lang.String name, TokenStream tokenStream)` Create a tokenized and indexed field that is not stored.
`Field(java.lang.String name, TokenStream tokenStream, Field.TermVector termVector)` Create a tokenized and indexed field that is not stored, optionally with storing term vectors.

Uses of TokenStream in org.apache.lucene.index.memory

Subclasses of TokenStream in org.apache.lucene.index.memory
`class` `SynonymTokenFilter` Injects additional tokens for synonyms of token terms fetched from the underlying child stream; the child stream must deliver lowercase tokens for synonyms to be found.

Methods in org.apache.lucene.index.memory that return TokenStream
`TokenStream` `MemoryIndex.keywordTokenStream(java.util.Collection keywords)` Convenience method; Creates and returns a token stream that generates a token for each keyword in the given collection, "as is", without any transforming text analysis.
`TokenStream` `PatternAnalyzer.tokenStream(java.lang.String fieldName, java.io.Reader reader)` Creates a token stream that tokenizes all the text in the given Reader; This implementation forwards to `tokenStream(String, String)` and is less efficient than `tokenStream(String, String)`.
`TokenStream` `PatternAnalyzer.tokenStream(java.lang.String fieldName, java.lang.String text)` Creates a token stream that tokenizes the given string into token terms (aka words).

Methods in org.apache.lucene.index.memory with parameters of type TokenStream
`void` `MemoryIndex.addField(java.lang.String fieldName, TokenStream stream)` Equivalent to `addField(fieldName, stream, 1.0f)`.
`void` `MemoryIndex.addField(java.lang.String fieldName, TokenStream stream, float boost)` Iterates over the given token stream and adds the resulting terms to the index; Equivalent to adding a tokenized, indexed, termVectorStored, unstored, Lucene `Field`.

Constructors in org.apache.lucene.index.memory with parameters of type TokenStream
`SynonymTokenFilter(TokenStream input, SynonymMap synonyms, int maxSynonyms)` Creates an instance for the given underlying stream and synonym table.

Uses of TokenStream in org.apache.lucene.search.highlight

Methods in org.apache.lucene.search.highlight that return TokenStream
`static TokenStream` `TokenSources.getAnyTokenStream(IndexReader reader, int docId, java.lang.String field, Analyzer analyzer)` A convenience method that tries a number of approaches to getting a token stream.
`static TokenStream` `TokenSources.getAnyTokenStream(IndexReader reader, int docId, java.lang.String field, Document doc, Analyzer analyzer)` A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to using the passed in `Document` to retrieve the TokenStream.
`TokenStream` `WeightedSpanTermExtractor.getTokenStream()`
`static TokenStream` `TokenSources.getTokenStream(Document doc, java.lang.String field, Analyzer analyzer)`
`static TokenStream` `TokenSources.getTokenStream(IndexReader reader, int docId, java.lang.String field)`
`static TokenStream` `TokenSources.getTokenStream(IndexReader reader, int docId, java.lang.String field, Analyzer analyzer)`
`static TokenStream` `TokenSources.getTokenStream(java.lang.String field, java.lang.String contents, Analyzer analyzer)`
`static TokenStream` `TokenSources.getTokenStream(TermPositionVector tpv)`
`static TokenStream` `TokenSources.getTokenStream(TermPositionVector tpv, boolean tokenPositionsGuaranteedContiguous)` Low level api.
`TokenStream` `QueryTermScorer.init(TokenStream tokenStream)`
`TokenStream` `QueryScorer.init(TokenStream tokenStream)`
`TokenStream` `Scorer.init(TokenStream tokenStream)` Called to init the Scorer with a `TokenStream`.

Methods in org.apache.lucene.search.highlight with parameters of type TokenStream
`java.lang.String` `Highlighter.getBestFragment(TokenStream tokenStream, java.lang.String text)` Highlights chosen terms in a text, extracting the most relevant section.
`java.lang.String[]` `Highlighter.getBestFragments(TokenStream tokenStream, java.lang.String text, int maxNumFragments)` Highlights chosen terms in a text, extracting the most relevant sections.
`java.lang.String` `Highlighter.getBestFragments(TokenStream tokenStream, java.lang.String text, int maxNumFragments, java.lang.String separator)` Highlights terms in the text , extracting the most relevant sections and concatenating the chosen fragments with a separator (typically "...").
`TextFragment[]` `Highlighter.getBestTextFragments(TokenStream tokenStream, java.lang.String text, boolean mergeContiguousFragments, int maxNumFragments)` Low level api to get the most relevant (formatted) sections of the document.
`java.util.Map` `WeightedSpanTermExtractor.getWeightedSpanTerms(Query query, TokenStream tokenStream)` Creates a Map of `WeightedSpanTerms` from the given `Query` and `TokenStream`.
`java.util.Map` `WeightedSpanTermExtractor.getWeightedSpanTerms(Query query, TokenStream tokenStream, java.lang.String fieldName)` Creates a Map of `WeightedSpanTerms` from the given `Query` and `TokenStream`.
`java.util.Map` `WeightedSpanTermExtractor.getWeightedSpanTermsWithScores(Query query, TokenStream tokenStream, java.lang.String fieldName, IndexReader reader)` Creates a Map of `WeightedSpanTerms` from the given `Query` and `TokenStream`.
`TokenStream` `QueryTermScorer.init(TokenStream tokenStream)`
`TokenStream` `QueryScorer.init(TokenStream tokenStream)`
`TokenStream` `Scorer.init(TokenStream tokenStream)` Called to init the Scorer with a `TokenStream`.
`void` `NullFragmenter.start(java.lang.String s, TokenStream tokenStream)`
`void` `SimpleFragmenter.start(java.lang.String originalText, TokenStream stream)`
`void` `Fragmenter.start(java.lang.String originalText, TokenStream tokenStream)` Initializes the Fragmenter.
`void` `SimpleSpanFragmenter.start(java.lang.String originalText, TokenStream tokenStream)`

Constructors in org.apache.lucene.search.highlight with parameters of type TokenStream
`TokenGroup(TokenStream tokenStream)`

Uses of TokenStream in org.apache.lucene.wikipedia.analysis

Subclasses of TokenStream in org.apache.lucene.wikipedia.analysis
`class` `WikipediaTokenizer` Extension of StandardTokenizer that is aware of Wikipedia syntax.

Copyright © 2000-2011 Apache Software Foundation. All Rights Reserved.