## org.apache.lucene.analysis
Class Tokenizer

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.Tokenizer

Direct Known Subclasses:
  CharTokenizer, KeywordTokenizer, SinkTokenizer, StandardTokenizer

    public abstract class Tokenizer

extends TokenStream

A Tokenizer is a TokenStream whose input is a Reader.

This is an abstract class; subclasses must override `TokenStream.incrementToken()`

NOTE: Subclasses overriding `TokenStream.incrementToken()` must call `AttributeSource.clearAttributes()` before setting attributes. Subclasses overriding `TokenStream.next(Token)` must call `Token.clear()` before setting Token attributes.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`AttributeSource.AttributeFactory, AttributeSource.State`

Field Summary
`protected java.io.Reader` `input` The text source for this Tokenizer.

Constructor Summary
`protected` `Tokenizer()` Construct a tokenizer with null input.
`protected` `Tokenizer(AttributeSource.AttributeFactory factory)` Construct a tokenizer with null input using the given AttributeFactory.
`protected` `Tokenizer(AttributeSource.AttributeFactory factory, java.io.Reader input)` Construct a token stream processing the given input using the given AttributeFactory.
`protected` `Tokenizer(AttributeSource source)` Construct a token stream processing the given input using the given AttributeSource.
`protected` `Tokenizer(AttributeSource source, java.io.Reader input)` Construct a token stream processing the given input using the given AttributeSource.
`protected` `Tokenizer(java.io.Reader input)` Construct a token stream processing the given input.

Method Summary
`void` `close()` By default, closes the input Reader.
`protected int` `correctOffset(int currentOff)` Return the corrected offset.
`void` `reset(java.io.Reader input)` Expert: Reset the tokenizer to a new reader.

Methods inherited from class org.apache.lucene.analysis.TokenStream
`end, getOnlyUseNewAPI, incrementToken, next, next, reset, setOnlyUseNewAPI`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Field Detail

### input

    protected java.io.Reader input

  The text source for this Tokenizer.

Constructor Detail

### Tokenizer

    protected Tokenizer()

  Construct a tokenizer with null input.

### Tokenizer

    protected Tokenizer(java.io.Reader input)

  Construct a token stream processing the given input.

### Tokenizer

    protected Tokenizer(AttributeSource.AttributeFactory factory)

  Construct a tokenizer with null input using the given AttributeFactory.

### Tokenizer

    protected Tokenizer(AttributeSource.AttributeFactory factory,
                        java.io.Reader input)

  Construct a token stream processing the given input using the given AttributeFactory.

### Tokenizer

    protected Tokenizer(AttributeSource source)

  Construct a token stream processing the given input using the given AttributeSource.

### Tokenizer

    protected Tokenizer(AttributeSource source,
                        java.io.Reader input)

  Construct a token stream processing the given input using the given AttributeSource.

Method Detail

### close

    public void close()
               throws java.io.IOException

  By default, closes the input Reader.

  Overrides:
    `close` in class `TokenStream`
  Throws:
    `java.io.IOException`

### correctOffset

    protected final int correctOffset(int currentOff)

  Return the corrected offset. If `input` is a `CharStream` subclass this method calls `CharStream.correctOffset(int)`, else returns `currentOff`.

  Parameters:
    `currentOff` - offset as seen in the output
  Returns:
    corrected offset based on the input
  See Also:
    `CharStream.correctOffset(int)`

### reset

    public void reset(java.io.Reader input)
               throws java.io.IOException

  Expert: Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

  Throws:
    `java.io.IOException`

Copyright © 2000-2011 Apache Software Foundation. All Rights Reserved.