Lucene++ - a full-featured, c++ search engine
API Documentation

- Main Page
- Related Pages
- Namespaces
- Data Structures
- Files

- Data Structures
- Class Hierarchy
- Data Fields

All Data Structures Namespaces Files Functions Variables Typedefs Enumerations Enumerator Macros Pages
Public Member Functions | Static Public Member Functions | Protected Member Functions
Lucene::TokenStream Class Referenceabstract

A TokenStream enumerates the sequence of tokens, either from Fields of a Document or from query text. More...

`#include <TokenStream.h>`

Inheritance diagram for Lucene::TokenStream:

## Public Member Functions

virtual | ~TokenStream ()
virtual String | getClassName ()
boost::shared_ptr< TokenStream > | shared_from_this ()
virtual bool | incrementToken ()=0
| Consumers (ie., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Attributes with the attributes of the next token. More...
virtual void | end ()
| This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature. More...
virtual void | reset ()

Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()). More...

virtual void | close ()
| Releases resources associated with this stream. More...
Public Member Functions inherited from Lucene::AttributeSource
| AttributeSource ()
| An AttributeSource using the default attribute factory DefaultAttributeFactory. More...
| AttributeSource (AttributeSourcePtr input)
| An AttributeSource that uses the same attributes as the supplied one. More...
| AttributeSource (AttributeFactoryPtr factory)
| An AttributeSource using the supplied AttributeFactory for creating new Attribute instances. More...
virtual | ~AttributeSource ()
boost::shared_ptr < AttributeSource > | shared_from_this ()
AttributeFactoryPtr | getAttributeFactory ()
| returns the used AttributeFactory. More...
template<class ATTR >
boost::shared_ptr< ATTR > | addAttribute ()
| This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned. More...
void | addAttribute (const String &className, AttributePtr attrImpl)
| Adds a custom Attribute instance. More...
bool | hasAttributes ()
| Returns true if this AttributeSource has any attributes. More...
template<class ATTR >
bool | hasAttribute ()
| Returns true, if this AttributeSource contains the passed-in Attribute. More...
template<class ATTR >
boost::shared_ptr< ATTR > | getAttribute ()
| Returns the instance of the passed in Attribute contained in this AttributeSource. More...
void | clearAttributes ()
| Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each Attribute implementation. More...
AttributeSourceStatePtr | captureState ()
| Captures the state of all Attributes. The return value can be passed to restoreState to restore the state of this or another AttributeSource. More...
void | restoreState (AttributeSourceStatePtr state)

Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation). More...

virtual int32_t | hashCode ()
| Return hash code for this object. More...
virtual bool | equals (LuceneObjectPtr other)
| Return whether two objects are equal. More...
virtual String | toString ()
| Returns a string representation of the object. More...
AttributeSourcePtr | cloneAttributes ()
| Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using AttributeSource(AttributeSource)) More...
Collection< AttributePtr > | getAttributes ()
| Return a vector of attributes based on currentState. More...
Public Member Functions inherited from Lucene::LuceneObject
virtual | ~LuceneObject ()
virtual void | initialize ()
| Called directly after instantiation to create objects that depend on this object being fully constructed. More...
virtual LuceneObjectPtr | clone (LuceneObjectPtr other=LuceneObjectPtr())
| Return clone of this object. More...
virtual int32_t | compareTo (LuceneObjectPtr other)
| Compare two objects. More...
Public Member Functions inherited from Lucene::LuceneSync
virtual | ~LuceneSync ()
virtual SynchronizePtr | getSync ()
| Return this object synchronize lock. More...
virtual LuceneSignalPtr | getSignal ()
| Return this object signal. More...
virtual void | lock (int32_t timeout=0)
| Lock this object using an optional timeout. More...
virtual void | unlock ()
| Unlock this object. More...
virtual bool | holdsLock ()
| Returns true if this object is currently locked by current thread. More...
virtual void | wait (int32_t timeout=0)
| Wait for signal using an optional timeout. More...
virtual void | notifyAll ()
| Notify all threads waiting for signal. More...

## Static Public Member Functions

static String | _getClassName ()
Static Public Member Functions inherited from Lucene::AttributeSource
static String | _getClassName ()

## Protected Member Functions

| TokenStream ()
| A TokenStream using the default attribute factory. More...
| TokenStream (AttributeSourcePtr input)
| A TokenStream that uses the same attributes as the supplied one. More...
| TokenStream (AttributeFactoryPtr factory)
| A TokenStream using the supplied AttributeFactory for creating new Attribute instances. More...
Protected Member Functions inherited from Lucene::AttributeSource
AttributePtr | getAttribute (const String &className)
| The caller must pass in a className value. This method checks if an instance of that class is already in this AttributeSource and returns it. More...
bool | hasAttribute (const String &className)
| Returns true, if this AttributeSource contains the passed-in Attribute. More...
void | computeCurrentState ()
Protected Member Functions inherited from Lucene::LuceneObject
| LuceneObject ()

## Additional Inherited Members

Protected Attributes inherited from Lucene::AttributeSource
AttributeFactoryPtr | factory
MapStringAttribute | attributes
AttributeSourceStatePtr | currentState

## Detailed Description

A TokenStream enumerates the sequence of tokens, either from Fields of a Document or from query text.

This is an abstract class; concrete subclasses are: Tokenizer, a TokenStream whose input is a Reader; and TokenFilter, a TokenStream whose input is another TokenStream.

A new TokenStream API has been introduced with Lucene 2.9. This API has moved from being Token-based to Attribute-based. While Token still exists in 2.9 as a convenience class, the preferred way to store the information of a Token is to use Attributes.

TokenStream now extends AttributeSource, which provides access to all of the token Attributes for the TokenStream. Note that only one instance per Attribute is created and reused for every token. This approach reduces object creation and allows local caching of references to the Attributes. See incrementToken() for further details.

The workflow of the new TokenStream API is as follows:

- Instantiation of TokenStream/TokenFilters which add/get attributes to/from the AttributeSource.
- The consumer calls TokenStream#reset().
- The consumer retrieves attributes from the stream and stores local references to all attributes it wants to access.
- The consumer calls incrementToken() until it returns false consuming the attributes after each call.
- The consumer calls end() so that any end-of-stream operations can be performed.
- The consumer calls close() to release any resource when finished using the TokenStream.

To make sure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in incrementToken().

Sometimes it is desirable to capture a current state of a TokenStream, eg., for buffering purposes (see CachingTokenFilter, TeeSinkTokenFilter). For this use case AttributeSource#captureState and AttributeSource#restoreState can be used.

## Constructor & Destructor Documentation

Lucene::TokenStream::TokenStream | ( | ) |  |

protected

A TokenStream using the default attribute factory.

Lucene::TokenStream::TokenStream | ( | AttributeSourcePtr | input) |  |

protected

A TokenStream that uses the same attributes as the supplied one.

Lucene::TokenStream::TokenStream | ( | AttributeFactoryPtr | factory) |  |

protected

A TokenStream using the supplied AttributeFactory for creating new Attribute instances.

virtual Lucene::TokenStream::~TokenStream | ( | ) |  |

virtual

## Member Function Documentation

static String Lucene::TokenStream::_getClassName | ( | ) |  |

inline static

virtual void Lucene::TokenStream::close | ( | ) |  |

virtual

Releases resources associated with this stream.

Reimplemented in Lucene::Tokenizer, and Lucene::TokenFilter.

virtual void Lucene::TokenStream::end | ( | ) |  |

virtual

This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Reimplemented in Lucene::SinkTokenStream, Lucene::StandardTokenizer, Lucene::TeeSinkTokenFilter, Lucene::KeywordTokenizer, Lucene::CharTokenizer, Lucene::TokenFilter, and Lucene::CachingTokenFilter.

virtual String Lucene::TokenStream::getClassName | ( | ) |  |

inline virtual

Reimplemented from Lucene::AttributeSource.

Reimplemented in Lucene::SinkTokenStream, Lucene::NumericTokenStream, Lucene::TeeSinkTokenFilter, Lucene::StandardTokenizer, Lucene::ASCIIFoldingFilter, Lucene::Tokenizer, Lucene::PorterStemFilter, Lucene::LowerCaseTokenizer, Lucene::StopFilter, Lucene::LetterTokenizer, Lucene::WhitespaceTokenizer, Lucene::ISOLatin1AccentFilter, Lucene::TokenFilter, Lucene::CachingTokenFilter, Lucene::KeywordTokenizer, Lucene::CharTokenizer, Lucene::LengthFilter, Lucene::StandardFilter, and Lucene::LowerCaseFilter.

virtual bool Lucene::TokenStream::incrementToken | ( | ) |  |

pure virtual

Consumers (ie., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Attributes with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use captureState to create a copy of the current attribute state.

This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to addAttribute(Class) and getAttribute(Class), references to all Attributes that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in incrementToken().

Returns
  false for end of stream; true otherwise

Implemented in Lucene::SinkTokenStream, Lucene::NumericTokenStream, Lucene::TeeSinkTokenFilter, Lucene::StandardTokenizer, Lucene::ASCIIFoldingFilter, Lucene::StopFilter, Lucene::PorterStemFilter, Lucene::KeywordTokenizer, Lucene::CharTokenizer, Lucene::StandardFilter, Lucene::ISOLatin1AccentFilter, Lucene::CachingTokenFilter, Lucene::LengthFilter, and Lucene::LowerCaseFilter.

virtual void Lucene::TokenStream::reset | ( | ) |  |

virtual

Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()).

Reimplemented in Lucene::SinkTokenStream, Lucene::NumericTokenStream, Lucene::TokenFilter, Lucene::KeywordTokenizer, and Lucene::CachingTokenFilter.

boost::shared_ptr< TokenStream > Lucene::TokenStream::shared_from_this | ( | ) |  |

inline

The documentation for this class was generated from the following file:

- TokenStream.h

clucene.sourceforge.net