Lucene++ - a full-featured, c++ search engine
API Documentation

- Main Page
- Related Pages
- Namespaces
- Data Structures
- Files

- Data Structures
- Class Hierarchy
- Data Fields

All Data Structures Namespaces Files Functions Variables Typedefs Enumerations Enumerator Macros Pages
Public Member Functions | Static Public Member Functions
Lucene::LetterTokenizer Class Reference

A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined UnicodeUtil::isAlpha(c) predicate. More...

`#include <LetterTokenizer.h>`

Inheritance diagram for Lucene::LetterTokenizer:

## Public Member Functions

| LetterTokenizer (ReaderPtr input)
| Construct a new LetterTokenizer. More...
| LetterTokenizer (AttributeSourcePtr source, ReaderPtr input)
| Construct a new LetterTokenizer using a given AttributeSource. More...
| LetterTokenizer (AttributeFactoryPtr factory, ReaderPtr input)
| Construct a new LetterTokenizer using a given AttributeFactory. More...
virtual | ~LetterTokenizer ()
virtual String | getClassName ()
boost::shared_ptr < LetterTokenizer > | shared_from_this ()
virtual bool | isTokenChar (wchar_t c)
| Collects only characters which satisfy UnicodeUtil::isAlpha(c). More...
Public Member Functions inherited from Lucene::CharTokenizer
| CharTokenizer (ReaderPtr input)
| CharTokenizer (AttributeSourcePtr source, ReaderPtr input)
| CharTokenizer (AttributeFactoryPtr factory, ReaderPtr input)
virtual | ~CharTokenizer ()
boost::shared_ptr< CharTokenizer > | shared_from_this ()
virtual bool | incrementToken ()
| Consumers (ie., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Attributes with the attributes of the next token. More...
virtual void | end ()
| This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature. More...
virtual void | reset (ReaderPtr input)
| Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer. More...
Public Member Functions inherited from Lucene::Tokenizer
virtual | ~Tokenizer ()
boost::shared_ptr< Tokenizer > | shared_from_this ()
virtual void | close ()
| By default, closes the input Reader. More...
virtual int32_t | correctOffset (int32_t currentOff)
| Return the corrected offset. If input is a CharStream subclass this method calls CharStream#correctOffset, else returns currentOff. More...
Public Member Functions inherited from Lucene::TokenStream
virtual | ~TokenStream ()
boost::shared_ptr< TokenStream > | shared_from_this ()
virtual void | reset ()

Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()). More...

Public Member Functions inherited from Lucene::AttributeSource
| AttributeSource ()
| An AttributeSource using the default attribute factory DefaultAttributeFactory. More...
| AttributeSource (AttributeSourcePtr input)
| An AttributeSource that uses the same attributes as the supplied one. More...
| AttributeSource (AttributeFactoryPtr factory)
| An AttributeSource using the supplied AttributeFactory for creating new Attribute instances. More...
virtual | ~AttributeSource ()
boost::shared_ptr < AttributeSource > | shared_from_this ()
AttributeFactoryPtr | getAttributeFactory ()
| returns the used AttributeFactory. More...
template<class ATTR >
boost::shared_ptr< ATTR > | addAttribute ()
| This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned. More...
void | addAttribute (const String &className, AttributePtr attrImpl)
| Adds a custom Attribute instance. More...
bool | hasAttributes ()
| Returns true if this AttributeSource has any attributes. More...
template<class ATTR >
bool | hasAttribute ()
| Returns true, if this AttributeSource contains the passed-in Attribute. More...
template<class ATTR >
boost::shared_ptr< ATTR > | getAttribute ()
| Returns the instance of the passed in Attribute contained in this AttributeSource. More...
void | clearAttributes ()
| Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each Attribute implementation. More...
AttributeSourceStatePtr | captureState ()
| Captures the state of all Attributes. The return value can be passed to restoreState to restore the state of this or another AttributeSource. More...
void | restoreState (AttributeSourceStatePtr state)

Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation). More...

virtual int32_t | hashCode ()
| Return hash code for this object. More...
virtual bool | equals (LuceneObjectPtr other)
| Return whether two objects are equal. More...
virtual String | toString ()
| Returns a string representation of the object. More...
AttributeSourcePtr | cloneAttributes ()
| Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using AttributeSource(AttributeSource)) More...
Collection< AttributePtr > | getAttributes ()
| Return a vector of attributes based on currentState. More...
Public Member Functions inherited from Lucene::LuceneObject
virtual | ~LuceneObject ()
virtual void | initialize ()
| Called directly after instantiation to create objects that depend on this object being fully constructed. More...
virtual LuceneObjectPtr | clone (LuceneObjectPtr other=LuceneObjectPtr())
| Return clone of this object. More...
virtual int32_t | compareTo (LuceneObjectPtr other)
| Compare two objects. More...
Public Member Functions inherited from Lucene::LuceneSync
virtual | ~LuceneSync ()
virtual SynchronizePtr | getSync ()
| Return this object synchronize lock. More...
virtual LuceneSignalPtr | getSignal ()
| Return this object signal. More...
virtual void | lock (int32_t timeout=0)
| Lock this object using an optional timeout. More...
virtual void | unlock ()
| Unlock this object. More...
virtual bool | holdsLock ()
| Returns true if this object is currently locked by current thread. More...
virtual void | wait (int32_t timeout=0)
| Wait for signal using an optional timeout. More...
virtual void | notifyAll ()
| Notify all threads waiting for signal. More...

## Static Public Member Functions

static String | _getClassName ()
Static Public Member Functions inherited from Lucene::CharTokenizer
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::Tokenizer
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::TokenStream
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::AttributeSource
static String | _getClassName ()

## Additional Inherited Members

Protected Member Functions inherited from Lucene::CharTokenizer
virtual wchar_t | normalize (wchar_t c)
| Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, eg., lowercase tokens. More...
Protected Attributes inherited from Lucene::CharTokenizer
int32_t | offset
int32_t | bufferIndex
int32_t | dataLen
CharArray | ioBuffer
TermAttributePtr | termAtt
OffsetAttributePtr | offsetAtt
Static Protected Attributes inherited from Lucene::CharTokenizer
static const int32_t | MAX_WORD_LEN
static const int32_t | IO_BUFFER_SIZE

## Detailed Description

A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined UnicodeUtil::isAlpha(c) predicate.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

## Constructor & Destructor Documentation

Lucene::LetterTokenizer::LetterTokenizer | ( | ReaderPtr | input) |  |

Construct a new LetterTokenizer.

Lucene::LetterTokenizer::LetterTokenizer | ( | AttributeSourcePtr | source,
|  | ReaderPtr | input
| ) |  |

Construct a new LetterTokenizer using a given AttributeSource.

Lucene::LetterTokenizer::LetterTokenizer | ( | AttributeFactoryPtr | factory,
|  | ReaderPtr | input
| ) |  |

Construct a new LetterTokenizer using a given AttributeFactory.

virtual Lucene::LetterTokenizer::~LetterTokenizer | ( | ) |  |

virtual

## Member Function Documentation

static String Lucene::LetterTokenizer::_getClassName | ( | ) |  |

inline static

virtual String Lucene::LetterTokenizer::getClassName | ( | ) |  |

inline virtual

Reimplemented from Lucene::CharTokenizer.

Reimplemented in Lucene::LowerCaseTokenizer.

virtual bool Lucene::LetterTokenizer::isTokenChar | ( | wchar_t | c) |  |

virtual

Collects only characters which satisfy UnicodeUtil::isAlpha(c).

Implements Lucene::CharTokenizer.

boost::shared_ptr< LetterTokenizer > Lucene::LetterTokenizer::shared_from_this | ( | ) |  |

inline

The documentation for this class was generated from the following file:

- LetterTokenizer.h

clucene.sourceforge.net