Lucene++ - a full-featured, c++ search engine
API Documentation

Loading...
Searching...
No Matches
Public Member Functions | Static Public Member Functions
Lucene::LowerCaseTokenizer Class Reference

LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation. More...

`#include <LowerCaseTokenizer.h>`

Inheritance diagram for Lucene::LowerCaseTokenizer:

## Public Member Functions

| LowerCaseTokenizer (const ReaderPtr &input)
| Construct a new LowerCaseTokenizer.
| LowerCaseTokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
| Construct a new LowerCaseTokenizer using a given `AttributeSource`.
| LowerCaseTokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
| Construct a new LowerCaseTokenizer using a given `AttributeFactory`.
virtual | ~LowerCaseTokenizer ()
virtual String | getClassName ()
boost::shared_ptr< LowerCaseTokenizer > | shared_from_this ()
virtual wchar_t | normalize (wchar_t c)
| Converts char to lower case CharFolder::toLower<wchar_t>.
Public Member Functions inherited from Lucene::LetterTokenizer
| LetterTokenizer (const ReaderPtr &input)
| Construct a new LetterTokenizer.
| LetterTokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
| Construct a new LetterTokenizer using a given `AttributeSource`.
| LetterTokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
| Construct a new LetterTokenizer using a given `AttributeFactory`.
virtual | ~LetterTokenizer ()
boost::shared_ptr< LetterTokenizer > | shared_from_this ()
virtual bool | isTokenChar (wchar_t c)
| Collects only characters which satisfy UnicodeUtil::isAlpha(c).
Public Member Functions inherited from Lucene::CharTokenizer
| CharTokenizer (const ReaderPtr &input)
| CharTokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
| CharTokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
virtual | ~CharTokenizer ()
boost::shared_ptr< CharTokenizer > | shared_from_this ()
virtual bool | incrementToken ()
| Consumers (ie., `IndexWriter`) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate `Attribute`s with the attributes of the next token.
virtual void | end ()
| This method is called by the consumer after the last token has been consumed, after `incrementToken()` returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.
virtual void | reset (const ReaderPtr &input)
| Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.
Public Member Functions inherited from Lucene::Tokenizer
virtual | ~Tokenizer ()
boost::shared_ptr< Tokenizer > | shared_from_this ()
virtual void | close ()
| By default, closes the input Reader.
virtual int32_t | correctOffset (int32_t currentOff)
| Return the corrected offset. If `input` is a `CharStream` subclass this method calls `CharStream#correctOffset`, else returns currentOff.
virtual void | reset ()

Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. `reset()` is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement `reset()`. Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after `reset()`).

Public Member Functions inherited from Lucene::TokenStream
virtual | ~TokenStream ()
boost::shared_ptr< TokenStream > | shared_from_this ()
Public Member Functions inherited from Lucene::AttributeSource
| AttributeSource ()
| An AttributeSource using the default attribute factory `DefaultAttributeFactory`.
| AttributeSource (const AttributeSourcePtr &input)
| An AttributeSource that uses the same attributes as the supplied one.
| AttributeSource (const AttributeFactoryPtr &factory)
| An AttributeSource using the supplied `AttributeFactory` for creating new `Attribute` instances.
virtual | ~AttributeSource ()
boost::shared_ptr< AttributeSource > | shared_from_this ()
AttributeFactoryPtr | getAttributeFactory ()
| returns the used AttributeFactory.
template<class ATTR >
boost::shared_ptr< ATTR > | addAttribute ()
| This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned.
void | addAttribute (const String &className, const AttributePtr &attrImpl)
| Adds a custom Attribute instance.
bool | hasAttributes ()
| Returns true if this AttributeSource has any attributes.
template<class ATTR >
bool | hasAttribute ()
| Returns true, if this AttributeSource contains the passed-in Attribute.
template<class ATTR >
boost::shared_ptr< ATTR > | getAttribute ()
| Returns the instance of the passed in Attribute contained in this AttributeSource.
void | clearAttributes ()
| Resets all Attributes in this AttributeSource by calling `AttributeImpl#clear()` on each Attribute implementation.
AttributeSourceStatePtr | captureState ()
| Captures the state of all Attributes. The return value can be passed to `restoreState` to restore the state of this or another AttributeSource.
void | restoreState (const AttributeSourceStatePtr &state)

Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation).

virtual int32_t | hashCode ()
| Return hash code for this object.
virtual bool | equals (const LuceneObjectPtr &other)
| Return whether two objects are equal.
virtual String | toString ()
| Returns a string representation of the object.
AttributeSourcePtr | cloneAttributes ()
| Performs a clone of all `AttributeImpl` instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using `AttributeSource(AttributeSource)`)
Collection< AttributePtr > | getAttributes ()
| Return a vector of attributes based on currentState.
Public Member Functions inherited from Lucene::LuceneObject
virtual | ~LuceneObject ()
virtual void | initialize ()
| Called directly after instantiation to create objects that depend on this object being fully constructed.
virtual LuceneObjectPtr | clone (const LuceneObjectPtr &other=LuceneObjectPtr())
| Return clone of this object.
virtual int32_t | compareTo (const LuceneObjectPtr &other)
| Compare two objects.
Public Member Functions inherited from Lucene::LuceneSync
virtual | ~LuceneSync ()
virtual SynchronizePtr | getSync ()
| Return this object synchronize lock.
virtual LuceneSignalPtr | getSignal ()
| Return this object signal.
virtual void | lock (int32_t timeout=0)
| Lock this object using an optional timeout.
virtual void | unlock ()
| Unlock this object.
virtual bool | holdsLock ()
| Returns true if this object is currently locked by current thread.
virtual void | wait (int32_t timeout=0)
| Wait for signal using an optional timeout.
virtual void | notifyAll ()
| Notify all threads waiting for signal.

## Static Public Member Functions

static String | _getClassName ()
Static Public Member Functions inherited from Lucene::LetterTokenizer
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::CharTokenizer
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::Tokenizer
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::TokenStream
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::AttributeSource
static String | _getClassName ()

## Additional Inherited Members

Protected Member Functions inherited from Lucene::Tokenizer
| Tokenizer ()
| Construct a tokenizer with null input.
| Tokenizer (const ReaderPtr &input)
| Construct a token stream processing the given input.
| Tokenizer (const AttributeFactoryPtr &factory)
| Construct a tokenizer with null input using the given AttributeFactory.
| Tokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
| Construct a token stream processing the given input using the given AttributeFactory.
| Tokenizer (const AttributeSourcePtr &source)
| Construct a token stream processing the given input using the given AttributeSource.
| Tokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
| Construct a token stream processing the given input using the given AttributeSource.
Protected Member Functions inherited from Lucene::TokenStream
| TokenStream ()
| A TokenStream using the default attribute factory.
| TokenStream (const AttributeSourcePtr &input)
| A TokenStream that uses the same attributes as the supplied one.
| TokenStream (const AttributeFactoryPtr &factory)
| A TokenStream using the supplied AttributeFactory for creating new `Attribute` instances.
Protected Member Functions inherited from Lucene::AttributeSource
AttributePtr | getAttribute (const String &className)
| The caller must pass in a className value. This method checks if an instance of that class is already in this AttributeSource and returns it.
bool | hasAttribute (const String &className)
| Returns true, if this AttributeSource contains the passed-in Attribute.
void | computeCurrentState ()
Protected Member Functions inherited from Lucene::LuceneObject
| LuceneObject ()
Protected Attributes inherited from Lucene::CharTokenizer
int32_t | offset
int32_t | bufferIndex
int32_t | dataLen
CharArray | ioBuffer
TermAttributePtr | termAtt
OffsetAttributePtr | offsetAtt
Protected Attributes inherited from Lucene::Tokenizer
ReaderPtr | input
| The text source for this Tokenizer.
CharStreamPtr | charStream
Protected Attributes inherited from Lucene::AttributeSource
AttributeFactoryPtr | factory
MapStringAttribute | attributes
AttributeSourceStatePtr | currentState
Protected Attributes inherited from Lucene::LuceneSync
SynchronizePtr | objectLock
LuceneSignalPtr | objectSignal
Static Protected Attributes inherited from Lucene::CharTokenizer
static const int32_t | MAX_WORD_LEN
static const int32_t | IO_BUFFER_SIZE

## Detailed Description

LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

## Constructor & Destructor Documentation

## ◆ LowerCaseTokenizer() [1/3]

Lucene::LowerCaseTokenizer::LowerCaseTokenizer | ( | const ReaderPtr & | input | ) |

Construct a new LowerCaseTokenizer.

## ◆ LowerCaseTokenizer() [2/3]

Lucene::LowerCaseTokenizer::LowerCaseTokenizer | ( | const AttributeSourcePtr & | source,
|  | const ReaderPtr & | input
| ) |  |

Construct a new LowerCaseTokenizer using a given `AttributeSource`.

## ◆ LowerCaseTokenizer() [3/3]

Lucene::LowerCaseTokenizer::LowerCaseTokenizer | ( | const AttributeFactoryPtr & | factory,
|  | const ReaderPtr & | input
| ) |  |

Construct a new LowerCaseTokenizer using a given `AttributeFactory`.

## ◆ ~LowerCaseTokenizer()

virtual Lucene::LowerCaseTokenizer::~LowerCaseTokenizer | ( |  | ) |

virtual

## Member Function Documentation

## ◆ _getClassName()

static String Lucene::LowerCaseTokenizer::_getClassName | ( |  | ) |

inline static

## ◆ getClassName()

virtual String Lucene::LowerCaseTokenizer::getClassName | ( |  | ) |

inline virtual

Reimplemented from Lucene::LetterTokenizer.

## ◆ normalize()

virtual wchar_t Lucene::LowerCaseTokenizer::normalize | ( | wchar_t | c | ) |

virtual

Converts char to lower case CharFolder::toLower<wchar_t>.

Reimplemented from Lucene::CharTokenizer.

## ◆ shared_from_this()

boost::shared_ptr< LowerCaseTokenizer > Lucene::LowerCaseTokenizer::shared_from_this | ( |  | ) |

inline

The documentation for this class was generated from the following file:

- LowerCaseTokenizer.h

clucene.sourceforge.net