Lucene++ - a full-featured, c++ search engine
API Documentation

Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Member Functions | Protected Attributes
Lucene::StandardTokenizer Class Reference

A grammar-based tokenizer. More...

`#include <StandardTokenizer.h>`

Inheritance diagram for Lucene::StandardTokenizer:

## Public Member Functions

| StandardTokenizer (LuceneVersion::Version matchVersion, const ReaderPtr &input)
| Creates a new instance of the StandardTokenizer. Attaches the input to the newly created scanner. More...
| StandardTokenizer (LuceneVersion::Version matchVersion, const AttributeSourcePtr &source, const ReaderPtr &input)
| Creates a new StandardTokenizer with a given AttributeSource. More...
| StandardTokenizer (LuceneVersion::Version matchVersion, const AttributeFactoryPtr &factory, const ReaderPtr &input)
| Creates a new StandardTokenizer with a given AttributeSource.AttributeFactory. More...
virtual | ~StandardTokenizer ()
virtual String | getClassName ()
boost::shared_ptr< StandardTokenizer > | shared_from_this ()
void | setMaxTokenLength (int32_t length)
| Set the max allowed token length. Any token longer than this is skipped. More...
int32_t | getMaxTokenLength ()
virtual bool | incrementToken ()
virtual void | end ()
| This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature. More...
virtual void | reset (const ReaderPtr &input)
| Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer. More...
bool | isReplaceInvalidAcronym ()
void | setReplaceInvalidAcronym (bool replaceInvalidAcronym)
Public Member Functions inherited from Lucene::Tokenizer
virtual | ~Tokenizer ()
boost::shared_ptr< Tokenizer > | shared_from_this ()
virtual void | close ()
| By default, closes the input Reader. More...
virtual int32_t | correctOffset (int32_t currentOff)
| Return the corrected offset. If input is a CharStream subclass this method calls CharStream#correctOffset, else returns currentOff. More...
virtual void | reset ()

Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()). More...

Public Member Functions inherited from Lucene::TokenStream
virtual | ~TokenStream ()
boost::shared_ptr< TokenStream > | shared_from_this ()
Public Member Functions inherited from Lucene::AttributeSource
| AttributeSource ()
| An AttributeSource using the default attribute factory DefaultAttributeFactory. More...
| AttributeSource (const AttributeSourcePtr &input)
| An AttributeSource that uses the same attributes as the supplied one. More...
| AttributeSource (const AttributeFactoryPtr &factory)
| An AttributeSource using the supplied AttributeFactory for creating new Attribute instances. More...
virtual | ~AttributeSource ()
boost::shared_ptr< AttributeSource > | shared_from_this ()
AttributeFactoryPtr | getAttributeFactory ()
| returns the used AttributeFactory. More...
template<class ATTR >
boost::shared_ptr< ATTR > | addAttribute ()
| This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned. More...
void | addAttribute (const String &className, const AttributePtr &attrImpl)
| Adds a custom Attribute instance. More...
bool | hasAttributes ()
| Returns true if this AttributeSource has any attributes. More...
template<class ATTR >
bool | hasAttribute ()
| Returns true, if this AttributeSource contains the passed-in Attribute. More...
template<class ATTR >
boost::shared_ptr< ATTR > | getAttribute ()
| Returns the instance of the passed in Attribute contained in this AttributeSource. More...
void | clearAttributes ()
| Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each Attribute implementation. More...
AttributeSourceStatePtr | captureState ()
| Captures the state of all Attributes. The return value can be passed to restoreState to restore the state of this or another AttributeSource. More...
void | restoreState (const AttributeSourceStatePtr &state)

Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation). More...

virtual int32_t | hashCode ()
| Return hash code for this object. More...
virtual bool | equals (const LuceneObjectPtr &other)
| Return whether two objects are equal. More...
virtual String | toString ()
| Returns a string representation of the object. More...
AttributeSourcePtr | cloneAttributes ()
| Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using {}) More...
Collection< AttributePtr > | getAttributes ()
| Return a vector of attributes based on currentState. More...
Public Member Functions inherited from Lucene::LuceneObject
virtual | ~LuceneObject ()
virtual void | initialize ()
| Called directly after instantiation to create objects that depend on this object being fully constructed. More...
virtual LuceneObjectPtr | clone (const LuceneObjectPtr &other=LuceneObjectPtr())
| Return clone of this object. More...
virtual int32_t | compareTo (const LuceneObjectPtr &other)
| Compare two objects. More...
Public Member Functions inherited from Lucene::LuceneSync
virtual | ~LuceneSync ()
virtual SynchronizePtr | getSync ()
| Return this object synchronize lock. More...
virtual LuceneSignalPtr | getSignal ()
| Return this object signal. More...
virtual void | lock (int32_t timeout=0)
| Lock this object using an optional timeout. More...
virtual void | unlock ()
| Unlock this object. More...
virtual bool | holdsLock ()
| Returns true if this object is currently locked by current thread. More...
virtual void | wait (int32_t timeout=0)
| Wait for signal using an optional timeout. More...
virtual void | notifyAll ()
| Notify all threads waiting for signal. More...

## Static Public Member Functions

static String | _getClassName ()
static const Collection< String > | TOKEN_TYPES ()
| String token types that correspond to token type int constants. More...
Static Public Member Functions inherited from Lucene::Tokenizer
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::TokenStream
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::AttributeSource
static String | _getClassName ()

## Static Public Attributes

static const int32_t | ALPHANUM
static const int32_t | APOSTROPHE
static const int32_t | ACRONYM
static const int32_t | COMPANY
static const int32_t | EMAIL
static const int32_t | HOST
static const int32_t | NUM
static const int32_t | CJ
static const int32_t | ACRONYM_DEP

## Protected Member Functions

void | init (const ReaderPtr &input, LuceneVersion::Version matchVersion)
Protected Member Functions inherited from Lucene::Tokenizer
| Tokenizer ()
| Construct a tokenizer with null input. More...
| Tokenizer (const ReaderPtr &input)
| Construct a token stream processing the given input. More...
| Tokenizer (const AttributeFactoryPtr &factory)
| Construct a tokenizer with null input using the given AttributeFactory. More...
| Tokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
| Construct a token stream processing the given input using the given AttributeFactory. More...
| Tokenizer (const AttributeSourcePtr &source)
| Construct a token stream processing the given input using the given AttributeSource. More...
| Tokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
| Construct a token stream processing the given input using the given AttributeSource. More...
Protected Member Functions inherited from Lucene::TokenStream
| TokenStream ()
| A TokenStream using the default attribute factory. More...
| TokenStream (const AttributeSourcePtr &input)
| A TokenStream that uses the same attributes as the supplied one. More...
| TokenStream (const AttributeFactoryPtr &factory)
| A TokenStream using the supplied AttributeFactory for creating new Attribute instances. More...
Protected Member Functions inherited from Lucene::AttributeSource
AttributePtr | getAttribute (const String &className)
| The caller must pass in a className value. This method checks if an instance of that class is already in this AttributeSource and returns it. More...
bool | hasAttribute (const String &className)
| Returns true, if this AttributeSource contains the passed-in Attribute. More...
void | computeCurrentState ()
Protected Member Functions inherited from Lucene::LuceneObject
| LuceneObject ()

## Protected Attributes

StandardTokenizerImplPtr | scanner
| A private instance of the scanner. More...
bool | replaceInvalidAcronym
int32_t | maxTokenLength
TermAttributePtr | termAtt
OffsetAttributePtr | offsetAtt
PositionIncrementAttributePtr | posIncrAtt
TypeAttributePtr | typeAtt
Protected Attributes inherited from Lucene::Tokenizer
ReaderPtr | input
| The text source for this Tokenizer. More...
CharStreamPtr | charStream
Protected Attributes inherited from Lucene::AttributeSource
AttributeFactoryPtr | factory
MapStringAttribute | attributes
AttributeSourceStatePtr | currentState
Protected Attributes inherited from Lucene::LuceneSync
SynchronizePtr | objectLock
LuceneSignalPtr | objectSignal

## Detailed Description

A grammar-based tokenizer.

This should be a good tokenizer for most European-language documents:

- Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
- Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
- Recognizes email addresses and internet hostnames as one token.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

You must specify the required Version compatibility when creating StandardAnalyzer:

- As of 2.4, Tokens incorrectly identified as acronyms are corrected

## Constructor & Destructor Documentation

## ◆ StandardTokenizer() [1/3]

Lucene::StandardTokenizer::StandardTokenizer | ( | LuceneVersion::Version | matchVersion,
|  | const ReaderPtr & | input
| ) |  |

Creates a new instance of the StandardTokenizer. Attaches the input to the newly created scanner.

Parameters

  input | The input reader

## ◆ StandardTokenizer() [2/3]

Lucene::StandardTokenizer::StandardTokenizer | ( | LuceneVersion::Version | matchVersion,
|  | const AttributeSourcePtr & | source,
|  | const ReaderPtr & | input
| ) |  |

Creates a new StandardTokenizer with a given AttributeSource.

## ◆ StandardTokenizer() [3/3]

Lucene::StandardTokenizer::StandardTokenizer | ( | LuceneVersion::Version | matchVersion,
|  | const AttributeFactoryPtr & | factory,
|  | const ReaderPtr & | input
| ) |  |

Creates a new StandardTokenizer with a given AttributeSource.AttributeFactory.

## ◆ ~StandardTokenizer()

virtual Lucene::StandardTokenizer::~StandardTokenizer | ( |  | ) |

virtual

## Member Function Documentation

## ◆ _getClassName()

static String Lucene::StandardTokenizer::_getClassName | ( |  | ) |

inline static

## ◆ end()

virtual void Lucene::StandardTokenizer::end | ( |  | ) |

virtual

This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Reimplemented from Lucene::TokenStream.

## ◆ getClassName()

virtual String Lucene::StandardTokenizer::getClassName | ( |  | ) |

inline virtual

Reimplemented from Lucene::Tokenizer.

## ◆ getMaxTokenLength()

int32_t Lucene::StandardTokenizer::getMaxTokenLength | ( |  | ) |

See also
  setMaxTokenLength

## ◆ incrementToken()

virtual bool Lucene::StandardTokenizer::incrementToken | ( |  | ) |

virtual

See also
  TokenStream::next()

Implements Lucene::TokenStream.

## ◆ init()

void Lucene::StandardTokenizer::init | ( | const ReaderPtr & | input,
|  | LuceneVersion::Version | matchVersion
| ) |  |

protected

## ◆ isReplaceInvalidAcronym()

bool Lucene::StandardTokenizer::isReplaceInvalidAcronym | ( |  | ) |

Returns
  true if StandardTokenizer now returns these tokens as Hosts, otherwise false

Deprecated:
  Remove in 3.X and make true the only valid value

## ◆ reset()

virtual void Lucene::StandardTokenizer::reset | ( | const ReaderPtr & | input | ) |

virtual

Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

Reimplemented from Lucene::Tokenizer.

## ◆ setMaxTokenLength()

void Lucene::StandardTokenizer::setMaxTokenLength | ( | int32_t | length | ) |

Set the max allowed token length. Any token longer than this is skipped.

## ◆ setReplaceInvalidAcronym()

void Lucene::StandardTokenizer::setReplaceInvalidAcronym | ( | bool | replaceInvalidAcronym | ) |

Parameters

  replaceInvalidAcronym | Set to true to replace mischaracterized acronyms as HOST.

Deprecated:
  Remove in 3.X and make true the only valid value

## ◆ shared_from_this()

boost::shared_ptr< StandardTokenizer > Lucene::StandardTokenizer::shared_from_this | ( |  | ) |

inline

## ◆ TOKEN_TYPES()

static const Collection<String> Lucene::StandardTokenizer::TOKEN_TYPES | ( |  | ) |

static

String token types that correspond to token type int constants.

## Field Documentation

## ◆ ACRONYM

const int32_t Lucene::StandardTokenizer::ACRONYM

static

## ◆ ACRONYM_DEP

const int32_t Lucene::StandardTokenizer::ACRONYM_DEP

static

Deprecated:
  this solves a bug where HOSTs that end with '.' are identified as ACRONYMs.

## ◆ ALPHANUM

const int32_t Lucene::StandardTokenizer::ALPHANUM

static

## ◆ APOSTROPHE

const int32_t Lucene::StandardTokenizer::APOSTROPHE

static

## ◆ CJ

const int32_t Lucene::StandardTokenizer::CJ

static

## ◆ COMPANY

const int32_t Lucene::StandardTokenizer::COMPANY

static

## ◆ EMAIL

const int32_t Lucene::StandardTokenizer::EMAIL

static

## ◆ HOST

const int32_t Lucene::StandardTokenizer::HOST

static

## ◆ maxTokenLength

int32_t Lucene::StandardTokenizer::maxTokenLength

protected

## ◆ NUM

const int32_t Lucene::StandardTokenizer::NUM

static

## ◆ offsetAtt

OffsetAttributePtr Lucene::StandardTokenizer::offsetAtt

protected

## ◆ posIncrAtt

PositionIncrementAttributePtr Lucene::StandardTokenizer::posIncrAtt

protected

## ◆ replaceInvalidAcronym

bool Lucene::StandardTokenizer::replaceInvalidAcronym

protected

## ◆ scanner

StandardTokenizerImplPtr Lucene::StandardTokenizer::scanner

protected

A private instance of the scanner.

## ◆ termAtt

TermAttributePtr Lucene::StandardTokenizer::termAtt

protected

## ◆ typeAtt

TypeAttributePtr Lucene::StandardTokenizer::typeAtt

protected

The documentation for this class was generated from the following file:

- StandardTokenizer.h

clucene.sourceforge.net