Lucene++ - a full-featured, c++ search engine
API Documentation

- Main Page
- Related Pages
- Namespaces
- Data Structures
- Files

- Data Structures
- Class Hierarchy
- Data Fields

All Data Structures Namespaces Files Functions Variables Typedefs Enumerations Enumerator Macros Pages
Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Member Functions | Static Protected Member Functions | Protected Attributes | Static Protected Attributes
Lucene::StandardTokenizerImpl Class Reference

`#include <StandardTokenizerImpl.h>`

Inheritance diagram for Lucene::StandardTokenizerImpl:

## Public Member Functions

| StandardTokenizerImpl (ReaderPtr in)
| Creates a new scanner. More...
virtual | ~StandardTokenizerImpl ()
virtual String | getClassName ()
boost::shared_ptr < StandardTokenizerImpl > | shared_from_this ()
int32_t | yychar ()
void | reset (ReaderPtr r)
| Resets the Tokenizer to a new Reader. More...
void | getText (TokenPtr t)
| Fills Lucene token with the current token text. More...
void | getText (TermAttributePtr t)
| Fills TermAttribute with the current token text. More...
void | yyclose ()
| Closes the input stream. More...
void | yyreset (ReaderPtr reader)
| Resets the scanner to read from a new input stream. Does not close the old reader. More...
int32_t | yystate ()
| Returns the current lexical state. More...
void | yybegin (int32_t newState)
| Enters a new lexical state. More...
String | yytext ()
| Returns the text matched by the current regular expression. More...
wchar_t | yycharat (int32_t pos)
| Returns the character at position pos from the matched text. More...
int32_t | yylength ()
| Returns the length of the matched text region. More...
void | yypushback (int32_t number)
| Pushes the specified amount of characters back into the input stream. More...
int32_t | getNextToken ()
| Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O- Error occurs. More...
Public Member Functions inherited from Lucene::LuceneObject
virtual | ~LuceneObject ()
virtual void | initialize ()
| Called directly after instantiation to create objects that depend on this object being fully constructed. More...
virtual LuceneObjectPtr | clone (LuceneObjectPtr other=LuceneObjectPtr())
| Return clone of this object. More...
virtual int32_t | hashCode ()
| Return hash code for this object. More...
virtual bool | equals (LuceneObjectPtr other)
| Return whether two objects are equal. More...
virtual int32_t | compareTo (LuceneObjectPtr other)
| Compare two objects. More...
virtual String | toString ()
| Returns a string representation of the object. More...
Public Member Functions inherited from Lucene::LuceneSync
virtual | ~LuceneSync ()
virtual SynchronizePtr | getSync ()
| Return this object synchronize lock. More...
virtual LuceneSignalPtr | getSignal ()
| Return this object signal. More...
virtual void | lock (int32_t timeout=0)
| Lock this object using an optional timeout. More...
virtual void | unlock ()
| Unlock this object. More...
virtual bool | holdsLock ()
| Returns true if this object is currently locked by current thread. More...
virtual void | wait (int32_t timeout=0)
| Wait for signal using an optional timeout. More...
virtual void | notifyAll ()
| Notify all threads waiting for signal. More...

## Static Public Member Functions

static String | _getClassName ()

## Static Public Attributes

static const int32_t | YYEOF
| This character denotes the end of file. More...
static const int32_t | YYINITIAL
| Lexical states. More...

## Protected Member Functions

bool | zzRefill ()
| Refills the input buffer. More...
void | zzScanError (int32_t errorCode)
| Reports an error that occurred while scanning. More...
Protected Member Functions inherited from Lucene::LuceneObject
| LuceneObject ()

## Static Protected Member Functions

static const wchar_t * | ZZ_CMAP ()
| Translates characters to character classes. More...
static const int32_t * | ZZ_ACTION ()
| Translates DFA states to action switch labels. More...
static const int32_t * | ZZ_ROWMAP ()
| Translates a state to a row index in the transition table. More...
static const int32_t * | ZZ_TRANS ()
| The transition table of the DFA. More...
static const int32_t * | ZZ_ATTRIBUTE ()
| ZZ_ATTRIBUTE[aState] contains the attributes of state aState. More...

## Protected Attributes

ReaderPtr | zzReader
| The input device. More...
int32_t | zzState
| The current state of the DFA. More...
int32_t | zzLexicalState
| The current lexical state. More...
CharArray | zzBuffer
| This buffer contains the current text to be matched and is the source of the yytext() string. More...
int32_t | zzMarkedPos
| The text position at the last accepting state. More...
int32_t | zzPushbackPos
| The text position at the last state to be included in yytext. More...
int32_t | zzCurrentPos
| The current text position in the buffer. More...
int32_t | zzStartRead
| StartRead marks the beginning of the yytext() string in the buffer. More...
int32_t | zzEndRead
| EndRead marks the last character in the buffer, that has been read from input. More...
int32_t | yyline
| Number of newlines encountered up to the start of the matched text. More...
int32_t | _yychar
| The number of characters up to the start of the matched text. More...
int32_t | yycolumn
| The number of characters from the last newline up to the start of the matched text. More...
bool | zzAtBOL
| zzAtBOL == true if the scanner is currently at the beginning of a line More...
bool | zzAtEOF
| zzAtEOF == true if the scanner is at the EOF More...

## Static Protected Attributes

static const int32_t | ZZ_BUFFERSIZE
| Initial size of the lookahead buffer. More...
static const wchar_t | ZZ_CMAP_PACKED []
| Translates characters to character classes. More...
static const int32_t | ZZ_CMAP_LENGTH
static const int32_t | ZZ_CMAP_PACKED_LENGTH
static const wchar_t | ZZ_ACTION_PACKED_0 []
| Translates DFA states to action switch labels. More...
static const int32_t | ZZ_ACTION_LENGTH
static const int32_t | ZZ_ACTION_PACKED_LENGTH
static const wchar_t | ZZ_ROWMAP_PACKED_0 []
| Translates a state to a row index in the transition table. More...
static const int32_t | ZZ_ROWMAP_LENGTH
static const int32_t | ZZ_ROWMAP_PACKED_LENGTH
static const wchar_t | ZZ_TRANS_PACKED_0 []
| The transition table of the DFA. More...
static const int32_t | ZZ_TRANS_LENGTH
static const int32_t | ZZ_TRANS_PACKED_LENGTH
static const int32_t | ZZ_UNKNOWN_ERROR
static const int32_t | ZZ_NO_MATCH
static const int32_t | ZZ_PUSHBACK_2BIG
static const wchar_t * | ZZ_ERROR_MSG []
static const wchar_t | ZZ_ATTRIBUTE_PACKED_0 []
| ZZ_ATTRIBUTE[aState] contains the attributes of state aState. More...
static const int32_t | ZZ_ATTRIBUTE_LENGTH
static const int32_t | ZZ_ATTRIBUTE_PACKED_LENGTH

## Constructor & Destructor Documentation

Lucene::StandardTokenizerImpl::StandardTokenizerImpl | ( | ReaderPtr | in) |  |

Creates a new scanner.

Parameters

  in | the Reader to read input from.

virtual Lucene::StandardTokenizerImpl::~StandardTokenizerImpl | ( | ) |  |

virtual

## Member Function Documentation

static String Lucene::StandardTokenizerImpl::_getClassName | ( | ) |  |

inline static

virtual String Lucene::StandardTokenizerImpl::getClassName | ( | ) |  |

inline virtual

int32_t Lucene::StandardTokenizerImpl::getNextToken | ( | ) |  |

Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O- Error occurs.

void Lucene::StandardTokenizerImpl::getText | ( | TokenPtr | t) |  |

Fills Lucene token with the current token text.

void Lucene::StandardTokenizerImpl::getText | ( | TermAttributePtr | t) |  |

Fills TermAttribute with the current token text.

void Lucene::StandardTokenizerImpl::reset | ( | ReaderPtr | r) |  |

Resets the Tokenizer to a new Reader.

boost::shared_ptr< StandardTokenizerImpl > Lucene::StandardTokenizerImpl::shared_from_this | ( | ) |  |

inline

void Lucene::StandardTokenizerImpl::yybegin | ( | int32_t | newState) |  |

Enters a new lexical state.

Parameters

  newState | the new lexical state.

int32_t Lucene::StandardTokenizerImpl::yychar | ( | ) |  |

wchar_t Lucene::StandardTokenizerImpl::yycharat | ( | int32_t | pos) |  |

Returns the character at position pos from the matched text.

It is equivalent to yytext()[pos], but faster

Parameters

  pos | the position of the character to fetch. A value from 0 to yylength() - 1.

Returns
  the character at position pos.

void Lucene::StandardTokenizerImpl::yyclose | ( | ) |  |

Closes the input stream.

int32_t Lucene::StandardTokenizerImpl::yylength | ( | ) |  |

Returns the length of the matched text region.

void Lucene::StandardTokenizerImpl::yypushback | ( | int32_t | number) |  |

Pushes the specified amount of characters back into the input stream.

They will be read again by then next call of the scanning method

Parameters

  number | the number of characters to be read again. This number must not be greater than yylength()

void Lucene::StandardTokenizerImpl::yyreset | ( | ReaderPtr | reader) |  |

Resets the scanner to read from a new input stream. Does not close the old reader.

All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL.

Parameters

  reader | the new input stream.

int32_t Lucene::StandardTokenizerImpl::yystate | ( | ) |  |

Returns the current lexical state.

String Lucene::StandardTokenizerImpl::yytext | ( | ) |  |

Returns the text matched by the current regular expression.

static const int32_t* Lucene::StandardTokenizerImpl::ZZ_ACTION | ( | ) |  |

static protected

Translates DFA states to action switch labels.

static const int32_t* Lucene::StandardTokenizerImpl::ZZ_ATTRIBUTE | ( | ) |  |

static protected

ZZ_ATTRIBUTE[aState] contains the attributes of state aState.

static const wchar_t* Lucene::StandardTokenizerImpl::ZZ_CMAP | ( | ) |  |

static protected

Translates characters to character classes.

static const int32_t* Lucene::StandardTokenizerImpl::ZZ_ROWMAP | ( | ) |  |

static protected

Translates a state to a row index in the transition table.

static const int32_t* Lucene::StandardTokenizerImpl::ZZ_TRANS | ( | ) |  |

static protected

The transition table of the DFA.

bool Lucene::StandardTokenizerImpl::zzRefill | ( | ) |  |

protected

Refills the input buffer.

void Lucene::StandardTokenizerImpl::zzScanError | ( | int32_t | errorCode) |  |

protected

Reports an error that occurred while scanning.

In a well-formed scanner (no or only correct usage of yypushback(int32_t) and a match-all fallback rule) this method will only be called with things that "Can't Possibly Happen". If this method is called, something is seriously wrong.

Usual syntax/scanner level error handling should be done in error fallback rules.

Parameters

  errorCode | The code of the errormessage to display.

## Field Documentation

int32_t Lucene::StandardTokenizerImpl::_yychar

protected

The number of characters up to the start of the matched text.

int32_t Lucene::StandardTokenizerImpl::yycolumn

protected

The number of characters from the last newline up to the start of the matched text.

const int32_t Lucene::StandardTokenizerImpl::YYEOF

static

This character denotes the end of file.

const int32_t Lucene::StandardTokenizerImpl::YYINITIAL

static

Lexical states.

int32_t Lucene::StandardTokenizerImpl::yyline

protected

Number of newlines encountered up to the start of the matched text.

const int32_t Lucene::StandardTokenizerImpl::ZZ_ACTION_LENGTH

static protected

const wchar_t Lucene::StandardTokenizerImpl::ZZ_ACTION_PACKED_0[]

static protected

Translates DFA states to action switch labels.

const int32_t Lucene::StandardTokenizerImpl::ZZ_ACTION_PACKED_LENGTH

static protected

const int32_t Lucene::StandardTokenizerImpl::ZZ_ATTRIBUTE_LENGTH

static protected

const wchar_t Lucene::StandardTokenizerImpl::ZZ_ATTRIBUTE_PACKED_0[]

static protected

ZZ_ATTRIBUTE[aState] contains the attributes of state aState.

const int32_t Lucene::StandardTokenizerImpl::ZZ_ATTRIBUTE_PACKED_LENGTH

static protected

const int32_t Lucene::StandardTokenizerImpl::ZZ_BUFFERSIZE

static protected

Initial size of the lookahead buffer.

const int32_t Lucene::StandardTokenizerImpl::ZZ_CMAP_LENGTH

static protected

const wchar_t Lucene::StandardTokenizerImpl::ZZ_CMAP_PACKED[]

static protected

Translates characters to character classes.

const int32_t Lucene::StandardTokenizerImpl::ZZ_CMAP_PACKED_LENGTH

static protected

const wchar_t* Lucene::StandardTokenizerImpl::ZZ_ERROR_MSG[]

static protected

const int32_t Lucene::StandardTokenizerImpl::ZZ_NO_MATCH

static protected

const int32_t Lucene::StandardTokenizerImpl::ZZ_PUSHBACK_2BIG

static protected

const int32_t Lucene::StandardTokenizerImpl::ZZ_ROWMAP_LENGTH

static protected

const wchar_t Lucene::StandardTokenizerImpl::ZZ_ROWMAP_PACKED_0[]

static protected

Translates a state to a row index in the transition table.

const int32_t Lucene::StandardTokenizerImpl::ZZ_ROWMAP_PACKED_LENGTH

static protected

const int32_t Lucene::StandardTokenizerImpl::ZZ_TRANS_LENGTH

static protected

const wchar_t Lucene::StandardTokenizerImpl::ZZ_TRANS_PACKED_0[]

static protected

The transition table of the DFA.

const int32_t Lucene::StandardTokenizerImpl::ZZ_TRANS_PACKED_LENGTH

static protected

const int32_t Lucene::StandardTokenizerImpl::ZZ_UNKNOWN_ERROR

static protected

bool Lucene::StandardTokenizerImpl::zzAtBOL

protected

zzAtBOL == true if the scanner is currently at the beginning of a line

bool Lucene::StandardTokenizerImpl::zzAtEOF

protected

zzAtEOF == true if the scanner is at the EOF

CharArray Lucene::StandardTokenizerImpl::zzBuffer

protected

This buffer contains the current text to be matched and is the source of the yytext() string.

int32_t Lucene::StandardTokenizerImpl::zzCurrentPos

protected

The current text position in the buffer.

int32_t Lucene::StandardTokenizerImpl::zzEndRead

protected

EndRead marks the last character in the buffer, that has been read from input.

int32_t Lucene::StandardTokenizerImpl::zzLexicalState

protected

The current lexical state.

int32_t Lucene::StandardTokenizerImpl::zzMarkedPos

protected

The text position at the last accepting state.

int32_t Lucene::StandardTokenizerImpl::zzPushbackPos

protected

The text position at the last state to be included in yytext.

ReaderPtr Lucene::StandardTokenizerImpl::zzReader

protected

The input device.

int32_t Lucene::StandardTokenizerImpl::zzStartRead

protected

StartRead marks the beginning of the yytext() string in the buffer.

int32_t Lucene::StandardTokenizerImpl::zzState

protected

The current state of the DFA.

The documentation for this class was generated from the following file:

- StandardTokenizerImpl.h

clucene.sourceforge.net