ReverseTokenizer Class Reference

`#include <reverseTokenizer.h>`

Inheritance diagram for ReverseTokenizer:
Collaboration diagram for ReverseTokenizer:

## Public Member Functions

| ReverseTokenizer (std::istream &stream, const std::string blanks, const std::string separs)
virtual | ~ReverseTokenizer ()
virtual int | countTokens ()
virtual bool | hasMoreTokens () const
virtual std::string | nextToken ()
virtual double | progress () const
Public Member Functions inherited from Tokenizer
| Tokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual | ~Tokenizer ()
void | blankspaceChars (const std::string)
std::string | blankspaceChars () const
void | separatorChars (const std::string)
std::string | separatorChars () const
void | lowercaseMode (const bool)
bool | lowercaseMode () const
std::string | streamToString () const

## Additional Inherited Members

Protected Member Functions inherited from Tokenizer
bool | isBlankspace (const int character) const
bool | isSeparator (const int character) const
Protected Attributes inherited from Tokenizer
std::istream & | stream
std::ios::iostate | sstate
std::streamoff | offbeg
std::streamoff | offend
std::streamoff | offset

## Detailed Description

Tokenizes a stream from the end to the beginning.

|_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file reverseTokenizer.h.

## Constructor & Destructor Documentation

## ReverseTokenizer()

ReverseTokenizer::ReverseTokenizer | ( | std::istream & | stream,
|  | const std::string | blanks,
|  | const std::string | separs )

Definition at line 27 of file reverseTokenizer.cpp.

References Tokenizer::offend, Tokenizer::offset, Tokenizer::stream, and Tokenizer::Tokenizer().

Here is the call graph for this function:

## ~ReverseTokenizer()

ReverseTokenizer::~ReverseTokenizer | ( |  | ) |

virtual

Definition at line 44 of file reverseTokenizer.cpp.

## Member Function Documentation

## countTokens()

int ReverseTokenizer::countTokens | ( |  | ) |

virtual

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 47 of file reverseTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

## hasMoreTokens()

bool ReverseTokenizer::hasMoreTokens | ( |  | ) | const

virtual

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 69 of file reverseTokenizer.cpp.

References Tokenizer::offbeg, and Tokenizer::offset.

Referenced by countTokens(), and ContextTracker::getToken().

Here is the caller graph for this function:

## nextToken()

std::string ReverseTokenizer::nextToken | ( |  | ) |

virtual

Returns the next token.

Implements Tokenizer.

Definition at line 80 of file reverseTokenizer.cpp.

References Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offbeg, Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by ContextChangeDetector::change(), countTokens(), and ContextTracker::getToken().

Here is the call graph for this function:
Here is the caller graph for this function:

## progress()

double ReverseTokenizer::progress | ( |  | ) | const

virtual

Returns progress percentage.

Implements Tokenizer.

Definition at line 174 of file reverseTokenizer.cpp.

References Tokenizer::offbeg, Tokenizer::offend, and Tokenizer::offset.

The documentation for this class was generated from the following files:

- /build/presage-XwIsB9/presage-0.9.1/src/lib/core/tokenizer/reverseTokenizer.h
- /build/presage-XwIsB9/presage-0.9.1/src/lib/core/tokenizer/reverseTokenizer.cpp