ForwardTokenizer Class Reference

`#include <forwardTokenizer.h>`

Inheritance diagram for ForwardTokenizer:
Collaboration diagram for ForwardTokenizer:

## Public Member Functions

| ForwardTokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual | ~ForwardTokenizer ()
virtual int | countTokens ()
virtual bool | hasMoreTokens () const
virtual std::string | nextToken ()
virtual double | progress () const
Public Member Functions inherited from Tokenizer
| Tokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual | ~Tokenizer ()
void | blankspaceChars (const std::string)
std::string | blankspaceChars () const
void | separatorChars (const std::string)
std::string | separatorChars () const
void | lowercaseMode (const bool)
bool | lowercaseMode () const
std::string | streamToString () const

## Additional Inherited Members

Protected Member Functions inherited from Tokenizer
bool | isBlankspace (const int character) const
bool | isSeparator (const int character) const
Protected Attributes inherited from Tokenizer
std::istream & | stream
std::ios::iostate | sstate
std::streamoff | offbeg
std::streamoff | offend
std::streamoff | offset

## Detailed Description

Tokenizes a stream from the beginning to the end.

|_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file forwardTokenizer.h.

## Constructor & Destructor Documentation

ForwardTokenizer::ForwardTokenizer | ( | std::istream & | stream,
|  | const std::string | blankspaces,
|  | const std::string | separators
| ) |  |

Definition at line 27 of file forwardTokenizer.cpp.

References Tokenizer::offbeg, and Tokenizer::offset.

ForwardTokenizer::~ForwardTokenizer | ( |  | ) |

virtual

Definition at line 36 of file forwardTokenizer.cpp.

## Member Function Documentation

int ForwardTokenizer::countTokens | ( |  | ) |

virtual

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 39 of file forwardTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offbeg, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

bool ForwardTokenizer::hasMoreTokens | ( |  | ) | const

virtual

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 61 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by countTokens(), main(), and ContextTracker::update().

Here is the caller graph for this function:

std::string ForwardTokenizer::nextToken | ( |  | ) |

virtual

Returns the next token.

Implements Tokenizer.

Definition at line 72 of file forwardTokenizer.cpp.

References endl(), Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by countTokens(), main(), and ContextTracker::update().

Here is the call graph for this function:

Here is the caller graph for this function:

double ForwardTokenizer::progress | ( |  | ) | const

virtual

Returns progress percentage.

Implements Tokenizer.

Definition at line 151 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by main().

Here is the caller graph for this function:

The documentation for this class was generated from the following files:

- /build/buildd/presage-0.8.8/src/lib/core/tokenizer/forwardTokenizer.h
- /build/buildd/presage-0.8.8/src/lib/core/tokenizer/forwardTokenizer.cpp