Package com.sun.speech.freetts.en

# Class TokenizerImpl

java.lang.Object
com.sun.speech.freetts.en.TokenizerImpl

All Implemented Interfaces:
  `Tokenizer`

public class TokenizerImpl extends Object implements Tokenizer
Implements the tokenizer interface. Breaks an input sequence of characters into a set of tokens.

## Field Summary

Fields
Modifier and Type
Field
Description
`static final String`
`DEFAULT_POSTPUNCTUATION_SYMBOLS`
A string containing the default post-punctuation characters.
`static final String`
`DEFAULT_PREPUNCTUATION_SYMBOLS`
A string containing the default pre-punctuation characters.
`static final String`
`DEFAULT_SINGLE_CHAR_SYMBOLS`
A string containing the default single characters.
`static final String`
`DEFAULT_WHITESPACE_SYMBOLS`
A string containing the default whitespace characters.
`static final int`
`EOF`
A constant indicating that the end of the stream has been read.

## Constructor Summary

Constructors
Constructor
Description
`TokenizerImpl()`
Constructs a Tokenizer.
`TokenizerImpl(Reader file)`
Creates a tokenizer that will return tokens from the given file.
`TokenizerImpl(String string)`
Creates a tokenizer that will return tokens from the given string.

## Method Summary

Modifier and Type
Method
Description
`String`
`getErrorDescription()`
if hasErrors returns `true`, this will return a description of the error encountered, otherwise it will return `null`
`Token`
`getNextToken()`
Returns the next token.
`boolean`
`hasErrors()`
Returns `true` if there were errors while reading tokens
`boolean`
`hasMoreTokens()`
Returns `true` if there are more tokens, `false` otherwise.
`boolean`
`isBreak()`
Determines if the current token should start a new sentence.
`void`
`setInputReader(Reader reader)`
Sets the input reader
`void`
`setInputText(String inputString)`
Sets the text to tokenize.
`void`
`setPostpunctuationSymbols(String symbols)`
Sets the postpunctuation symbols of this Tokenizer to the given symbols.
`void`
`setPrepunctuationSymbols(String symbols)`
Sets the prepunctuation symbols of this Tokenizer to the given symbols.
`void`
`setSingleCharSymbols(String symbols)`
Sets the single character symbols of this Tokenizer to the given symbols.
`void`
`setWhitespaceSymbols(String symbols)`
Sets the whitespace symbols of this Tokenizer to the given symbols.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Field Details

### EOF

public static final int EOF
A constant indicating that the end of the stream has been read.

See Also:
  - Constant Field Values

### DEFAULT_WHITESPACE_SYMBOLS

public static final String DEFAULT_WHITESPACE_SYMBOLS
A string containing the default whitespace characters.

See Also:
  - Constant Field Values

### DEFAULT_SINGLE_CHAR_SYMBOLS

public static final String DEFAULT_SINGLE_CHAR_SYMBOLS
A string containing the default single characters.

See Also:
  - Constant Field Values

### DEFAULT_PREPUNCTUATION_SYMBOLS

public static final String DEFAULT_PREPUNCTUATION_SYMBOLS
A string containing the default pre-punctuation characters.

See Also:
  - Constant Field Values

### DEFAULT_POSTPUNCTUATION_SYMBOLS

public static final String DEFAULT_POSTPUNCTUATION_SYMBOLS
A string containing the default post-punctuation characters.

See Also:
  - Constant Field Values

## Constructor Details

### TokenizerImpl

public TokenizerImpl()
Constructs a Tokenizer.

### TokenizerImpl

public TokenizerImpl(String string)
Creates a tokenizer that will return tokens from the given string.

Parameters:
  `string` - the string to tokenize

### TokenizerImpl

public TokenizerImpl(Reader file)
Creates a tokenizer that will return tokens from the given file.

Parameters:
  `file` - where to read the input from

## Method Details

### setWhitespaceSymbols

public void setWhitespaceSymbols(String symbols)
Sets the whitespace symbols of this Tokenizer to the given symbols.

Specified by:
  `setWhitespaceSymbols` in interface `Tokenizer`
Parameters:
  `symbols` - the whitespace symbols

### setSingleCharSymbols

public void setSingleCharSymbols(String symbols)
Sets the single character symbols of this Tokenizer to the given symbols.

Specified by:
  `setSingleCharSymbols` in interface `Tokenizer`
Parameters:
  `symbols` - the single character symbols

### setPrepunctuationSymbols

public void setPrepunctuationSymbols(String symbols)
Sets the prepunctuation symbols of this Tokenizer to the given symbols.

Specified by:
  `setPrepunctuationSymbols` in interface `Tokenizer`
Parameters:
  `symbols` - the prepunctuation symbols

### setPostpunctuationSymbols

public void setPostpunctuationSymbols(String symbols)
Sets the postpunctuation symbols of this Tokenizer to the given symbols.

Specified by:
  `setPostpunctuationSymbols` in interface `Tokenizer`
Parameters:
  `symbols` - the postpunctuation symbols

### setInputText

public void setInputText(String inputString)
Sets the text to tokenize.

Specified by:
  `setInputText` in interface `Tokenizer`
Parameters:
  `inputString` - the string to tokenize

### setInputReader

public void setInputReader(Reader reader)
Sets the input reader

Specified by:
  `setInputReader` in interface `Tokenizer`
Parameters:
  `reader` - the input source

### getNextToken

public Token getNextToken()
Returns the next token.

Specified by:
  `getNextToken` in interface `Tokenizer`
Returns:
  the next token if it exists, `null` if no more tokens

### hasMoreTokens

public boolean hasMoreTokens()
Returns `true` if there are more tokens, `false` otherwise.

Specified by:
  `hasMoreTokens` in interface `Tokenizer`
Returns:
  `true` if there are more tokens `false` otherwise

### hasErrors

public boolean hasErrors()
Returns `true` if there were errors while reading tokens

Specified by:
  `hasErrors` in interface `Tokenizer`
Returns:
  `true` if there were errors; `false` otherwise

### getErrorDescription

public String getErrorDescription()
if hasErrors returns `true`, this will return a description of the error encountered, otherwise it will return `null`

Specified by:
  `getErrorDescription` in interface `Tokenizer`
Returns:
  a description of the last error that occurred.

### isBreak

public boolean isBreak()
Determines if the current token should start a new sentence.

Specified by:
  `isBreak` in interface `Tokenizer`
Returns:
  `true` if a new sentence should be started