Package weka.core.tokenizers

# Class WordTokenizer

java.lang.Object
weka.core.tokenizers.Tokenizer
weka.core.tokenizers.CharacterDelimitedTokenizer
weka.core.tokenizers.WordTokenizer

All Implemented Interfaces:
  `Serializable`, `Enumeration`, `OptionHandler`, `RevisionHandler`

public class WordTokenizer extends CharacterDelimitedTokenizer
A simple tokenizer that is using the java.util.StringTokenizer class to tokenize the strings.

Valid options are:

     -delimiters <value>
      The delimiters to use
      (default ' \r\n\t.,;:'"()?!').

Version:
  $Revision: 1.4 $
Author:
  FracPete (fracpete at waikato dot ac dot nz)
See Also:
  - Serialized Form

## Constructor Summary

Constructors
Constructor
Description
`WordTokenizer()`

## Method Summary

Modifier and Type
Method
Description
`String`
`getRevision()`
Returns the revision string.
`String`
`globalInfo()`
Returns a string describing the stemmer
`boolean`
`hasMoreElements()`
Tests if this enumeration contains more elements.
`static void`
`main(String[] args)`
Runs the tokenizer with the given options and strings to tokenize.
`Object`
`nextElement()`
Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
`void`
`tokenize(String s)`
Sets the string to tokenize.

### Methods inherited from class weka.core.tokenizers.CharacterDelimitedTokenizer

`delimitersTipText, getDelimiters, getOptions, listOptions, setDelimiters, setOptions`

### Methods inherited from class weka.core.tokenizers.Tokenizer

`runTokenizer, tokenize`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Methods inherited from interface java.util.Enumeration

`asIterator`

## Constructor Details

### WordTokenizer

public WordTokenizer()

## Method Details

### globalInfo

public String globalInfo()
Returns a string describing the stemmer

Specified by:
  `globalInfo` in class `Tokenizer`
Returns:
  a description suitable for displaying in the explorer/experimenter gui

### hasMoreElements

public boolean hasMoreElements()
Tests if this enumeration contains more elements.

Specified by:
  `hasMoreElements` in interface `Enumeration`
Specified by:
  `hasMoreElements` in class `Tokenizer`
Returns:
  true if and only if this enumeration object contains at least one more element to provide; false otherwise.

### nextElement

public Object nextElement()
Returns the next element of this enumeration if this enumeration object has at least one more element to provide.

Specified by:
  `nextElement` in interface `Enumeration`
Specified by:
  `nextElement` in class `Tokenizer`
Returns:
  the next element of this enumeration.

### tokenize

public void tokenize(String s)
Sets the string to tokenize. Tokenization happens immediately.

Specified by:
  `tokenize` in class `Tokenizer`
Parameters:
  `s` - the string to tokenize

### getRevision

public String getRevision()
Returns the revision string.

Returns:
  the revision

### main

public static void main(String[] args)
Runs the tokenizer with the given options and strings to tokenize. The tokens are printed to stdout.

Parameters:
  `args` - the commandline options and strings to tokenize