# Class Tokenizer

java.lang.Object
uk.ac.starlink.ttools.Tokenizer

public class Tokenizer extends Object
Utility class to help with tokenizing strings.

Since:
  1 Sep 2005
Author:
  Mark Taylor

## Method Summary

Modifier and Type
Method
Description
`static String[]`
`tokenizeLines(String text)`
Splits a string up into lines, separated by semicolons or newlines.
`static String[]`
`tokenizeWords(String line)`
Chops up a line of text into tokens.

### Methods inherited from class Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Method Details

### tokenizeLines

public static String[] tokenizeLines(String text) throws uk.ac.starlink.task.UsageException
Splits a string up into lines, separated by semicolons or newlines. Semicolons may appear inside quoted strings without terminating a line.

Parameters:
  `text` - input string
Returns:
  array of lines
Throws:
  `uk.ac.starlink.task.UsageException`

### tokenizeWords

public static String[] tokenizeWords(String line) throws uk.ac.starlink.task.UsageException
Chops up a line of text into tokens. Works roughly like the shell, as regards quotes, whitespace and comments.

Parameters:
  `line` - line of text
Returns:
  array of words corresponding to `line`
Throws:
  `uk.ac.starlink.task.UsageException`