# Whitespace Tokenizer

class py_stringmatching.tokenizer.whitespace_tokenizer.WhitespaceTokenizer(return_set=False)

  Segments the input string using whitespaces then returns the segments as tokens.

  Currently using the split function in Python, so whitespace character refers to the actual whitespace character as well as the tab and newline characters.

  Parameters

    return_set (boolean) – A flag to indicate whether to return a set of tokens instead of a bag of tokens (defaults to False).

  return_set

    An attribute to store the flag return_set.

    Type

      boolean

  get_delim_set()

    Gets the current set of delimiters.

    Returns

      A Python set which is the current set of delimiters.

  get_return_set()

    Gets the value of the return_set flag.

    Returns

      The boolean value of the return_set flag.

  set_delim_set(delim_set)

    Sets the current set of delimiters.

    Parameters

      delim_set (set) – A set of delimiter strings.

  set_return_set(return_set)

    Sets the value of the return_set flag.

    Parameters

      return_set (boolean) – a flag to indicate whether to return a set of tokens instead of a bag of tokens.

  tokenize(input_string)

    Tokenizes input string based on white space.

    Parameters

      input_string (str) – The string to be tokenized.

    Returns

      A Python list, which is a set or a bag of tokens, depending on whether return_set is True or False.

    Raises

      TypeError – If the input is not a string.

    Examples

        >>> ws_tok = WhitespaceTokenizer()
        >>> ws_tok.tokenize('data science')
        ['data', 'science']
        >>> ws_tok.tokenize('data        science')
        ['data', 'science']
        >>> ws_tok.tokenize('data   science')
        ['data', 'science']
        >>> ws_tok = WhitespaceTokenizer(return_set=True)
        >>> ws_tok.tokenize('data   science data integration')
        ['data', 'science', 'integration']