| Extracted from Pike v7.8 release 866 at 2018-02-09.

[Top]
Search
Search.Grammar
Search.Grammar.Lexer

# Method Search.Grammar.Lexer.tokenize()

Method tokenize

  `string|array(array(Token|string)) tokenize(string query)`

Description

  Tokenizes a query into tokens for later use by a parser.

Parameter `query`

  The query to tokenize.

Returns

  An array containing the tokens: `({ ({ TOKEN_WORD, "foo" }), ... })` Or, in case of an error, a string with the error message.