Split given source into PHP tokens

- « Tokenizer Functions

- token_name »

- PHP Manual
- Tokenizer Functions
- Split given source into PHP tokens

# token_get_all

(PHP 4 >= 4.2.0, PHP 5, PHP 7, PHP 8)

token_get_all — Split given source into PHP tokens

### Description

token_get_all(string `$code`, int `$flags` = 0): array

token_get_all() parses the given `code` string into PHP language tokens using the Zend engine's lexical scanner.

For a list of parser tokens, see List of Parser Tokens, or use token_name() to translate a token value into its string representation.

### Parameters

`code`

  The PHP source to parse.

`flags`

  Valid flags:

  - `TOKEN_PARSE` - Recognises the ability to use reserved words in specific contexts.

### Return Values

An array of token identifiers. Each individual token identifier is either a single character (i.e.: `;`, `.`, `>`, `!`, etc...), or a three element array containing the token index in element 0, the string content of the original token in element 1 and the line number in element 2.

### Examples

Example #1 token_get_all() example

    <?php
    $tokens = token_get_all('<?php echo; ?>');

    foreach ($tokens as $token) {
        if (is_array($token)) {
            echo "Line {$token[2]}: ", token_name($token[0]), " ('{$token[1]}')", PHP_EOL;
        }
    }
    ?>

The above example will output something similar to:

    Line 1: T_OPEN_TAG ('<?php ')
    Line 1: T_ECHO ('echo')
    Line 1: T_WHITESPACE (' ')
    Line 1: T_CLOSE_TAG ('?>')

Example #2 token_get_all() incorrect usage example

    <?php
    $tokens = token_get_all('/* comment */');

    foreach ($tokens as $token) {
        if (is_array($token)) {
            echo "Line {$token[2]}: ", token_name($token[0]), " ('{$token[1]}')", PHP_EOL;
        }
    }
    ?>

The above example will output something similar to:

    Line 1: T_INLINE_HTML ('/* comment */')

Note in the previous example that the string is parsed as `T_INLINE_HTML` rather than the expected `T_COMMENT`. This is because no open tag was used in the code provided. This would be equivalent to putting a comment outside of the PHP tags in a normal file.

Example #3 token_get_all() on a class using a reserved word example

    <?php

    $source = <<<'code'
    <?php

    class A
    {
        const PUBLIC = 1;
    }
    code;

    $tokens = token_get_all($source, TOKEN_PARSE);

    foreach ($tokens as $token) {
        if (is_array($token)) {
            echo token_name($token[0]) , PHP_EOL;
        }
    }
    ?>

The above example will output something similar to:

    T_OPEN_TAG
    T_WHITESPACE
    T_CLASS
    T_WHITESPACE
    T_STRING
    T_CONST
    T_WHITESPACE
    T_STRING
    T_LNUMBER

Without the `TOKEN_PARSE` flag, the penultimate token (`T_STRING`) would have been `T_PUBLIC`.

### See Also

- PhpToken::tokenize() - Splits given source into PHP tokens, represented by PhpToken objects.
- token_name() - Get the symbolic name of a given PHP token