Skip to content

# `mistral_common.tokens.tokenizers.tekken`

## `ModelData`

Bases: `TypedDict`

The data of the tekken tokenizer model.

Attributes:

Name | Type | Description
--- | --- | ---
`vocab` | `list[TokenInfo]` | The vocabulary of the tokenizer.
`config` | `TekkenConfig` | The configuration of the tokenizer.
`version` | `int` | The version of the tokenizer.
`type` | `str` | The type of the tokenizer.
`image` | `ImageConfig` | The image configuration of the tokenizer.

## `SpecialTokenInfo`

Bases: `TypedDict`

Special token information in the JSON file.

Attributes:

Name | Type | Description
--- | --- | ---
`rank` | `int` | The rank of the token.
`token_str` | `str` | The token in string format.
`is_control` | `bool` | Whether the token is a control token.

## `TekkenConfig`

Bases: `TypedDict`

Tekken configuration in the JSON file.

Attributes:

Name | Type | Description
--- | --- | ---
`pattern` | `str` | The pattern of the tokenizer.
`num_vocab_tokens` | `int` | The number of vocabulary tokens.
`default_vocab_size` | `int` | The default vocabulary size.
`default_num_special_tokens` | `int` | The default number of special tokens.
`version` | `str` | The version of the tokenizer.

## `Tekkenizer(vocab, special_tokens, pattern, vocab_size, num_special_tokens, version, *, name='tekkenizer', _path=None, image_config=None, audio_config=None, model_settings_builder=None)`

Bases: `Tokenizer`

Tekken tokenizer.

This tokenizer is based on the tiktoken library. It fastens the tokenization for multiple languages.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`vocab` | `list[TokenInfo]` | The vocabulary of the tokenizer. | required
`special_tokens` | `list[SpecialTokenInfo]` | The special tokens of the tokenizer. | required
`pattern` | `str` | The pattern of the tokenizer. | required
`vocab_size` | `int` | The vocabulary size of the tokenizer. | required
`num_special_tokens` | `int` | The number of special tokens of the tokenizer. | required
`version` | `TokenizerVersion` | The version of the tokenizer. | required
`name` | `str` | The name of the tokenizer. | `'tekkenizer'`
`image_config` | `ImageConfig | None` | The image configuration of the tokenizer. | `None`
`audio_config` | `AudioConfig | None` | The audio configuration of the tokenizer. | `None`
`model_settings_builder` | `ModelSettingsBuilder | None` | The builder for model settings, or None if unsupported. | `None`

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    194
    195
    196
    197
    198
    199
    200
    201
    202
    203
    204
    205
    206
    207
    208
    209
    210
    211
    212
    213
    214
    215
    216
    217
    218
    219
    220
    221
    222
    223
    224
    225
    226
    227
    228

    def __init__(
        self,
        vocab: list[TokenInfo],
        special_tokens: list[SpecialTokenInfo],
        pattern: str,
        vocab_size: int,
        num_special_tokens: int,
        version: TokenizerVersion,
        *,
        name: str = "tekkenizer",
        _path: str | Path | None = None,
        image_config: ImageConfig | None = None,
        audio_config: AudioConfig | None = None,
        model_settings_builder: ModelSettingsBuilder | None = None,
    ):
        r"""Initialize the tekken tokenizer.

        Args:
            vocab: The vocabulary of the tokenizer.
            special_tokens: The special tokens of the tokenizer.
            pattern: The pattern of the tokenizer.
            vocab_size: The vocabulary size of the tokenizer.
            num_special_tokens: The number of special tokens of the tokenizer.
            version: The version of the tokenizer.
            name: The name of the tokenizer.
            image_config: The image configuration of the tokenizer.
            audio_config: The audio configuration of the tokenizer.
            model_settings_builder: The builder for model settings, or None if unsupported.
        """
        if not version.supports_model_settings and model_settings_builder is not None:
            raise ValueError(
                f"model_settings_builder is not supported for {version=} but got {model_settings_builder=}"
            )

        assert vocab_size <= len(vocab) + num_special_tokens, (
            vocab_size,
            len(vocab),
            num_special_tokens,
        )
        self._vocab_size = vocab_size

        # The number of special tokens defined in the tokenizer json
        num_defined_special_tokens = len(set([t["token_str"] for t in special_tokens]))

        assert len(special_tokens) == num_defined_special_tokens, f"Special tokens must be unique: {special_tokens}"
        assert len(special_tokens) <= num_special_tokens

        special_filler = [
            SpecialTokenInfo(rank=i, token_str=self.SPECIAL_TOKEN_TEMPLATE.format(id=i), is_control=True)
            for i in range(len(special_tokens), num_special_tokens)
        ]
        if special_filler:
            logger.info(
                f"Adding special tokens {special_filler[0]['token_str']}, ..., {special_filler[-1]['token_str']}"
            )
        special_tokens = special_tokens + special_filler

        assert len(set([t["token_str"] for t in special_tokens])) == len(special_tokens) == num_special_tokens, (
            special_tokens
        )
        inner_vocab_size = vocab_size - num_special_tokens

        # reload vocab
        logger.info(f"Non special vocabulary size is {inner_vocab_size} with {num_special_tokens} special tokens.")
        self._tekken_token2id_nospecial = _reload_mergeable_ranks(vocab, max_vocab=inner_vocab_size)
        assert set(range(inner_vocab_size)) == set(self._tekken_token2id_nospecial.values()), (
            inner_vocab_size,
            self._tekken_token2id_nospecial,
        )
        self._model = tiktoken.Encoding(
            name=name,
            pat_str=pattern,
            mergeable_ranks=self._tekken_token2id_nospecial,
            special_tokens={},  # special tokens are handled manually
        )

        self._version = version

        self._image_config = image_config
        self._audio_config = audio_config

        self._all_special_tokens = special_tokens
        self._special_token_ids = {t["rank"] for t in special_tokens}
        self._special_tokens_reverse_vocab = {t["token_str"]: t["rank"] for t in special_tokens}
        self._vocab = [self.id_to_piece(i) for i in range(vocab_size)]
        self._special_token_policy = SpecialTokenPolicy.IGNORE
        self._file_path = Path(_path) if _path is not None else None
        self._model_settings_builder = model_settings_builder

### `audio` `property` `writable`

The audio configuration of the tokenizer.

Returns:

Type | Description
--- | ---
`AudioConfig | None` | The audio configuration object if it exists, otherwise None.

### `bos_id` `cached` `property`

The beginning of sentence token id.

### `eos_id` `cached` `property`

The end of sentence token id.

### `file_path` `property`

The path to the tokenizer file.

### `image` `property` `writable`

The image configuration of the tokenizer.

### `model_settings_builder` `property`

The model settings builder, or None if unsupported by this version.

### `n_words` `property`

Vocabulary size of the tokenizer.

### `num_special_tokens` `property`

The number of special tokens of the tokenizer.

### `pad_id` `cached` `property`

The padding token id.

### `special_ids` `cached` `property`

Ids of the special tokens.

### `unk_id` `cached` `property`

The unknown token id.

### `version` `property`

The version of the tokenizer.

### `decode(tokens, special_token_policy=SpecialTokenPolicy.IGNORE)`

Decode a list of token ids into a string.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`tokens` | `list[int]` | The list of token ids to decode. | required
`special_token_policy` | `SpecialTokenPolicy` | The policy for handling special tokens. | `IGNORE`

Returns:

Type | Description
--- | ---
`str` | The decoded string.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    474
    475
    476
    477
    478
    479
    480
    481
    482
    483
    484
    485
    486
    487
    488
    489
    490
    491
    492

    def decode(self, tokens: list[int], special_token_policy: SpecialTokenPolicy = SpecialTokenPolicy.IGNORE) -> str:
        r"""Decode a list of token ids into a string.

        Args:
            tokens: The list of token ids to decode.
            special_token_policy: The policy for handling special tokens.

        Returns:
            The decoded string.
        """
        try:
            special_token_policy = SpecialTokenPolicy(special_token_policy)
        except ValueError as e:
            valid = ", ".join(repr(p.value) for p in SpecialTokenPolicy)
            raise ValueError(
                f"Invalid `special_token_policy` {special_token_policy!r}. Expected one of: {valid}."
            ) from e

        return "".join(self._decode_all(tokens, special_token_policy=special_token_policy))

### `encode(s, bos, eos)`

Encode a string into a list of token ids.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`s` | `str` | The string to encode. | required
`bos` | `bool` | Whether to add the beginning of sentence token. | required
`eos` | `bool` | Whether to add the end of sentence token. | required

Returns:

Type | Description
--- | ---
`list[int]` | The list of token ids.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    403
    404
    405
    406
    407
    408
    409
    410
    411
    412
    413
    414
    415
    416
    417
    418
    419
    420

    def encode(self, s: str, bos: bool, eos: bool) -> list[int]:
        r"""Encode a string into a list of token ids.

        Args:
            s: The string to encode.
            bos: Whether to add the beginning of sentence token.
            eos: Whether to add the end of sentence token.

        Returns:
            The list of token ids.
        """
        tokens: list[int] = self._model.encode(s)
        tokens = [t + self.num_special_tokens for t in tokens]
        if bos:
            tokens = [self.bos_id, *tokens]
        if eos:
            tokens = [*tokens, self.eos_id]
        return tokens

### `from_file(path)` `classmethod`

Load the tekken tokenizer from a file.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `str | Path` | The path to the tokenizer file. | required

Returns:

Type | Description
--- | ---
`Tekkenizer` | The tekken tokenizer.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    242
    243
    244
    245
    246
    247
    248
    249
    250
    251
    252
    253
    254
    255
    256
    257
    258
    259
    260
    261
    262
    263
    264
    265
    266
    267
    268
    269
    270
    271
    272
    273
    274
    275
    276
    277
    278
    279
    280
    281
    282
    283
    284
    285
    286
    287
    288
    289
    290
    291
    292
    293
    294
    295
    296
    297
    298
    299
    300
    301
    302
    303
    304
    305
    306
    307
    308
    309
    310
    311
    312
    313
    314
    315
    316
    317
    318
    319
    320
    321
    322

    @classmethod
    def from_file(cls: type["Tekkenizer"], path: str | Path) -> "Tekkenizer":
        r"""Load the tekken tokenizer from a file.

        Args:
            path: The path to the tokenizer file.

        Returns:
            The tekken tokenizer.
        """
        if isinstance(path, str):
            path = Path(path)
        assert path.exists(), path
        with open(path, "r", encoding="utf-8") as f:
            untyped = json.load(f)

        _version_str = untyped["config"].get("version")
        if _version_str not in TokenizerVersion.__members__:
            raise ValueError(
                f"Unknown version: {_version_str} in {path}. "
                f"Make sure to use a valid version string: {list(TokenizerVersion.__members__)}"
            )

        assert _version_str is not None
        version = TokenizerVersion(_version_str)

        special_tokens_dicts: list[SpecialTokenInfo] | None = untyped.get("special_tokens", None)
        if special_tokens_dicts is None:
            # Tokenizer > v7 should find special tokens in the tokenizer file
            if version > TokenizerVersion.v7:
                raise ValueError(
                    f"Special tokens not found in {path}. "
                    "Please update your tokenizer file and include all special tokens you need."
                )
            else:
                special_tokens = list(Tekkenizer.DEPRECATED_SPECIAL_TOKENS)
        else:
            special_tokens = [token for token in special_tokens_dicts]

        untyped["special_tokens"] = special_tokens

        if mm := untyped.get("multimodal"):
            # deprecated - only allowed for tokenizers <= v11
            if version > TokenizerVersion.v11:
                raise ValueError(
                    f"The image config has to be called 'image' in {path} for tokenizers of version {version.value}."
                )

            untyped["image"] = ImageConfig(**mm)
        elif image := untyped.get("image"):
            untyped["image"] = ImageConfig(**image)

        if audio := untyped.get("audio"):
            encoding_config = audio.pop("audio_encoding_config")
            encoding_config = AudioSpectrogramConfig(**encoding_config)
            untyped["audio"] = AudioConfig(encoding_config=encoding_config, **audio)

        if (
            model_settings_builder := untyped.get("model_settings_builder")
        ) is not None and not version.supports_model_settings:
            raise ValueError(
                f"model_settings_builder is not supported for {version=} but got {model_settings_builder=}"
            )
        elif model_settings_builder is not None:
            model_settings_builder = ModelSettingsBuilder.model_validate(model_settings_builder)

        model_data: ModelData = untyped

        return cls(
            vocab=model_data["vocab"],
            special_tokens=special_tokens,
            pattern=model_data["config"]["pattern"],
            vocab_size=model_data["config"]["default_vocab_size"],
            num_special_tokens=model_data["config"]["default_num_special_tokens"],
            version=version,
            name=path.name.replace(".json", ""),
            image_config=model_data.get("image"),
            audio_config=model_data.get("audio"),
            model_settings_builder=model_settings_builder,
            _path=path,
        )

### `get_special_token(s)`

Get the token id of a special token.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    454
    455
    456
    457
    458
    459

    def get_special_token(self, s: str) -> int:
        r"""Get the token id of a special token."""
        if s in self._special_tokens_reverse_vocab:
            return self._special_tokens_reverse_vocab[s]
        else:
            raise ValueError(f"Unknown control token {s}")

### `id_to_byte_piece(token_id, special_token_policy=SpecialTokenPolicy.IGNORE)`

Convert a token id to its byte representation.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`token_id` | `int` | The token id to convert. | required
`special_token_policy` | `SpecialTokenPolicy` | The policy for handling special tokens. | `IGNORE`

Returns:

Type | Description
--- | ---
`bytes` | The byte representation of the token.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    501
    502
    503
    504
    505
    506
    507
    508
    509
    510
    511
    512
    513
    514
    515
    516
    517
    518
    519
    520
    521
    522
    523

    def id_to_byte_piece(
        self, token_id: int, special_token_policy: SpecialTokenPolicy = SpecialTokenPolicy.IGNORE
    ) -> bytes:
        r"""Convert a token id to its byte representation.

        Args:
            token_id: The token id to convert.
            special_token_policy: The policy for handling special tokens.

        Returns:
            The byte representation of the token.
        """
        if token_id < self.num_special_tokens:
            if special_token_policy == SpecialTokenPolicy.KEEP:
                return self._all_special_tokens[token_id]["token_str"].encode("utf-8")
            elif special_token_policy == SpecialTokenPolicy.RAISE:
                raise ValueError(f"{token_id} is a special token")
            elif special_token_policy == SpecialTokenPolicy.IGNORE:
                return b""
            else:
                raise ValueError(f"Unknown special token policy {special_token_policy}")

        return self._model.decode_single_token_bytes(token_id - self.num_special_tokens)

### `id_to_piece(token_id)`

Convert a token id to its string representation.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    497
    498
    499

    def id_to_piece(self, token_id: int) -> str:
        r"""Convert a token id to its string representation."""
        return self.decode([token_id], special_token_policy=SpecialTokenPolicy.KEEP)

### `is_byte(token_id)`

Check if a token id is a byte token.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    450
    451
    452

    def is_byte(self, token_id: int) -> bool:
        r"""Check if a token id is a byte token."""
        return 0 <= token_id - self.num_special_tokens < 256

### `is_special(token)`

Return `True` if the passed `token` is a special token.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    461
    462
    463
    464
    465
    466
    467
    468

    def is_special(self, token: int | np.integer | str) -> bool:
        """Return `True` if the passed `token` is a special token."""
        if isinstance(token, (int, np.integer)):
            return token in self._special_token_ids
        elif isinstance(token, str):
            return token in self._special_tokens_reverse_vocab
        else:
            raise TypeError(f"Expected int or str, got {type(token).__name__}")

### `vocab()`

Get all tokens in the vocabulary as strings.

Note

This will collapse all tokens for which we have a decoding error into the <?> string. This is bad and results in things like len(set(vocab)) != len(vocab)).

Returns:

Type | Description
--- | ---
`list[str]` | The vocabulary of the tokenizer.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    386
    387
    388
    389
    390
    391
    392
    393
    394
    395
    396
    397
    398
    399
    400
    401

    def vocab(self) -> list[str]:
        r"""Get all tokens in the vocabulary as strings.

        Note:
           This will collapse all tokens for which we have a decoding error into
           the <?> string. This is bad and results in things like len(set(vocab)) != len(vocab)).

        Returns:
            The vocabulary of the tokenizer.
        """
        # when returning self._vocab this will collapse
        # all tokens for which we have a decoding error into
        # the <?> string. This is bad and results in things
        # like len(set(vocab)) != len(vocab))
        # be careful when using self._vocab
        return self._vocab

## `TokenInfo`

Bases: `TypedDict`

Token information in the JSON file.

Attributes:

Name | Type | Description
--- | --- | ---
`rank` | `int` | The rank of the token.
`token_bytes` | `str` | The token in bytes, base64 encoded.
`token_str` | `str | None` | The token in string format.

## `is_tekken(path)`

Check if the given path is a tekken tokenizer file.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    33
    34
    35
    36
    37

    def is_tekken(path: str | Path) -> bool:
        r"""Check if the given path is a tekken tokenizer file."""
        if isinstance(path, str):
            path = Path(path)
        return path.is_file() and "tekken" in path.name and path.suffix == ".json"

## `is_tekkenizer(tokenizer)`

Returns whether the tokenizer is a Tekkenizer.

Source code in `src/mistral_common/tokens/tokenizers/tekken.py`

    553
    554
    555

    def is_tekkenizer(tokenizer: Tokenizer) -> TypeGuard[Tekkenizer]:
        r"""Returns whether the tokenizer is a Tekkenizer."""
        return isinstance(tokenizer, Tekkenizer)