Skip to content

# `mistral_common.tokens.tokenizers.mistral`

## `MistralTokenizer(instruct_tokenizer, validator, request_normalizer)`

Bases: `Generic[UserMessageType, AssistantMessageType, ToolMessageType, SystemMessageType, TokenizedType]`

Mistral tokenizer.

This class is a wrapper around a InstructTokenizer, a MistralRequestValidator and a InstructRequestNormalizer.

It provides a convenient interface to tokenize, validate ad normalize Mistral requests.

Attributes:

Name | Type | Description
--- | --- | ---
`instruct_tokenizer` | `InstructTokenizer[InstructRequest, FIMRequest, TokenizedType, AssistantMessageType]` | The instruct tokenizer to use. See [InstructTokenizer][mistral_common.tokens.tokenizers.instruct.InstructTokenizer].

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`instruct_tokenizer` | `InstructTokenizer[InstructRequest, FIMRequest, TokenizedType, AssistantMessageType]` | The instruct tokenizer to use. | required
`validator` | `MistralRequestValidator[UserMessageType, AssistantMessageType, ToolMessageType, SystemMessageType]` | The request validator to use. | required
`request_normalizer` | `InstructRequestNormalizer[UserMessageType, AssistantMessageType, ToolMessageType, SystemMessageType, InstructRequestType]` | The request normalizer to use. | required

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138

    def __init__(
        self,
        instruct_tokenizer: InstructTokenizer[InstructRequest, FIMRequest, TokenizedType, AssistantMessageType],
        validator: MistralRequestValidator[UserMessageType, AssistantMessageType, ToolMessageType, SystemMessageType],
        request_normalizer: InstructRequestNormalizer[
            UserMessageType, AssistantMessageType, ToolMessageType, SystemMessageType, InstructRequestType
        ],
    ):
        r"""Initializes a `MistralTokenizer`.

        Args:
            instruct_tokenizer: The instruct tokenizer to use.
            validator: The request validator to use.
            request_normalizer: The request normalizer to use.
        """
        self._chat_completion_request_validator = validator
        self._instruct_request_normalizer = request_normalizer
        self.instruct_tokenizer: InstructTokenizer[InstructRequest, FIMRequest, TokenizedType, AssistantMessageType] = (
            instruct_tokenizer
        )

### `mode` `property`

The validation mode of the tokenizer.

### `version` `property`

The version of the tokenizer.

### `__reduce__()`

Provides a recipe for pickling (serializing) this object, which is necessary for use with multiprocessing.

Returns:

Type | Description
--- | ---
`tuple[Callable, tuple[Any, ...]]` | A tuple of the factory function and the arguments to reconstruct the object from its source file.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    140
    141
    142
    143
    144
    145
    146
    147

    def __reduce__(self) -> tuple[Callable, tuple[Any, ...]]:
        """
        Provides a recipe for pickling (serializing) this object, which is necessary for use with multiprocessing.

        Returns:
            A tuple of the factory function and the arguments to reconstruct the object from its source file.
        """
        return MistralTokenizer.from_file, (self.instruct_tokenizer.tokenizer.file_path, self.mode)

### `decode(tokens, special_token_policy=SpecialTokenPolicy.IGNORE)`

Decodes a list of tokens into a string.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`tokens` | `list[int]` | The tokens to decode. | required
`special_token_policy` | `SpecialTokenPolicy` | The policy to use for special tokens. | `IGNORE`

Returns:

Type | Description
--- | ---
`str` | The decoded string.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    433
    434
    435
    436
    437
    438
    439
    440
    441
    442
    443

    def decode(self, tokens: list[int], special_token_policy: SpecialTokenPolicy = SpecialTokenPolicy.IGNORE) -> str:
        r"""Decodes a list of tokens into a string.

        Args:
            tokens: The tokens to decode.
            special_token_policy: The policy to use for special tokens.

        Returns:
            The decoded string.
        """
        return self.instruct_tokenizer.decode(tokens, special_token_policy=special_token_policy)

### `encode_chat_completion(request, max_model_input_len=None)`

Encodes a chat completion request.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `ChatCompletionRequest[UATS]` | The chat completion request to encode. | required
`max_model_input_len` | `int | None` | The maximum length of the input to the model. If `None`, the input will not be truncated. | `None`

Returns:

Type | Description
--- | ---
`TokenizedType` | The encoded chat completion request.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    369
    370
    371
    372
    373
    374
    375
    376
    377
    378
    379
    380
    381
    382
    383
    384
    385
    386
    387
    388
    389
    390
    391
    392
    393
    394
    395
    396
    397
    398

    def encode_chat_completion(
        self, request: ChatCompletionRequest[UATS], max_model_input_len: int | None = None
    ) -> TokenizedType:
        r"""Encodes a chat completion request.

        Args:
            request: The chat completion request to encode.
            max_model_input_len: The maximum length of the input to the model.
                If `None`, the input will not be truncated.

        Returns:
            The encoded chat completion request.
        """

        validated_request = self._chat_completion_request_validator.validate_request(request)

        if max_model_input_len is None and request.truncate_for_context_length:
            # the max_model_input_len arg should not be optional ;
            # but this function is used in many small scripts that have no use
            # for truncation, and don't provide the max model len
            raise TokenizerException(
                "encoding a chat completion request with truncation, but no max model len was provided",
            )

        instruct_request = self._instruct_request_normalizer.from_chat_completion_request(validated_request)

        if request.truncate_for_context_length:
            instruct_request.truncate_at_max_tokens = max_model_input_len

        return self.instruct_tokenizer.encode_instruct(instruct_request)

### `encode_fim(request)`

Encodes a fill in the middle request.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `FIMRequest` | The fill in the middle request to encode. | required

Returns:

Type | Description
--- | ---
`TokenizedType` | The encoded fill in the middle request.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    422
    423
    424
    425
    426
    427
    428
    429
    430
    431

    def encode_fim(self, request: FIMRequest) -> TokenizedType:
        r"""Encodes a fill in the middle request.

        Args:
            request: The fill in the middle request to encode.

        Returns:
            The encoded fill in the middle request.
        """
        return self.instruct_tokenizer.encode_fim(request)

### `encode_speech_request(request)`

Encodes a speech synthesis request.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `SpeechRequest` | The speech request to encode. | required

Returns:

Type | Description
--- | ---
`TokenizedType` | The encoded speech request.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    411
    412
    413
    414
    415
    416
    417
    418
    419
    420

    def encode_speech_request(self, request: SpeechRequest) -> TokenizedType:
        r"""Encodes a speech synthesis request.

        Args:
            request: The speech request to encode.

        Returns:
            The encoded speech request.
        """
        return self.instruct_tokenizer.encode_speech_request(request)

### `encode_transcription(request)`

Encodes a transcription request.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `TranscriptionRequest` | The transcription request to encode. | required

Returns:

Type | Description
--- | ---
`TokenizedType` | The encoded transcription request.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    400
    401
    402
    403
    404
    405
    406
    407
    408
    409

    def encode_transcription(self, request: TranscriptionRequest) -> TokenizedType:
        r"""Encodes a transcription request.

        Args:
            request: The transcription request to encode.

        Returns:
            The encoded transcription request.
        """
        return self.instruct_tokenizer.encode_transcription(request)

### `from_file(tokenizer_filename, mode=ValidationMode.test)` `classmethod`

Loads a tokenizer from a file.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`tokenizer_filename` | `str | Path` | The path to the tokenizer file. | required
`mode` | `ValidationMode` | The validation mode to use. | `test`

Returns:

Type | Description
--- | ---
`MistralTokenizer` | The loaded tokenizer.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    280
    281
    282
    283
    284
    285
    286
    287
    288
    289
    290
    291
    292
    293
    294
    295
    296
    297
    298
    299
    300
    301
    302
    303
    304
    305
    306
    307
    308
    309
    310
    311
    312
    313
    314
    315
    316
    317
    318
    319
    320
    321
    322
    323
    324
    325
    326
    327
    328
    329
    330
    331
    332
    333
    334
    335
    336
    337
    338
    339
    340
    341
    342
    343
    344
    345
    346
    347
    348
    349
    350
    351
    352
    353
    354
    355
    356
    357
    358
    359
    360
    361
    362
    363
    364
    365
    366
    367

    @classmethod
    def from_file(
        cls,
        tokenizer_filename: str | Path,
        mode: ValidationMode = ValidationMode.test,
    ) -> "MistralTokenizer":
        r"""Loads a tokenizer from a file.

        Args:
            tokenizer_filename: The path to the tokenizer file.
            mode: The validation mode to use.

        Returns:
            The loaded tokenizer.
        """
        tokenizer: SentencePieceTokenizer | Tekkenizer

        if is_tekken(tokenizer_filename):
            tokenizer = Tekkenizer.from_file(tokenizer_filename)
            image_config = tokenizer.image
            audio_config = tokenizer.audio
        elif is_sentencepiece(tokenizer_filename):
            tokenizer = SentencePieceTokenizer(tokenizer_filename)
            image_config = get_image_config(tokenizer_filename)
            # spm can't have audio
            audio_config = None
        else:
            raise TokenizerException(f"Unrecognized tokenizer file: {tokenizer_filename}")

        image_encoder = load_image_encoder(image_config, tokenizer) if image_config is not None else None

        audio_encoder = None
        if audio_config is not None:
            assert isinstance(tokenizer, Tekkenizer), "Audio is only supported for tekken tokenizers"
            audio_encoder = load_audio_encoder(audio_config, tokenizer)

        request_normalizer = get_normalizer(tokenizer.version, tokenizer.model_settings_builder)
        validator = get_validator(tokenizer.version, mode=mode)

        if tokenizer.version == TokenizerVersion.v1:
            assert image_encoder is None, "Tokenizer version needs to be >= v3"
            assert audio_encoder is None, "Tokenizer version needs to be >= v7"
            return MistralTokenizer(
                InstructTokenizerV1(tokenizer),
                validator=validator,
                request_normalizer=request_normalizer,
            )
        elif tokenizer.version == TokenizerVersion.v2:
            assert image_encoder is None, "Tokenizer version needs to be >= v3"
            assert audio_encoder is None, "Tokenizer version needs to be >= v7"
            return MistralTokenizer(
                InstructTokenizerV2(tokenizer),
                validator=validator,
                request_normalizer=request_normalizer,
            )
        elif tokenizer.version == TokenizerVersion.v3:
            assert audio_encoder is None, "Tokenizer version needs to be >= v7"
            return MistralTokenizer(
                InstructTokenizerV3(tokenizer, image_encoder=image_encoder),
                validator=validator,
                request_normalizer=request_normalizer,
            )
        elif tokenizer.version == TokenizerVersion.v7:
            return MistralTokenizer(
                InstructTokenizerV7(tokenizer, image_encoder=image_encoder, audio_encoder=audio_encoder),
                validator=validator,
                request_normalizer=request_normalizer,
            )
        elif tokenizer.version == TokenizerVersion.v11:
            return MistralTokenizer(
                InstructTokenizerV11(tokenizer, image_encoder=image_encoder, audio_encoder=audio_encoder),
                validator=validator,
                request_normalizer=request_normalizer,
            )
        elif tokenizer.version == TokenizerVersion.v13:
            return MistralTokenizer(
                InstructTokenizerV13(tokenizer, image_encoder=image_encoder, audio_encoder=audio_encoder),
                validator=validator,
                request_normalizer=request_normalizer,
            )
        elif tokenizer.version == TokenizerVersion.v15:
            return MistralTokenizer(
                InstructTokenizerV15(tokenizer, image_encoder=image_encoder, audio_encoder=audio_encoder),
                validator=validator,
                request_normalizer=request_normalizer,
            )

        raise TokenizerException(f"Unrecognized tokenizer filename: {tokenizer_filename}")

### `from_hf_hub(repo_id, token=None, revision=None, force_download=False, local_files_only=False, mode=ValidationMode.test)` `staticmethod`

Download the Mistral tokenizer for a given Hugging Face repository ID.

See here for a list of our OSS models.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`repo_id` | `str` | The Hugging Face repo ID. | required
`token` | `bool | str | None` | The Hugging Face token to use to download the tokenizer. | `None`
`revision` | `str | None` | The revision of the model to use. If `None`, the latest revision will be used. | `None`
`mode` | `ValidationMode` | The validation mode to use. | `test`
`force_download` | `bool` | Whether to force the download of the tokenizer. If `True`, the tokenizer will be downloaded even if it is already cached. | `False`
`local_files_only` | `bool` | Whether to only use local files. If `True`, the tokenizer will be downloaded only if it is already cached. | `False`

Returns:

Type | Description
--- | ---
`MistralTokenizer` | The Mistral tokenizer for the given model.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    245
    246
    247
    248
    249
    250
    251
    252
    253
    254
    255
    256
    257
    258
    259
    260
    261
    262
    263
    264
    265
    266
    267
    268
    269
    270
    271
    272
    273
    274
    275
    276
    277
    278

    @staticmethod
    def from_hf_hub(
        repo_id: str,
        token: bool | str | None = None,
        revision: str | None = None,
        force_download: bool = False,
        local_files_only: bool = False,
        mode: ValidationMode = ValidationMode.test,
    ) -> "MistralTokenizer":
        r"""Download the Mistral tokenizer for a given Hugging Face repository ID.

        See [here](https://huggingface.co/mistralai/models) for a list of our OSS models.

        Args:
            repo_id: The Hugging Face repo ID.
            token: The Hugging Face token to use to download the tokenizer.
            revision: The revision of the model to use. If `None`, the latest revision will be used.
            mode: The validation mode to use.
            force_download: Whether to force the download of the tokenizer. If `True`, the tokenizer will be downloaded
                even if it is already cached.
            local_files_only: Whether to only use local files. If `True`, the tokenizer will be downloaded only if it is
                already cached.

        Returns:
            The Mistral tokenizer for the given model.
        """
        tokenizer_path = download_tokenizer_from_hf_hub(
            repo_id=repo_id,
            token=token,
            revision=revision,
            force_download=force_download,
            local_files_only=local_files_only,
        )
        return MistralTokenizer.from_file(tokenizer_path, mode=mode)

### `from_model(model, strict=True)` `classmethod`

Deprecated in favor of `from_hf_hub` or `from_file`, will be removed in 1.13.0.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`model` | `str` | The model name. | required
`strict` | `bool` | Has to be True, not used. | `True`

Returns:

Type | Description
--- | ---
`MistralTokenizer` | The Mistral tokenizer for the given model.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    217
    218
    219
    220
    221
    222
    223
    224
    225
    226
    227
    228
    229
    230
    231
    232
    233
    234
    235
    236
    237
    238
    239
    240
    241
    242
    243

    @classmethod
    def from_model(cls, model: str, strict: bool = True) -> "MistralTokenizer":
        r"""Deprecated in favor of `from_hf_hub` or `from_file`, will be removed in 1.13.0.

        Args:
            model: The model name.
            strict: Has to be True, not used.

        Returns:
            The Mistral tokenizer for the given model.
        """
        warnings.warn(
            "`MistralTokenizer.from_model` is deprecated and will be removed in 1.13.0. "
            "Use `MistralTokenizer.from_hf_hub(...)` or `MistralTokenizer.from_file(...)` instead.",
            FutureWarning,
        )

        if not strict:
            raise ValueError("strict has to be `True` since v1.10.0.")

        if model not in MODEL_NAME_TO_TOKENIZER_CLS:
            raise TokenizerException(
                f"Unrecognized model: {model}. Use `MistralTokenizer.from_hf_hub(...)` "
                f"or `MistralTokenizer.from_file(...)` to load newer or custom models."
            )

        return MODEL_NAME_TO_TOKENIZER_CLS[model]()

### `v1()` `classmethod`

Get the Mistral tokenizer v1.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    163
    164
    165
    166

    @classmethod
    def v1(cls) -> "MistralTokenizer":
        r"""Get the Mistral tokenizer v1."""
        return cls.from_file(str(cls._data_path() / "tokenizer.model.v1"), mode=ValidationMode.test)

### `v2()` `classmethod`

Get the Mistral tokenizer v2.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    168
    169
    170
    171
    172
    173

    @classmethod
    def v2(cls) -> "MistralTokenizer":
        r"""Get the Mistral tokenizer v2."""
        return cls.from_file(
            str(cls._data_path() / "mistral_instruct_tokenizer_240216.model.v2"), mode=ValidationMode.test
        )

### `v3(is_tekken=False, is_mm=False)` `classmethod`

Get the Mistral tokenizer v3.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`is_tekken` | `bool` | Whether the tokenizer is a tekken tokenizer. See Tekkenizer. | `False`
`is_mm` | `bool` | Whether to load image tokenizer. | `False`

Returns:

Type | Description
--- | ---
`MistralTokenizer` | The Mistral tokenizer v3.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    194
    195
    196

    @classmethod
    def v3(cls, is_tekken: bool = False, is_mm: bool = False) -> "MistralTokenizer":
        r"""Get the Mistral tokenizer v3.

        Args:
            is_tekken: Whether the tokenizer is a tekken tokenizer. See
                [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer].
            is_mm: Whether to load image tokenizer.

        Returns:
            The Mistral tokenizer v3.
        """
        if is_tekken and is_mm:
            tokenizer_name = "tekken_240911.json"
        elif is_tekken and not is_mm:
            tokenizer_name = "tekken_240718.json"
        elif not is_tekken and is_mm:
            raise ValueError("Multimodal tokenizer is currently only supported for tekken")
        else:
            tokenizer_name = "mistral_instruct_tokenizer_240323.model.v3"

        return cls.from_file(str(cls._data_path() / tokenizer_name), mode=ValidationMode.test)

### `v7(is_mm=False)` `classmethod`

Get the Mistral tokenizer v7.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`is_mm` | `bool` | Whether to load the image tokenizer. | `False`

Returns:

Type | Description
--- | ---
`MistralTokenizer` | The Mistral tokenizer v7.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    198
    199
    200
    201
    202
    203
    204
    205
    206
    207
    208
    209
    210
    211
    212
    213
    214
    215

    @classmethod
    def v7(cls, is_mm: bool = False) -> "MistralTokenizer":
        """Get the Mistral tokenizer v7.

        Args:
            is_mm: Whether to load the image tokenizer.

        Returns:
            The Mistral tokenizer v7.
        """
        if is_mm:
            return cls.from_file(
                str(cls._data_path() / "mistral_instruct_tokenizer_241114.model.v7m1"), mode=ValidationMode.test
            )
        else:
            return cls.from_file(
                str(cls._data_path() / "mistral_instruct_tokenizer_241114.model.v7"), mode=ValidationMode.test
            )

## `load_audio_encoder(audio_config, tokenizer)`

Load a audio encoder from a config and a tokenizer.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`audio_config` | `AudioConfig` | The audio config. | required
`tokenizer` | `Tekkenizer` | The tokenizer. | required

Returns:

Type | Description
--- | ---
`AudioEncoder` | The audio encoder.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100

    def load_audio_encoder(audio_config: AudioConfig, tokenizer: Tekkenizer) -> AudioEncoder:
        r"""Load a audio encoder from a config and a tokenizer.

        Args:
            audio_config: The audio config.
            tokenizer: The tokenizer.

        Returns:
            The audio encoder.
        """

        def get_special_token_or_none(token: str) -> int | None:
            if not tokenizer.is_special(token):
                return None

            return tokenizer.get_special_token(token)

        special_ids = SpecialAudioIDs(
            audio=get_special_token_or_none(SpecialTokens.audio.value),
            begin_audio=get_special_token_or_none(SpecialTokens.begin_audio.value),
            streaming_pad=get_special_token_or_none(SpecialTokens.streaming_pad.value),
            text_to_audio=get_special_token_or_none(SpecialTokens.text_to_audio.value),
            audio_to_text=get_special_token_or_none(SpecialTokens.audio_to_text.value),
        )
        return AudioEncoder(audio_config, special_ids)

## `load_image_encoder(image_config, tokenizer)`

Load a image encoder from a config and a tokenizer.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`image_config` | `ImageConfig` | The image config. | required
`tokenizer` | `Tekkenizer | SentencePieceTokenizer` | The tokenizer. | required

Returns:

Type | Description
--- | ---
`ImageEncoder` | The image encoder.

Source code in `src/mistral_common/tokens/tokenizers/mistral.py`

    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73

    def load_image_encoder(image_config: ImageConfig, tokenizer: Tekkenizer | SentencePieceTokenizer) -> ImageEncoder:
        r"""Load a image encoder from a config and a tokenizer.

        Args:
            image_config: The image config.
            tokenizer: The tokenizer.

        Returns:
            The image encoder.
        """
        special_ids = SpecialImageIDs(
            img=tokenizer.get_special_token(SpecialTokens.img.value),
            img_break=tokenizer.get_special_token(SpecialTokens.img_break.value),
            img_end=tokenizer.get_special_token(SpecialTokens.img_end.value),
        )
        return ImageEncoder(image_config, special_ids)