Skip to content

# `mistral_common.tokens.tokenizers.base`

## `InstructTokenizer(tokenizer, image_encoder, audio_encoder)`

Bases: `Generic[InstructRequestType, FIMRequestType, TokenizedType, AssistantMessageType]`

Base class for instruct tokenizers.

Attributes:

Name | Type | Description
--- | --- | ---
`tokenizer` | `Tokenizer` | The tokenizer to use.
`image_encoder` | `ImageEncoder | None` | The image encoder to use if any.
`audio_encoder` | `AudioEncoder | None` | The audio encoder to use if any.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`tokenizer` | `Tokenizer` | The tokenizer to use. | required
`image_encoder` | `ImageEncoder | None` | The image encoder to use if any. | required
`audio_encoder` | `AudioEncoder | None` | The audio encoder to use if any. | required

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    328
    329
    330
    331
    332
    333
    334
    335
    336
    337

    def __init__(
        self, tokenizer: Tokenizer, image_encoder: ImageEncoder | None, audio_encoder: AudioEncoder | None
    ) -> None:
        r"""Initialize the instruct tokenizer.

        Args:
            tokenizer: The tokenizer to use.
            image_encoder: The image encoder to use if any.
            audio_encoder: The audio encoder to use if any.
        """

### `version` `property`

The version of the tokenizer.

### `decode(tokens, special_token_policy)` `abstractmethod`

Convert token ids to string

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`tokens` | `list[int]` | The token ids to decode. | required
`special_token_policy` | `SpecialTokenPolicy` | The policy to use for special tokens. | required

Returns:

Type | Description
--- | ---
`str` | The decoded string.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    380
    381
    382
    383
    384
    385
    386
    387
    388
    389
    390

    @abstractmethod
    def decode(self, tokens: list[int], special_token_policy: SpecialTokenPolicy) -> str:
        r"""Convert token ids to string

        Args:
            tokens: The token ids to decode.
            special_token_policy: The policy to use for special tokens.

        Returns:
            The decoded string.
        """

### `encode_fim(request)` `abstractmethod`

FIM request to Tokenized object

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `FIMRequestType` | The FIM request to encode. | required

Returns:

Type | Description
--- | ---
`TokenizedType` | The tokenized FIM request.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    392
    393
    394
    395
    396
    397
    398
    399
    400
    401

    @abstractmethod
    def encode_fim(self, request: FIMRequestType) -> TokenizedType:
        r"""FIM request to Tokenized object

        Args:
            request: The FIM request to encode.

        Returns:
            The tokenized FIM request.
        """

### `encode_instruct(request)` `abstractmethod`

Instruct request to Tokenized object

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `InstructRequestType` | The instruct request to encode. | required

Returns:

Type | Description
--- | ---
`TokenizedType` | The tokenized instruct request.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    339
    340
    341
    342
    343
    344
    345
    346
    347
    348

    @abstractmethod
    def encode_instruct(self, request: InstructRequestType) -> TokenizedType:
        r"""Instruct request to Tokenized object

        Args:
            request: The instruct request to encode.

        Returns:
            The tokenized instruct request.
        """

### `encode_speech_request(request)` `abstractmethod`

Encodes a speech synthesis request into a tokenized format.

This method processes a speech request containing text input and optional reference audio or voice preset, and returns the tokenized output.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `SpeechRequest` | The speech request object containing the text and voice/audio data. | required

Returns:

Type | Description
--- | ---
`Tokenized` | The tokenized representation of the speech request.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    366
    367
    368
    369
    370
    371
    372
    373
    374
    375
    376
    377
    378

    @abstractmethod
    def encode_speech_request(self, request: SpeechRequest) -> TokenizedType:
        r"""Encodes a speech synthesis request into a tokenized format.

        This method processes a speech request containing text input and
        optional reference audio or voice preset, and returns the tokenized output.

        Args:
            request: The speech request object containing the text and voice/audio data.

        Returns:
            Tokenized: The tokenized representation of the speech request.
        """

### `encode_transcription(request)` `abstractmethod`

Encodes an audio transcription request into a tokenized format.

This method processes a transcription request containing audio data, encodes the user message, and returns the tokenized output.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`request` | `TranscriptionRequest` | The transcription request object containing the audio data to be encoded. | required

Returns:

Type | Description
--- | ---
`Tokenized` | The tokenized representation of the audio data, including processed audio and tokens

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    350
    351
    352
    353
    354
    355
    356
    357
    358
    359
    360
    361
    362
    363
    364

    @abstractmethod
    def encode_transcription(self, request: TranscriptionRequest) -> TokenizedType:
        r"""
        Encodes an audio transcription request into a tokenized format.

        This method processes a transcription request containing audio data,
        encodes the user message, and returns the tokenized output.

        Args:
            request: The transcription request object containing
                the audio data to be encoded.

        Returns:
            Tokenized: The tokenized representation of the audio data, including processed audio and tokens
        """

### `encode_user_content(content, is_last, system_prompt=None, force_img_first=False)` `abstractmethod`

Encode a user content.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`content` | `str | list[ContentChunk]` | The user content to encode. | required
`is_last` | `bool` | Whether the content is the last one. | required
`system_prompt` | `str | None` | The system prompt. | `None`
`force_img_first` | `bool` | Whether to force the image to be first. | `False`

Returns:

Type | Description
--- | ---
`tuple[list[int], list[ndarray], list[Audio]]` | The encoded tokens and images.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    428
    429
    430
    431
    432
    433
    434
    435
    436
    437
    438
    439
    440
    441
    442
    443
    444
    445
    446
    447

    @abstractmethod
    def encode_user_content(
        self,
        content: str | list[ContentChunk],
        is_last: bool,
        system_prompt: str | None = None,
        force_img_first: bool = False,
    ) -> tuple[list[int], list[np.ndarray], list[Audio]]:
        r"""Encode a user content.

        Args:
            content: The user content to encode.
            is_last: Whether the content is the last one.
            system_prompt: The system prompt.
            force_img_first: Whether to force the image to be first.

        Returns:
            The encoded tokens and images.
        """
        ...

### `encode_user_message(message, available_tools, is_last, is_first, system_prompt=None, force_img_first=False)` `abstractmethod`

Encode a user message.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`message` | `UserMessage` | The user message to encode. | required
`available_tools` | `list[Tool] | None` | The available tools. | required
`is_last` | `bool` | Whether the message is the last one. | required
`is_first` | `bool` | Whether the message is the first one. | required
`system_prompt` | `str | None` | The system prompt. | `None`
`force_img_first` | `bool` | Whether to force the image to be first. | `False`

Returns:

Type | Description
--- | ---
`tuple[list[int], list[ndarray], list[Audio]]` | The encoded tokens and images.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    403
    404
    405
    406
    407
    408
    409
    410
    411
    412
    413
    414
    415
    416
    417
    418
    419
    420
    421
    422
    423
    424
    425
    426

    @abstractmethod
    def encode_user_message(
        self,
        message: UserMessage,
        available_tools: list[Tool] | None,
        is_last: bool,
        is_first: bool,
        system_prompt: str | None = None,
        force_img_first: bool = False,
    ) -> tuple[list[int], list[np.ndarray], list[Audio]]:
        r"""Encode a user message.

        Args:
            message: The user message to encode.
            available_tools: The available tools.
            is_last: Whether the message is the last one.
            is_first: Whether the message is the first one.
            system_prompt: The system prompt.
            force_img_first: Whether to force the image to be first.

        Returns:
            The encoded tokens and images.
        """
        ...

## `SpecialTokenPolicy`

Bases: `str`, `Enum`

What to do with special tokens when encoding/decoding.

Attributes:

Name | Type | Description
--- | --- | ---
`IGNORE` |  | Ignore special tokens.
`KEEP` |  | Keep special tokens.
`RAISE` |  | Raise an error if special tokens are found.

## `SpecialTokens`

Bases: `str`, `Enum`

Enum of special tokens used in the tokenizer.

Attributes:

Name | Type | Description
--- | --- | ---
`unk` |  | The unknown token.
`bos` |  | The beginning of string token.
`eos` |  | The end of string token.
`begin_inst` |  | The beginning of instruction token.
`end_inst` |  | The end of instruction token.
`begin_tools` |  | The beginning of tools token.
`end_tools` |  | The end of tools token.
`begin_tool_results` |  | The beginning of tool results token.
`end_tool_results` |  | The end of tool results token.
`tool_calls` |  | The tool calls token.
`img` |  | The image token.
`pad` |  | The pad token.
`img_break` |  | The image break token.
`img_end` |  | The image end token.
`prefix` |  | The prefix token for FIM.
`middle` |  | The middle token for FIM.
`suffix` |  | The suffix token for FIM.
`begin_system` |  | The beginning of system prompt token.
`end_system` |  | The end of system prompt token.
`begin_tool_content` |  | The beginning of tool content token.
`args` |  | The args token.
`call_id` |  | The call id token.
`audio` |  | The audio token.
`begin_audio` |  | The beginning of audio token.
`transcribe` |  | The transcribe token.
`begin_think` |  | The beginning of think token.
`end_think` |  | The end of think token.
`streaming_pad` |  | The streaming pad token.
`streaming_word` |  | The streaming word token.
`text_to_audio` |  | The text to audio token.
`audio_to_text` |  | The audio to text token.

Examples:

    >>> unk = SpecialTokens.unk

## `Tokenized(**data)`

Bases: `MistralBase`

A tokenized `InstructRequest`.

Attributes:

Name | Type | Description
--- | --- | ---
`tokens` | `list[int]` | The token ids.
`text` | `str | None` | The text representation of the tokens.
`prefix_ids` | `list[int] | None` | The prefix ids for FIM.
`images` | `list[ndarray]` | The loaded images associated with the tokens.

Examples:

    >>> tokenized = Tokenized(tokens=[1, 2, 3], text="Hello world", prefix_ids=[1], images=[])

Source code in `pydantic/main.py`

    253
    254
    255
    256
    257
    258
    259
    260
    261
    262
    263
    264
    265
    266
    267
    268
    269
    270

    def __init__(self, /, **data: Any) -> None:
        """Create a new model by parsing and validating input data from keyword arguments.

        Raises [`ValidationError`][pydantic_core.ValidationError] if the input data cannot be
        validated to form a valid model.

        `self` is explicitly positional-only to allow `self` as a field name.
        """
        # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks
        __tracebackhide__ = True
        validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
        if self is not validated_self:
            warnings.warn(
                'A custom validator is returning a value other than `self`.\n'
                "Returning anything other than `self` from a top level model validator isn't supported when validating via `__init__`.\n"
                'See the `model_validator` docs (https://docs.pydantic.dev/latest/concepts/validators/#model-validators) for more details.',
                stacklevel=2,
            )

## `Tokenizer`

Bases: `ABC`

### `bos_id` `abstractmethod` `property`

id of the Beginning of String token.

### `eos_id` `abstractmethod` `property`

id of the End of String token.

### `file_path` `abstractmethod` `property`

The file path of the tokenizer.

### `model_settings_builder` `abstractmethod` `property`

The model settings builder, or None if unsupported by this version.

### `n_words` `abstractmethod` `property`

Vocabulary size of the tokenizer.

### `num_special_tokens` `abstractmethod` `property`

The number of special tokens of the tokenizer.

### `pad_id` `abstractmethod` `property`

id of the Pad token.

### `special_ids` `abstractmethod` `property`

Ids of the special tokens.

### `unk_id` `abstractmethod` `property`

id of the Unk token.

### `version` `abstractmethod` `property`

Get the version of the tokenizer.

### `decode(tokens, special_token_policy=SpecialTokenPolicy.IGNORE)` `abstractmethod`

Decode the token ids to a string.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`tokens` | `list[int]` | The token ids to decode. | required
`special_token_policy` | `SpecialTokenPolicy` | The policy to use for special tokens. | `IGNORE`

Returns:

Type | Description
--- | ---
`str` | The decoded string.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    270
    271
    272
    273
    274
    275
    276
    277
    278
    279
    280

    @abstractmethod
    def decode(self, tokens: list[int], special_token_policy: SpecialTokenPolicy = SpecialTokenPolicy.IGNORE) -> str:
        r"""Decode the token ids to a string.

        Args:
            tokens: The token ids to decode.
            special_token_policy: The policy to use for special tokens.

        Returns:
            The decoded string.
        """

### `encode(s, bos, eos)` `abstractmethod`

Convert a string to a list of token ids.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    266
    267
    268

    @abstractmethod
    def encode(self, s: str, bos: bool, eos: bool) -> list[int]:
        """Convert a string to a list of token ids."""

### `get_special_token(s)` `abstractmethod`

Get the id of a control token.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    282
    283
    284

    @abstractmethod
    def get_special_token(self, s: str) -> int:
        r"""Get the id of a control token."""

### `id_to_piece(token_id)` `abstractmethod`

Convert a token id to the token str.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    242
    243
    244

    @abstractmethod
    def id_to_piece(self, token_id: int) -> str:
        r"""Convert a token id to the token str."""

### `is_special(token)` `abstractmethod`

Check if token id or token str is a special token.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    286
    287
    288

    @abstractmethod
    def is_special(self, token: int | np.integer | str) -> bool:
        r"""Check if token id or token str is a special token."""

### `vocab()` `abstractmethod`

All tokens in the vocabulary as strings.

Source code in `src/mistral_common/tokens/tokenizers/base.py`

    238
    239
    240

    @abstractmethod
    def vocab(self) -> list[str]:
        r"""All tokens in the vocabulary as strings."""

## `TokenizerVersion`

Bases: `str`, `Enum`

Enum of tokenizer versions.

Allow to distinguish between different versions of the tokenizer and maintain backward compatibility.

Attributes:

Name | Type | Description
--- | --- | ---
`v1` |  | The first version of the tokenizer.
`v2` |  | The second version of the tokenizer that includes special control tokens [INST], [\INST].
`v3` |  | The third version of the tokenizer that includes improved function calling.
`v7` |  | The seventh version of the tokenizer that includes improved system prompt and function calling.
`v11` |  | The eleventh version of the tokenizer that includes improved function calling.
`v13` |  | The thirteenth version of the tokenizer that includes no call id tokenization and better prompt caching.

Examples:

    >>> version = TokenizerVersion.v1

## `UserMessagePosition`

Bases: `str`, `Enum`

Where to encode available tools