Skip to content

# `mistral_common.tokens.tokenizers.sentencepiece`

## `SentencePieceTokenizer(model_path, tokenizer_version=None)`

Bases: `Tokenizer`

SentencePiece tokenizer.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`model_path` | `str | Path` | The path to the `SentencePiece` model. | required
`tokenizer_version` | `TokenizerVersion | None` | The version of the tokenizer. If not provided, it will be inferred from the model path. | `None`

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110

    def __init__(self, model_path: str | Path, tokenizer_version: TokenizerVersion | None = None) -> None:
        r"""Initialize the `SentencePieceTokenizer`.

        Args:
            model_path: The path to the `SentencePiece` model.
            tokenizer_version: The version of the tokenizer. If not provided, it will be inferred from the model path.
        """
        assert_sentencepiece_installed()

        self._logger = logging.getLogger(self.__class__.__name__)
        # reload tokenizer
        assert os.path.isfile(model_path), model_path
        self._model = SentencePieceProcessor(
            model_file=model_path if isinstance(model_path, str) else model_path.as_posix()
        )

        assert self._model.vocab_size() == self._model.get_piece_size()
        self._vocab = [self._model.id_to_piece(i) for i in range(self.n_words)]

        self._version: TokenizerVersion = tokenizer_version or get_spm_version(model_path, raise_deprecated=False)

        self._file_path = Path(model_path)
        super().__init__()

### `bos_id` `cached` `property`

The beginning of sentence token id.

### `eos_id` `cached` `property`

The end of sentence token id.

### `file_path` `property`

The path to the tokenizer model.

### `model_settings_builder` `property`

Always returns None as SentencePiece does not support `model_settings_builder`.

### `n_words` `property`

Vocabulary size of the tokenizer.

### `num_special_tokens` `property`

The number of special tokens of the tokenizer.

### `pad_id` `property`

The padding token id.

### `special_ids` `cached` `property`

Ids of the special tokens.

### `unk_id` `property`

The unknown token id.

### `version` `property`

The version of the tokenizer.

### `decode(tokens, special_token_policy=SpecialTokenPolicy.IGNORE)`

Decode the given list of token ids into a string.

Note

Using `special_token_policy=SpecialTokenPolicy.KEEP` will keep the special tokens and the normal tokens as SentencePiece pieces.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`tokens` | `list[int]` | The list of token ids. | required
`special_token_policy` | `SpecialTokenPolicy` | The policy to use for special tokens. | `IGNORE`

Returns:

Type | Description
--- | ---
`str` | The decoded string.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    202
    203
    204
    205
    206
    207
    208
    209
    210
    211
    212
    213
    214
    215
    216
    217
    218
    219
    220
    221
    222
    223
    224
    225
    226
    227
    228
    229

    def decode(self, tokens: list[int], special_token_policy: SpecialTokenPolicy = SpecialTokenPolicy.IGNORE) -> str:
        r"""Decode the given list of token ids into a string.

        Note:
            Using `special_token_policy=SpecialTokenPolicy.KEEP` will keep the special tokens and the normal tokens as
            SentencePiece pieces.

        Args:
            tokens: The list of token ids.
            special_token_policy: The policy to use for special tokens.

        Returns:
            The decoded string.
        """
        try:
            special_token_policy = SpecialTokenPolicy(special_token_policy)
        except ValueError as e:
            valid = ", ".join(repr(p.value) for p in SpecialTokenPolicy)
            raise ValueError(
                f"Invalid `special_token_policy` {special_token_policy!r}. Expected one of: {valid}."
            ) from e

        if special_token_policy == SpecialTokenPolicy.IGNORE:
            decoded = self._model.decode(tokens)
            assert isinstance(decoded, str), f"Sentencepiece model decoded a {type(decoded)}, not a string."
            return decoded

        return self._decode_with_special_tokens(tokens, special_token_policy)

### `encode(s, bos, eos)`

Encode the given string into a list of token ids.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`s` | `str` | The string to encode. | required
`bos` | `bool` | Whether to add the beginning of sentence token. | required
`eos` | `bool` | Whether to add the end of sentence token. | required

Returns:

Type | Description
--- | ---
`list[int]` | The list of token ids.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    194
    195
    196
    197
    198
    199
    200

    def encode(self, s: str, bos: bool, eos: bool) -> list[int]:
        r"""Encode the given string into a list of token ids.

        Args:
            s: The string to encode.
            bos: Whether to add the beginning of sentence token.
            eos: Whether to add the end of sentence token.

        Returns:
            The list of token ids.
        """
        assert isinstance(s, str)
        t: list[int] = self._model.encode(s)
        if bos:
            t = [self.bos_id, *t]
        if eos:
            t = [*t, self.eos_id]
        return t

### `get_special_token(s)`

Get the special token for the given string.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    129
    130
    131

    def get_special_token(self, s: str) -> int:
        r"""Get the special token for the given string."""
        return self._model.piece_to_id(s)  # type: ignore

### `id_to_piece(token_id)`

Convert the given token id to a token piece.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    231
    232
    233

    def id_to_piece(self, token_id: int) -> str:
        r"""Convert the given token id to a token piece."""
        return self._model.id_to_piece(token_id)  # type: ignore

### `is_special(token)`

Return `True` if the passed `token` is a special token.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    161
    162
    163
    164
    165
    166
    167
    168
    169

    def is_special(self, token: int | np.integer | str) -> bool:
        """Return `True` if the passed `token` is a special token."""
        if isinstance(token, (int, np.integer)):
            return self._model.IsControl(int(token))  # type: ignore
        elif isinstance(token, str):
            token_int = self._model.piece_to_id(token)
            return self._model.IsControl(token_int)  # type: ignore
        else:
            raise TypeError(f"Expected int or str, got {type(token).__name__}")

### `vocab()`

Get all tokens in the vocabulary as strings.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    147
    148
    149

    def vocab(self) -> list[str]:
        r"""Get all tokens in the vocabulary as strings."""
        return self._vocab

## `get_image_config(tokenizer_filename)`

Get the image config from the tokenizer filename.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82

    def get_image_config(tokenizer_filename: str | Path) -> ImageConfig | None:
        r"""Get the image config from the tokenizer filename."""
        tokenizer_filename = str(tokenizer_filename)

        _version_str = tokenizer_filename.split(".")[-1]
        if _version_str == "model" or "m" not in _version_str:
            return None

        _mm_version_str = "m" + _version_str.split("m")[-1]

        if _mm_version_str not in MultiModalVersion.__members__:
            raise TokenizerException(f"Unrecognized tokenizer filename: {tokenizer_filename}")

        return MultiModalVersion(_mm_version_str).config

## `get_spm_version(tokenizer_filename, raise_deprecated=False)`

Get the version of the tokenizer from the filename.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66

    def get_spm_version(tokenizer_filename: str | Path, raise_deprecated: bool = False) -> TokenizerVersion:
        r"""Get the version of the tokenizer from the filename."""
        tokenizer_filename = str(tokenizer_filename)

        _version_str = tokenizer_filename.split(".")[-1]
        if _version_str != "model":  # filter tokenizer_filename == "/path/to/tokenizer.model" case
            _version_str = _version_str.split("m")[0]

        if _version_str == "model":
            if raise_deprecated:
                raise TokenizerException(f"Make sure to rename your tokenizer file to end with {tokenizer_filename}.v1.")

            # tokenizer.model => tokenizer.model.v1
            return TokenizerVersion("v1")

        if _version_str not in TokenizerVersion.__members__:
            raise TokenizerException(f"Unrecognized tokenizer filename: {tokenizer_filename}")

        return TokenizerVersion(_version_str)

## `is_sentencepiece(path)`

Check if the given path is a SentencePiece model.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    36
    37
    38
    39
    40
    41
    42
    43
    44
    45

    def is_sentencepiece(path: str | Path) -> bool:
        r"""Check if the given path is a SentencePiece model."""
        if isinstance(path, str):
            path = Path(path)

        instruct_versions = list(TokenizerVersion.__members__)
        mm_versions = list(MultiModalVersion.__members__) + [""]  # allow no mm version
        suffixes = [f".model.{v}{m}" for v in instruct_versions for m in mm_versions] + [".model"]

        return path.is_file() and any(path.name.endswith(suffix) for suffix in suffixes)

## `is_sentencepiece_tokenizer(tokenizer)`

Returns whether the tokenizer is a SentencePieceTokenizer.

Source code in `src/mistral_common/tokens/tokenizers/sentencepiece.py`

    273
    274
    275

    def is_sentencepiece_tokenizer(tokenizer: Tokenizer) -> TypeGuard[SentencePieceTokenizer]:
        r"""Returns whether the tokenizer is a SentencePieceTokenizer."""
        return isinstance(tokenizer, SentencePieceTokenizer)