Skip to content

# `mistral_common.tokens.tokenizers.image`

## `ImageConfig(image_patch_size, max_image_size, spatial_merge_size=1)` `dataclass`

Configuration for the image tokenizers.

## `ImageEncoder(image_config, special_ids)`

Image encoder for the image tokenizer.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`image_config` | `ImageConfig` | Configuration for the image tokenizer. | required
`special_ids` | `SpecialImageIDs` | Special image tokens ids. | required

Source code in `src/mistral_common/tokens/tokenizers/image.py`

    164
    165
    166
    167
    168
    169
    170
    171
    172

    def __init__(self, image_config: ImageConfig, special_ids: SpecialImageIDs) -> None:
        r"""Initialize the image encoder.

        Args:
            image_config: Configuration for the image tokenizer.
            special_ids: Special image tokens ids.
        """
        self.image_config = image_config
        self.special_ids = special_ids

### `__call__(content)`

Converts an image chunk to an image encoding.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`content` | `ImageChunk | ImageURLChunk` | image chunk to be converted. | required

Returns:

Type | Description
--- | ---
`ImageEncoding` | Image encoding.

Source code in `src/mistral_common/tokens/tokenizers/image.py`

    196
    197
    198
    199
    200
    201
    202
    203
    204
    205
    206
    207
    208
    209
    210
    211
    212
    213
    214
    215
    216

    def __call__(self, content: ImageChunk | ImageURLChunk) -> ImageEncoding:
        r"""Converts an image chunk to an image encoding.

        Args:
            content: image chunk to be converted.

        Returns:
            Image encoding.
        """
        image = image_from_chunk(content)
        w, h = self._image_to_num_tokens(image)
        assert w > 0
        assert h > 0
        image_tokens = ([self.special_ids.img] * w + [self.special_ids.img_break]) * h
        image_tokens[-1] = self.special_ids.img_end
        new_image_size = (
            w * self.image_config.image_patch_size * self.image_config.spatial_merge_size,
            h * self.image_config.image_patch_size * self.image_config.spatial_merge_size,
        )
        processed_image = transform_image(image, new_image_size)
        return ImageEncoding(tokens=image_tokens, image=processed_image)

## `ImageEncoding(tokens, image)` `dataclass`

A tokenized image.

Attributes:

Name | Type | Description
--- | --- | ---
`tokens` | `list[int]` | The token ids.
`image` | `ndarray` | The image as a numpy array.

Examples:

    >>> import numpy as np
    >>> image_encoding = ImageEncoding(tokens=[1, 2, 3], image=np.array([[0., 0.5, 1.]]))

## `MultiModalVersion`

Bases: `str`, `Enum`

Version of the image tokenizer.

## `SpecialImageIDs(img, img_break, img_end)` `dataclass`

Special image tokens ids.

Attributes:

Name | Type | Description
--- | --- | ---
`img` | `int` | The image token id.
`img_break` | `int` | The image break token id.
`img_end` | `int` | The image end token id.

Examples:

    >>> special_image_ids = SpecialImageIDs(img=1, img_break=2, img_end=3)

## `image_from_chunk(chunk)`

Get a serializable image from a chunk.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`chunk` | `ImageURLChunk | ImageChunk` | The chunk to get the image from. | required

Returns:

Type | Description
--- | ---
`SerializableImage` | The image as a PIL Image object.

Source code in `src/mistral_common/tokens/tokenizers/image.py`

    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76

    def image_from_chunk(chunk: ImageURLChunk | ImageChunk) -> SerializableImage:
        r"""Get a serializable image from a chunk.

        Args:
            chunk: The chunk to get the image from.

        Returns:
            The image as a PIL Image object.
        """
        if isinstance(chunk, ImageChunk):
            return chunk.image
        if chunk.get_url().startswith("data:image"):
            data = chunk.get_url().split(",")[1]
            image_data = base64.b64decode(data)
            return Image.open(BytesIO(image_data))
        if chunk.get_url().startswith("file"):
            return Image.open(open(chunk.get_url().replace("file://", ""), "rb"))
        if chunk.get_url().startswith("http"):
            return download_image(chunk.get_url())

        raise RuntimeError(f"Unsupported image url scheme {chunk.get_url()}")

## `normalize(np_image, mean, std)`

Normalize a tensor image with mean and standard deviation.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`np_image` | `ndarray` | Image to be normalized. | required
`mean` | `tuple[float, float, float]` | Mean for each channel. | required
`std` | `tuple[float, float, float]` | Standard deviation for each channel. | required

Returns:

Type | Description
--- | ---
`ndarray` | Normalized image with shape (C, H, W).

Source code in `src/mistral_common/tokens/tokenizers/image.py`

    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142

    def normalize(
        np_image: np.ndarray,
        mean: tuple[float, float, float],
        std: tuple[float, float, float],
    ) -> np.ndarray:
        r"""Normalize a tensor image with mean and standard deviation.

        Args:
            np_image: Image to be normalized.
            mean: Mean for each channel.
            std: Standard deviation for each channel.

        Returns:
            Normalized image with shape (C, H, W).
        """
        np_image = np_image / 255.0

        assert len(np_image.shape) == 3, f"{np_image.shape=}"
        assert np_image.shape[2] == len(mean) == len(std), f"{np_image.shape=}, {mean=}, {std=}"

        np_image = (np_image - mean) / std

        return np_image.transpose(2, 0, 1)

## `transform_image(image, new_size)`

Transform an image to a numpy array with the given size.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`image` | `Image` | Image to be transformed. | required
`new_size` | `tuple[int, int]` | New size of the image. | required

Returns:

Type | Description
--- | ---
`ndarray` | Transformed image with shape (C, H, W).

Source code in `src/mistral_common/tokens/tokenizers/image.py`

    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158

    def transform_image(image: Image.Image, new_size: tuple[int, int]) -> np.ndarray:
        r"""Transform an image to a numpy array with the given size.

        Args:
            image: Image to be transformed.
            new_size: New size of the image.

        Returns:
            Transformed image with shape (C, H, W).
        """
        assert_opencv_installed()

        np_image = cv2.resize(np.array(_convert_to_rgb(image), dtype=np.float32), new_size, interpolation=cv2.INTER_CUBIC)
        return normalize(np_image, DATASET_MEAN, DATASET_STD)