# Pixel Data - Part 1: Introduction and accessing

This is part 1 of the tutorial on using pydicom with DICOM Pixel Data. It covers:

- An introduction to DICOM pixel data

- Converting pixel data to a NumPy `ndarray`

- Customizing the conversion process

It’s assumed that you’re already familiar with the dataset basics.

Prerequisites

Installing using pip:

    python -m pip install -U pydicom numpy matplotlib pylibjpeg[all]

Installing on conda:

    conda install numpy matplotlib
    conda install -c conda-forge pydicom
    pip install pylibjpeg[all]

## Introduction

Many DICOM SOP classes contain bulk pixel data, which typically represents medical imagery or 2D slices of a 3D volume. This data is most commonly found in the Pixel Data element, however it may be in Float Pixel Data or Double Float Pixel Data instead, depending on the SOP class. The table below lists these possible pixel data containing elements, although it’s important to note that only one may be present in any given dataset.

Tag | Description | Keyword | VR
--- | --- | --- | ---
(7FE0,0008) | Float Pixel Data | FloatPixelData | OF
(7FE0,0009) | Double Pixel Data | DoubleFloatPixelData | OD
(7FE0,0010) | Pixel Data | PixelData | OB or OW

All three elements use O* VRs (such as OB and OD), which in pydicom are stored as (and should be set using) `bytes`:

    >>> from pydicom import examples
    >>> ds = examples.jpeg2k
    >>> ds.group_dataset(0x7FE0)
    (7FE0,0010) Pixel Data                          OB: Array of 152326 elements
    >>> ds.PixelData[:50]
    b'\xfe\xff\x00\xe0\x00\x00\x00\x00\xfe\xff\x00\xe0\x00\x00\x01\x00\xffO\xffQ...

If the dataset’s been written using the DICOM File Format it should have a Transfer Syntax UID element which describes how the pixel data is encoded and whether it’s undergone compression:

    >>> tsyntax = ds.file_meta.TransferSyntaxUID
    >>> tsyntax.name
    'JPEG 2000 Image Compression (Lossless Only)'
    >>> tsyntax.is_compressed
    True

In the example above the Transfer Syntax UID indicates that the pixel data has been compressed using the JPEG 2000 compression method. Other things to keep in mind with compressed transfer syntaxes are:

- Only datasets that use the Pixel Data element may be compressed

- Each frame of pixel data is compressed separately

- The compressed frames are then `encapsulated` and the encapsulated data used to set the Pixel Data value

To access the encapsulated frames you can use `get_frame()` or the `generate_frames()` iterator:

    >>> from pydicom.encaps import get_frame
    >>> frame = get_frame(ds.PixelData, 0, number_of_frames=1)
    >>> print(len(frame))
    152294

The next example uses an uncompressed Transfer Syntax UID:

    >>> ds = examples.ct
    >>> tsyntax = ds.file_meta.TransferSyntaxUID
    >>> tsyntax.name
    'Explicit VR Little Endian'
    >>> tsyntax.is_compressed
    False

The pixel data in this dataset uses little-endian byte ordering and is uncompressed. Uncompressed transfer syntaxes never use encapsulation and may use any one of the three pixel data elements, although Pixel Data is the most common.

A dataset with pixel data should always contain group `0x0028` Image Pixel module elements, which are needed to properly interpret the encoded pixel data byte stream:

    >>> ds.group_dataset(0x0028)
    (0028,0002) Samples per Pixel                   US: 1
    (0028,0004) Photometric Interpretation          CS: 'MONOCHROME2'
    (0028,0010) Rows                                US: 128
    (0028,0011) Columns                             US: 128
    (0028,0030) Pixel Spacing                       DS: [0.661468, 0.661468]
    (0028,0100) Bits Allocated                      US: 16
    (0028,0101) Bits Stored                         US: 16
    (0028,0102) High Bit                            US: 15
    (0028,0103) Pixel Representation                US: 1
    ...

An explanation of what these elements represent can be found in the glossary, but briefly, the above indicates that this dataset contains a single grayscale image with dimensions 128 x 128 and that each pixel should be interpreted as a 2-byte signed integer.

## Converting to an `ndarray`

Properly interpreting all the possible variations of a dataset’s pixel data requires a lot of specific domain knowledge, not just of DICOM but also the various JPEG compression schemes. For this reason pydicom offers a number of methods for converting the pixel data to a NumPy `ndarray`, the most high-level of which are the `pixel_array()` and `iter_pixels()` functions:

    import matplotlib.pyplot as plt

    from pydicom import examples
    from pydicom.pixels import pixel_array

    # Get an example dataset as a FileDataset instance
    ds = examples.ct

    # Convert the pixel data to an ndarray
    arr = pixel_array(ds)
    assert arr.shape == (128, 128)
    assert str(arr.dtype) == "int16"

    # Display the pixel data using matplotlib
    plt.imshow(arr, cmap="gray")
    plt.show()

This will convert the entire pixel data to an `ndarray` before using matplotlib to display it. If the dataset has multiple frames but you’re only interested in a particular one, then you can use the index parameter to return it:

    from pydicom import examples
    from pydicom.pixels import pixel_array

    # Get an example multi-frame dataset
    ds = examples.rt_dose
    assert ds.NumberOfFrames == '15'

    # Return all frames
    arr = pixel_array(ds)
    assert arr.shape == (15, 10, 10)

    # Return only the first frame
    arr = pixel_array(ds, index=0)
    assert arr.shape == (10, 10)

`iter_pixels()` can be used to iterate through either all the available frames or those specified by the indices parameter:

    from pydicom import examples
    from pydicom.pixels import iter_pixels

    # Iterate through all frames
    for arr in iter_pixels(examples.rt_dose):
        assert arr.shape == (10, 10)

    # Iterate through the first 3 even frames
    for arr in iter_pixels(examples.rt_dose, indices=[1, 3, 5]):
        assert arr.shape == (10, 10)

### Controlling decoding

The default decoding options for `pixel_array()` and `iter_pixels()` have been chosen to return the pixel data in its most commonly used form; for multi-sample data this means RGB is returned by default. Datasets with pixel data in YCbCr color space are converted using `convert_color_space()` prior to the array being returned. If you’d like to skip this conversion and return the data as found in the dataset you can pass `raw=True`:

    import matplotlib.pyplot as plt

    from pydicom import examples
    from pydicom.pixels import pixel_array

    ds = examples.ybr_color
    assert ds.PhotometricInterpretation == "YBR_FULL_422"

    ybr = pixel_array(ds, index=0, raw=True)
    rgb = pixel_array(ds, index=0)

    fig, (im1, im2) = plt.subplots(1, 2)
    im1.imshow(ybr)
    im1.set_title("Original (in YCbCr)")
    im2.imshow(rgb)
    im2.set_title("Converted (in RGB)")
    plt.show()

Further customization of the returned `ndarray` is possible by passing one or more decoding options to `pixel_array()` and `iter_pixels()`.

### Compressed transfer syntaxes

When converting datasets with a compressed transfer syntax, one or more additional packages are needed to perform the actual decompression (via their corresponding decoding plugins). By default, all available plugins will be tried and the first successful one will have its results returned:

    from pydicom import examples
    from pydicom.pixels import pixel_array

    ds = examples.jpeg2k

    # Returns the results from the first successful decoding plugin
    arr = pixel_array(ds)

If no plugins are available for the given transfer syntax due to missing dependencies you’ll get an exception:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File ".../pydicom/pixels/utils.py", line 1386, in pixel_array
        return decoder.as_array(
               ^^^^^^^^^^^^^^^^^
      File ".../pydicom/pixels/decoders/base.py", line 971, in as_array
        self._validate_plugins(decoding_plugin),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File ".../pydicom/pixels/common.py", line 249, in _validate_plugins
        raise RuntimeError(
    RuntimeError: Unable to decompress 'JPEG 2000 Image Compression (Lossless Only)' pixel data because all plugins are missing dependencies:
        gdcm - requires gdcm>=3.0.10
        pylibjpeg - requires pylibjpeg>=2.0 and pylibjpeg-openjpeg>=2.0
        pillow - requires numpy and pillow>=10.0

While the resulting `ndarray` for lossless compression methods should be identical no matter which plugin is used, there may be slight differences for lossy compression methods. To ensure consistency you can use the decoding_plugin argument to use the specified decompression plugin:

    from pydicom import examples
    from pydicom.pixels import pixel_array

    ds = examples.jpeg2k

    # Return the results from the 'pylibjpeg' decoding plugin
    arr = pixel_array(ds, decoding_plugin="pylibjpeg")

And of course if the specified plugin isn’t available you’ll get an exception:

    >>> from pydicom import examples
    >>> from pydicom.pixels import pixel_array
    >>> pixel_array(examples.jpeg2k, decoding_plugin="pillow")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File ".../pydicom/pixels/utils.py", line 1386, in pixel_array
        return decoder.as_array(
               ^^^^^^^^^^^^^^^^^
      File ".../pydicom/pixels/decoders/base.py", line 971, in as_array
        self._validate_plugins(decoding_plugin),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File ".../pydicom/pixels/common.py", line 230, in _validate_plugins
        raise RuntimeError(
    RuntimeError: Unable to decompress 'JPEG 2000 Image Compression (Lossless Only)' pixel data because the specified plugin is missing dependencies:
        pillow - requires numpy and pillow>=10.0

### Minimizing memory usage

Sometimes a dataset’s pixel data may be very large due to it having a large number of frames and you’d like to avoid having the entire thing read into memory. By passing the path to the dataset (as `str` or `pathlib.Path`) to `pixel_array()` only the Image Pixel module elements and the minimum amount of required pixel data will be loaded:

    from pydicom import examples
    from pydicom.pixels import pixel_array

    # Get the path to the 'examples.rt_dose' dataset
    path = examples.get_path("rt_dose")

    # Return the first frame of the pixel data
    arr = pixel_array(path, index=0)

The same is true for `iter_pixels()`:

    import matplotlib.pyplot as plt
    import numpy as np

    from pydicom import examples
    from pydicom.pixels import iter_pixels

    # Get the path to the 'examples.ybr_color' dataset
    path = examples.get_path("ybr_color")

    # Create an empty ndarray and use it to initialize the display
    im = plt.imshow(np.zeros((ds.Rows, ds.Columns), dtype="u1"))

    # Iterate through the frames and update the display
    for frame in iter_pixels(path):
        im.set_data(frame)
        plt.pause(0.033)

If you’re supplying a path to `pixel_array()` or `iter_pixels()` and you need access to the Image Pixel elements to perform image processing operations on the array (such as `rescale` or `windowing`) you can access them by passing an empty `Dataset` instance via the ds_out argument, or alternatively by using `dcmread()` with `stop_before_pixels=True`:

    from pydicom import Dataset, examples
    from pydicom.pixels import pixel_array, apply_rescale

    # Get the path to the 'examples.ct' dataset
    path = examples.get_path("ct")

    ds = Dataset()
    arr = pixel_array(path, ds_out=ds)

    assert ds.RescaleIntercept == "-1024.0"
    assert ds.RescaleSlope == "1.0"

    # Convert raw CT values to Hounsfield units
    hu = apply_rescale(arr, ds)

## Converting to an `ndarray` with metadata

While `pixel_array()` and `iter_pixels()` should cover most use cases, you may want more information about the returned `ndarray`, such as what color space it’s in. The `Decoder.as_array()` and `Decoder.iter_array()` methods provide mid-level access to pydicom’s pixel data decoding functionality while still handling most of the complexity of conversion to an array. More importantly, they return or yield a tuple of (`ndarray`, `dict`), where the `dict` contains metadata describing the corresponding `ndarray`.

Warning

The `Decoder` class should not be used directly, instead use the class instance returned by `get_decoder()`.

    from pydicom import examples
    from pydicom.pixels import get_decoder

    ds = examples.ybr_color
    assert ds.PhotometricInterpretation == "YBR_FULL_422"

    # Get the 'Decoder' instance required to decode the dataset's pixel data
    decoder = get_decoder(ds.file_meta.TransferSyntaxUID)

    # Converts the pixel data to an ndarray in the original color space
    arr, meta = decoder.as_array(ds, raw=True, index=0)
    assert (meta["rows"], meta["columns"], meta["samples_per_pixel"]) == arr.shape
    assert meta["photometric_interpretation"] == "YBR_FULL_422"

    # Converts the pixel data to an ndarray in RGB color space
    arr, meta = decoder.as_array(ds, index=0)
    assert meta["photometric_interpretation"] == "RGB"

This is especially useful for non-conformant datasets where the Image Pixel module elements have values that don’t match the actual pixel data (such as Number of Frames or Photometric Interpretation).

## Conclusion and next steps

In part 1 of this tutorial you’ve been introduced to DICOM’s pixel data and learned how to use pydicom to access it, convert it to an `ndarray` and how to control the conversion process. In the next part you’ll learn how to create your own pixel data from scratch.