# Internal Design

This page gives an overview of the internal design of xarray.

In totality, the Xarray project defines 4 key data structures. In order of increasing complexity, they are:

- `xarray.Variable`,

- `xarray.DataArray`,

- `xarray.Dataset`,

- `xarray.DataTree`.

The user guide lists only `xarray.DataArray` and `xarray.Dataset`, but `Variable` is the fundamental object internally, and `DataTree` is a natural generalisation of `xarray.Dataset`.

Note

Our Development roadmap includes plans to document `Variable` as fully public API.

Internally private lazy indexing classes are used to avoid loading more data than necessary, and flexible indexes classes (derived from `Index`) provide performant label-based lookups.

## Data Structures

The Data Structures page in the user guide explains the basics and concentrates on user-facing behavior, whereas this section explains how xarray’s data structure classes actually work internally.

### Variable Objects

The core internal data structure in xarray is the `Variable`, which is used as the basic building block behind xarray’s `Dataset`, `DataArray` types. A `Variable` consists of:

- `dims`: A tuple of dimension names.

- `data`: The N-dimensional array (typically a NumPy or Dask array) storing the Variable’s data. It must have the same number of dimensions as the length of `dims`.

- `attrs`: A dictionary of metadata associated with this array. By convention, xarray’s built-in operations never use this metadata.

- `encoding`: Another dictionary used to store information about how these variable’s data is represented on disk. See Reading encoded data for more details.

`Variable` has an interface similar to NumPy arrays, but extended to make use of named dimensions. For example, it uses `dim` in preference to an `axis` argument for methods like `mean`, and supports Broadcasting by dimension name.

However, unlike `Dataset` and `DataArray`, the basic `Variable` does not include coordinate labels along each axis.

`Variable` is public API, but because of its incomplete support for labeled data, it is mostly intended for advanced uses, such as in xarray itself, for writing new backends, or when creating custom indexes. You can access the variable objects that correspond to xarray objects via the (readonly) `Dataset.variables` and `DataArray.variable` attributes.

### DataArray Objects

The simplest data structure used by most users is `DataArray`. A `DataArray` is a composite object consisting of multiple `Variable` objects which store related data.

A single `Variable` is referred to as the “data variable”, and stored under the `variable`` attribute. A `DataArray` inherits all of the properties of this data variable, i.e. `dims`, `data`, `attrs` and `encoding`, all of which are implemented by forwarding on to the underlying `Variable` object.

In addition, a `DataArray` stores additional `Variable` objects stored in a dict under the private `_coords` attribute, each of which is referred to as a “Coordinate Variable”. These coordinate variable objects are only allowed to have `dims` that are a subset of the data variable’s `dims`, and each dim has a specific length. This means that the full `size` of the dataarray can be represented by a dictionary mapping dimension names to integer sizes. The underlying data variable has this exact same size, and the attached coordinate variables have sizes which are some subset of the size of the data variable. Another way of saying this is that all coordinate variables must be “alignable” with the data variable.

When a coordinate is accessed by the user (e.g. via the dict-like `__getitem__` syntax), then a new `DataArray` is constructed by finding all coordinate variables that have compatible dimensions and re-attaching them before the result is returned. This is why most users never see the `Variable` class underlying each coordinate variable - it is always promoted to a `DataArray` before returning.

Lookups are performed by special `Index` objects, which are stored in a dict under the private `_indexes` attribute. Indexes must be associated with one or more coordinates, and essentially act by translating a query given in physical coordinate space (typically via the `sel()` method) into a set of integer indices in array index space that can be used to index the underlying n-dimensional array-like `data`. Indexing in array index space (typically performed via the `isel()` method) does not require consulting an `Index` object.

Finally a `DataArray` defines a `name` attribute, which refers to its data variable but is stored on the wrapping `DataArray` class. The `name` attribute is primarily used when one or more `DataArray` objects are promoted into a `Dataset` (e.g. via `to_dataset()`). Note that the underlying `Variable` objects are all unnamed, so they can always be referred to uniquely via a dict-like mapping.

### Dataset Objects

The `Dataset` class is a generalization of the `DataArray` class that can hold multiple data variables. Internally all data variables and coordinate variables are stored under a single `variables` dict, and coordinates are specified by storing their names in a private `_coord_names` dict.

The dataset’s `dims` are the set of all dims present across any variable, but (similar to in dataarrays) coordinate variables cannot have a dimension that is not present on any data variable.

When a data variable or coordinate variable is accessed, a new `DataArray` is again constructed from all compatible coordinates before returning.

Note

The way that selecting a variable from a `DataArray` or `Dataset` actually involves internally wrapping the `Variable` object back up into a `DataArray`/`Dataset` is the primary reason we recommend against subclassing Xarray objects. The main problem it creates is that we currently cannot easily guarantee that for example selecting a coordinate variable from your `SubclassedDataArray` would return an instance of `SubclassedDataArray` instead of just an `xarray.DataArray`. See GH issue for more details.

## Lazy Indexing Classes

### Lazy Loading

If we open a `Variable` object from disk using `open_dataset()` we can see that the actual values of the array wrapped by the data variable are not displayed.

    In [1]: da = xr.tutorial.open_dataset("air_temperature")["air"]
    ---------------------------------------------------------------------------
    PermissionError                           Traceback (most recent call last)
    File /usr/lib/python3/dist-packages/pooch/utils.py:262, in make_local_storage(path, env)
        258 if action == "create":
        259     # When running in parallel, it's possible that multiple jobs will
        260     # try to create the path at the same time. Use exist_ok to avoid
        261     # raising an error.
    --> 262     os.makedirs(path, exist_ok=True)
        263 else:

    File <frozen os>:217, in makedirs(name, mode, exist_ok)

    File <frozen os>:217, in makedirs(name, mode, exist_ok)

    File <frozen os>:227, in makedirs(name, mode, exist_ok)

    PermissionError: [Errno 13] Permission denied: '/sbuild-nonexistent'

    The above exception was the direct cause of the following exception:

    PermissionError                           Traceback (most recent call last)
    Cell In[1], line 1
    ----> 1 da = xr.tutorial.open_dataset("air_temperature")["air"]

    File /build/python-xarray-69jxB9/python-xarray-2025.01.2/xarray/tutorial.py:165, in open_dataset(name, cache, cache_dir, engine, **kws)
        162 downloader = pooch.HTTPDownloader(headers=headers)
        164 # retrieve the file
    --> 165 filepath = pooch.retrieve(
        166     url=url, known_hash=None, path=cache_dir, downloader=downloader
        167 )
        168 ds = _open_dataset(filepath, engine=engine, **kws)
        169 if not cache:

    File /usr/lib/python3/dist-packages/pooch/core.py:227, in retrieve(url, known_hash, fname, path, processor, downloader, progressbar)
        222 action, verb = download_action(full_path, known_hash)
        224 if action in ("download", "update"):
        225     # We need to write data, so create the local data directory if it
        226     # doesn't already exist.
    --> 227     make_local_storage(path)
        229     get_logger().info(
        230         "%s data from '%s' to file '%s'.",
        231         verb,
        232         url,
        233         str(full_path),
        234     )
        236     if downloader is None:

    File /usr/lib/python3/dist-packages/pooch/utils.py:276, in make_local_storage(path, env)
        272 if env is not None:
        273     message.append(
        274         f"Use environment variable '{env}' to specify a different location."
        275     )
    --> 276 raise PermissionError(" ".join(message)) from error

    PermissionError: [Errno 13] Permission denied: '/sbuild-nonexistent' | Pooch could not create data cache folder '/sbuild-nonexistent/.cache/xarray_tutorial_data'. Will not be able to download data files.

    In [2]: var = da.variable
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    Cell In[2], line 1
    ----> 1 var = da.variable

    AttributeError: module 'dask.array' has no attribute 'variable'

    In [3]: var
    Out[3]:
    <xarray.Variable (x: 10)> Size: 80B
    array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
    Attributes:
        scale_factor:  10
        add_offset:    2

We can see the size, and the dtype of the underlying array, but not the actual values. This is because the values have not yet been loaded.

If we look at the private attribute `_data()` containing the underlying array object, we see something interesting:

    In [4]: var._data
    Out[4]: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])

You’re looking at one of xarray’s internal Lazy Indexing Classes. These powerful classes are hidden from the user, but provide important functionality.

Calling the public `data` property loads the underlying array into memory.

    In [5]: var.data
    Out[5]: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])

This array is now cached, which we can see by accessing the private attribute again:

    In [6]: var._data
    Out[6]: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])

### Lazy Indexing

The purpose of these lazy indexing classes is to prevent more data being loaded into memory than is necessary for the subsequent analysis, by deferring loading data until after indexing is performed.

Let’s open the data from disk again.

    In [7]: da = xr.tutorial.open_dataset("air_temperature")["air"]
    ---------------------------------------------------------------------------
    PermissionError                           Traceback (most recent call last)
    File /usr/lib/python3/dist-packages/pooch/utils.py:262, in make_local_storage(path, env)
        258 if action == "create":
        259     # When running in parallel, it's possible that multiple jobs will
        260     # try to create the path at the same time. Use exist_ok to avoid
        261     # raising an error.
    --> 262     os.makedirs(path, exist_ok=True)
        263 else:

    File <frozen os>:217, in makedirs(name, mode, exist_ok)

    File <frozen os>:217, in makedirs(name, mode, exist_ok)

    File <frozen os>:227, in makedirs(name, mode, exist_ok)

    PermissionError: [Errno 13] Permission denied: '/sbuild-nonexistent'

    The above exception was the direct cause of the following exception:

    PermissionError                           Traceback (most recent call last)
    Cell In[7], line 1
    ----> 1 da = xr.tutorial.open_dataset("air_temperature")["air"]

    File /build/python-xarray-69jxB9/python-xarray-2025.01.2/xarray/tutorial.py:165, in open_dataset(name, cache, cache_dir, engine, **kws)
        162 downloader = pooch.HTTPDownloader(headers=headers)
        164 # retrieve the file
    --> 165 filepath = pooch.retrieve(
        166     url=url, known_hash=None, path=cache_dir, downloader=downloader
        167 )
        168 ds = _open_dataset(filepath, engine=engine, **kws)
        169 if not cache:

    File /usr/lib/python3/dist-packages/pooch/core.py:227, in retrieve(url, known_hash, fname, path, processor, downloader, progressbar)
        222 action, verb = download_action(full_path, known_hash)
        224 if action in ("download", "update"):
        225     # We need to write data, so create the local data directory if it
        226     # doesn't already exist.
    --> 227     make_local_storage(path)
        229     get_logger().info(
        230         "%s data from '%s' to file '%s'.",
        231         verb,
        232         url,
        233         str(full_path),
        234     )
        236     if downloader is None:

    File /usr/lib/python3/dist-packages/pooch/utils.py:276, in make_local_storage(path, env)
        272 if env is not None:
        273     message.append(
        274         f"Use environment variable '{env}' to specify a different location."
        275     )
    --> 276 raise PermissionError(" ".join(message)) from error

    PermissionError: [Errno 13] Permission denied: '/sbuild-nonexistent' | Pooch could not create data cache folder '/sbuild-nonexistent/.cache/xarray_tutorial_data'. Will not be able to download data files.

    In [8]: var = da.variable
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    Cell In[8], line 1
    ----> 1 var = da.variable

    AttributeError: module 'dask.array' has no attribute 'variable'

Now, notice how even after subsetting the data has does not get loaded:

    In [9]: var.isel(time=0)
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[9], line 1
    ----> 1 var.isel(time=0)

    File /build/python-xarray-69jxB9/python-xarray-2025.01.2/xarray/core/variable.py:1078, in Variable.isel(self, indexers, missing_dims, **indexers_kwargs)
       1054 """Return a new array indexed along the specified dimension(s).
       1055
       1056 Parameters
       (...)
       1074     indexer, in which case the data will be a copy.
       1075 """
       1076 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "isel")
    -> 1078 indexers = drop_dims_from_indexers(indexers, self.dims, missing_dims)
       1080 key = tuple(indexers.get(dim, slice(None)) for dim in self.dims)
       1081 return self[key]

    File /build/python-xarray-69jxB9/python-xarray-2025.01.2/xarray/core/utils.py:802, in drop_dims_from_indexers(indexers, dims, missing_dims)
        800     invalid = indexers.keys() - set(dims)
        801     if invalid:
    --> 802         raise ValueError(
        803             f"Dimensions {invalid} do not exist. Expected one or more of {dims}"
        804         )
        806     return indexers
        808 elif missing_dims == "warn":
        809     # don't modify input

    ValueError: Dimensions {'time'} do not exist. Expected one or more of ('x',)

The shape has changed, but the values are still not shown.

Looking at the private attribute again shows how this indexing information was propagated via the hidden lazy indexing classes:

    In [10]: var.isel(time=0)._data
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[10], line 1
    ----> 1 var.isel(time=0)._data

    File /build/python-xarray-69jxB9/python-xarray-2025.01.2/xarray/core/variable.py:1078, in Variable.isel(self, indexers, missing_dims, **indexers_kwargs)
       1054 """Return a new array indexed along the specified dimension(s).
       1055
       1056 Parameters
       (...)
       1074     indexer, in which case the data will be a copy.
       1075 """
       1076 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "isel")
    -> 1078 indexers = drop_dims_from_indexers(indexers, self.dims, missing_dims)
       1080 key = tuple(indexers.get(dim, slice(None)) for dim in self.dims)
       1081 return self[key]

    File /build/python-xarray-69jxB9/python-xarray-2025.01.2/xarray/core/utils.py:802, in drop_dims_from_indexers(indexers, dims, missing_dims)
        800     invalid = indexers.keys() - set(dims)
        801     if invalid:
    --> 802         raise ValueError(
        803             f"Dimensions {invalid} do not exist. Expected one or more of {dims}"
        804         )
        806     return indexers
        808 elif missing_dims == "warn":
        809     # don't modify input

    ValueError: Dimensions {'time'} do not exist. Expected one or more of ('x',)

Note

Currently only certain indexing operations are lazy, not all array operations. For discussion of making all array operations lazy see GH issue #5081.

### Lazy Dask Arrays

Note that xarray’s implementation of Lazy Indexing classes is completely separate from how `dask.array.Array` objects evaluate lazily. Dask-backed xarray objects delay almost all operations until `compute()` is called (either explicitly or implicitly via `plot()` for example). The exceptions to this laziness are operations whose output shape is data-dependent, such as when calling `where()`.