# Working with pandas

One of the most important features of xarray is the ability to convert to and from `pandas` objects to interact with the rest of the PyData ecosystem. For example, for plotting labeled data, we highly recommend using the visualization built in to pandas itself or provided by the pandas aware libraries such as Seaborn.

## Hierarchical and tidy data

Tabular data is easiest to work with when it meets the criteria for tidy data:

- Each column holds a different variable.

- Each rows holds a different observation.

In this “tidy data” format, we can represent any `Dataset` and `DataArray` in terms of `DataFrame` and `Series`, respectively (and vice-versa). The representation works by flattening non-coordinates to 1D, and turning the tensor product of coordinate indexes into a `pandas.MultiIndex`.

### Dataset and DataFrame

To convert any dataset to a `DataFrame` in tidy form, use the `Dataset.to_dataframe()` method:

    In [1]: ds = xr.Dataset(
       ...:     {"foo": (("x", "y"), np.random.randn(2, 3))},
       ...:     coords={
       ...:         "x": [10, 20],
       ...:         "y": ["a", "b", "c"],
       ...:         "along_x": ("x", np.random.randn(2)),
       ...:         "scalar": 123,
       ...:     },
       ...: )
       ...:

    In [2]: ds
    Out[2]:
    <xarray.Dataset>
    Dimensions:  (x: 2, y: 3)
    Coordinates:
      * x        (x) int64 10 20
      * y        (y) <U1 'a' 'b' 'c'
        along_x  (x) float64 0.1192 -1.044
        scalar   int64 123
    Data variables:
        foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732

    In [3]: df = ds.to_dataframe()

    In [4]: df
    Out[4]:
               foo   along_x  scalar
    x  y
    10 a  0.469112  0.119209     123
       b -0.282863  0.119209     123
       c -1.509059  0.119209     123
    20 a -1.135632 -1.044236     123
       b  1.212112 -1.044236     123
       c -0.173215 -1.044236     123

We see that each variable and coordinate in the Dataset is now a column in the DataFrame, with the exception of indexes which are in the index. To convert the `DataFrame` to any other convenient representation, use `DataFrame` methods like `reset_index()`, `stack()` and `unstack()`.

For datasets containing dask arrays where the data should be lazily loaded, see the `Dataset.to_dask_dataframe()` method.

To create a `Dataset` from a `DataFrame`, use the `Dataset.from_dataframe()` class method or the equivalent `pandas.DataFrame.to_xarray()` method:

    In [5]: xr.Dataset.from_dataframe(df)
    Out[5]:
    <xarray.Dataset>
    Dimensions:  (x: 2, y: 3)
    Coordinates:
      * x        (x) int64 10 20
      * y        (y) object 'a' 'b' 'c'
    Data variables:
        foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
        along_x  (x, y) float64 0.1192 0.1192 0.1192 -1.044 -1.044 -1.044
        scalar   (x, y) int64 123 123 123 123 123 123

Notice that that dimensions of variables in the `Dataset` have now expanded after the round-trip conversion to a `DataFrame`. This is because every object in a `DataFrame` must have the same indices, so we need to broadcast the data of each array to the full size of the new `MultiIndex`.

Likewise, all the coordinates (other than indexes) ended up as variables, because pandas does not distinguish non-index coordinates.

### DataArray and Series

`DataArray` objects have a complementary representation in terms of a `Series`. Using a Series preserves the `Dataset` to `DataArray` relationship, because `DataFrames` are dict-like containers of `Series`. The methods are very similar to those for working with DataFrames:

    In [6]: s = ds["foo"].to_series()

    In [7]: s
    Out[7]:
    x   y
    10  a    0.469112
        b   -0.282863
        c   -1.509059
    20  a   -1.135632
        b    1.212112
        c   -0.173215
    Name: foo, dtype: float64

    # or equivalently, with Series.to_xarray()
    In [8]: xr.DataArray.from_series(s)
    Out[8]:
    <xarray.DataArray 'foo' (x: 2, y: 3)>
    array([[ 0.469, -0.283, -1.509],
           [-1.136,  1.212, -0.173]])
    Coordinates:
      * x        (x) int64 10 20
      * y        (y) object 'a' 'b' 'c'

Both the `from_series` and `from_dataframe` methods use reindexing, so they work even if not the hierarchical index is not a full tensor product:

    In [9]: s[::2]
    Out[9]:
    x   y
    10  a    0.469112
        c   -1.509059
    20  b    1.212112
    Name: foo, dtype: float64

    In [10]: s[::2].to_xarray()
    Out[10]:
    <xarray.DataArray 'foo' (x: 2, y: 3)>
    array([[ 0.469,    nan, -1.509],
           [   nan,  1.212,    nan]])
    Coordinates:
      * x        (x) int64 10 20
      * y        (y) object 'a' 'b' 'c'

## Multi-dimensional data

Tidy data is great, but it sometimes you want to preserve dimensions instead of automatically stacking them into a `MultiIndex`.

`DataArray.to_pandas()` is a shortcut that lets you convert a DataArray directly into a pandas object with the same dimensionality, if available in pandas (i.e., a 1D array is converted to a `Series` and 2D to `DataFrame`):

    In [11]: arr = xr.DataArray(
       ....:     np.random.randn(2, 3), coords=[("x", [10, 20]), ("y", ["a", "b", "c"])]
       ....: )
       ....:

    In [12]: df = arr.to_pandas()

    In [13]: df
    Out[13]:
    y          a         b         c
    x
    10 -0.861849 -2.104569 -0.494929
    20  1.071804  0.721555 -0.706771

To perform the inverse operation of converting any pandas objects into a data array with the same shape, simply use the `DataArray` constructor:

    In [14]: xr.DataArray(df)
    Out[14]:
    <xarray.DataArray (x: 2, y: 3)>
    array([[-0.862, -2.105, -0.495],
           [ 1.072,  0.722, -0.707]])
    Coordinates:
      * x        (x) int64 10 20
      * y        (y) object 'a' 'b' 'c'

Both the `DataArray` and `Dataset` constructors directly convert pandas objects into xarray objects with the same shape. This means that they preserve all use of multi-indexes:

    In [15]: index = pd.MultiIndex.from_arrays(
       ....:     [["a", "a", "b"], [0, 1, 2]], names=["one", "two"]
       ....: )
       ....:

    In [16]: df = pd.DataFrame({"x": 1, "y": 2}, index=index)

    In [17]: ds = xr.Dataset(df)

    In [18]: ds
    Out[18]:
    <xarray.Dataset>
    Dimensions:  (dim_0: 3)
    Coordinates:
      * dim_0    (dim_0) MultiIndex
      - one      (dim_0) object 'a' 'a' 'b'
      - two      (dim_0) int64 0 1 2
    Data variables:
        x        (dim_0) int64 1 1 1
        y        (dim_0) int64 2 2 2

However, you will need to set dimension names explicitly, either with the `dims` argument on in the `DataArray` constructor or by calling `rename` on the new object.

## Transitioning from pandas.Panel to xarray

`Panel`, pandas’ data structure for 3D arrays, was always a second class data structure compared to the Series and DataFrame. To allow pandas developers to focus more on its core functionality built around the DataFrame, pandas removed `Panel` in favor of directing users who use multi-dimensional arrays to xarray.

Xarray has most of `Panel`’s features, a more explicit API (particularly around indexing), and the ability to scale to >3 dimensions with the same interface.

As discussed elsewhere in the docs, there are two primary data structures in xarray: `DataArray` and `Dataset`. You can imagine a `DataArray` as a n-dimensional pandas `Series` (i.e. a single typed array), and a `Dataset` as the `DataFrame` equivalent (i.e. a dict of aligned `DataArray` objects).

So you can represent a Panel, in two ways:

- As a 3-dimensional `DataArray`,

- Or as a `Dataset` containing a number of 2-dimensional DataArray objects.

Let’s take a look:

    In [19]: data = np.random.RandomState(0).rand(2, 3, 4)

    In [20]: items = list("ab")

    In [21]: major_axis = list("mno")

    In [22]: minor_axis = pd.date_range(start="2000", periods=4, name="date")

With old versions of pandas (prior to 0.25), this could stored in a `Panel`:

    In [23]: pd.Panel(data, items, major_axis, minor_axis)
    Out[23]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
    Items axis: a to b
    Major_axis axis: m to o
    Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-04 00:00:00

To put this data in a `DataArray`, write:

    In [24]: array = xr.DataArray(data, [items, major_axis, minor_axis])

    In [25]: array
    Out[25]:
    <xarray.DataArray (dim_0: 2, dim_1: 3, date: 4)>
    array([[[0.549, 0.715, 0.603, 0.545],
            [0.424, 0.646, 0.438, 0.892],
            [0.964, 0.383, 0.792, 0.529]],

           [[0.568, 0.926, 0.071, 0.087],
            [0.02 , 0.833, 0.778, 0.87 ],
            [0.979, 0.799, 0.461, 0.781]]])
    Coordinates:
      * dim_0    (dim_0) <U1 'a' 'b'
      * dim_1    (dim_1) <U1 'm' 'n' 'o'
      * date     (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04

As you can see, there are three dimensions (each is also a coordinate). Two of the axes of were unnamed, so have been assigned `dim_0` and `dim_1` respectively, while the third retains its name `date`.

You can also easily convert this data into `Dataset`:

    In [26]: array.to_dataset(dim="dim_0")
    Out[26]:
    <xarray.Dataset>
    Dimensions:  (dim_1: 3, date: 4)
    Coordinates:
      * dim_1    (dim_1) <U1 'm' 'n' 'o'
      * date     (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
    Data variables:
        a        (dim_1, date) float64 0.5488 0.7152 0.6028 ... 0.3834 0.7917 0.5289
        b        (dim_1, date) float64 0.568 0.9256 0.07104 ... 0.7992 0.4615 0.7805

Here, there are two data variables, each representing a DataFrame on panel’s `items` axis, and labeled as such. Each variable is a 2D array of the respective values along the `items` dimension.

While the xarray docs are relatively complete, a few items stand out for Panel users:

- A DataArray’s data is stored as a numpy array, and so can only contain a single type. As a result, a Panel that contains `DataFrame` objects with multiple types will be converted to `dtype=object`. A `Dataset` of multiple `DataArray` objects each with its own dtype will allow original types to be preserved.

- Indexing is similar to pandas, but more explicit and leverages xarray’s naming of dimensions.

- Because of those features, making much higher dimensional data is very practical.

- Variables in `Dataset` objects can use a subset of its dimensions. For example, you can have one dataset with Person x Score x Time, and another with Person x Score.

- You can use coordinates are used for both dimensions and for variables which _label_ the data variables, so you could have a coordinate Age, that labelled the Person dimension of a Dataset of Person x Score x Time.

While xarray may take some getting used to, it’s worth it! If anything is unclear, please post an issue on GitHub or StackOverflow, and we’ll endeavor to respond to the specific case or improve the general docs.