# Sparse data structures

We have implemented “sparse” versions of Series, DataFrame, and Panel. These are not sparse in the typical “mostly 0”. You can view these objects as being “compressed” where any data matching a specific value (NaN/missing by default, though any value can be chosen) is omitted. A special `SparseIndex` object tracks where data has been “sparsified”. This will make much more sense in an example. All of the standard pandas data structures have a `to_sparse` method:

    In [1]: ts = pd.Series(randn(10))

    In [2]: ts[2:-2] = np.nan

    In [3]: sts = ts.to_sparse()

    In [4]: sts
    Out[4]:
    0    0.469112
    1   -0.282863
    2         NaN
    3         NaN
    4         NaN
    5         NaN
    6         NaN
    7         NaN
    8   -0.861849
    9   -2.104569
    dtype: float64
    BlockIndex
    Block locations: array([0, 8], dtype=int32)
    Block lengths: array([2, 2], dtype=int32)

The `to_sparse` method takes a `kind` argument (for the sparse index, see below) and a `fill_value`. So if we had a mostly zero Series, we could convert it to sparse with `fill_value=0`:

    In [5]: ts.fillna(0).to_sparse(fill_value=0)
    Out[5]:
    0    0.469112
    1   -0.282863
    2    0.000000
    3    0.000000
    4    0.000000
    5    0.000000
    6    0.000000
    7    0.000000
    8   -0.861849
    9   -2.104569
    dtype: float64
    BlockIndex
    Block locations: array([0, 8], dtype=int32)
    Block lengths: array([2, 2], dtype=int32)

The sparse objects exist for memory efficiency reasons. Suppose you had a large, mostly NA DataFrame:

    In [6]: df = pd.DataFrame(randn(10000, 4))

    In [7]: df.ix[:9998] = np.nan

    In [8]: sdf = df.to_sparse()

    In [9]: sdf
    Out[9]:
                 0         1         2         3
    0          NaN       NaN       NaN       NaN
    1          NaN       NaN       NaN       NaN
    2          NaN       NaN       NaN       NaN
    3          NaN       NaN       NaN       NaN
    4          NaN       NaN       NaN       NaN
    5          NaN       NaN       NaN       NaN
    6          NaN       NaN       NaN       NaN
    ...        ...       ...       ...       ...
    9993       NaN       NaN       NaN       NaN
    9994       NaN       NaN       NaN       NaN
    9995       NaN       NaN       NaN       NaN
    9996       NaN       NaN       NaN       NaN
    9997       NaN       NaN       NaN       NaN
    9998       NaN       NaN       NaN       NaN
    9999  0.280249 -1.648493  1.490865 -0.890819

    [10000 rows x 4 columns]

    In [10]: sdf.density
    Out[10]: 0.0001

As you can see, the density (% of values that have not been “compressed”) is extremely low. This sparse object takes up much less memory on disk (pickled) and in the Python interpreter. Functionally, their behavior should be nearly identical to their dense counterparts.

Any sparse object can be converted back to the standard dense form by calling `to_dense`:

    In [11]: sts.to_dense()
    Out[11]:
    0    0.469112
    1   -0.282863
    2         NaN
    3         NaN
    4         NaN
    5         NaN
    6         NaN
    7         NaN
    8   -0.861849
    9   -2.104569
    dtype: float64

## SparseArray

`SparseArray` is the base layer for all of the sparse indexed data structures. It is a 1-dimensional ndarray-like object storing only values distinct from the `fill_value`:

    In [12]: arr = np.random.randn(10)

    In [13]: arr[2:5] = np.nan; arr[7:8] = np.nan

    In [14]: sparr = pd.SparseArray(arr)

    In [15]: sparr
    Out[15]:
    [-1.95566352972, -1.6588664276, nan, nan, nan, 1.15893288864, 0.145297113733, nan, 0.606027190513, 1.33421134013]
    Fill: nan
    IntIndex
    Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)

Like the indexed objects (SparseSeries, SparseDataFrame, SparsePanel), a `SparseArray` can be converted back to a regular ndarray by calling `to_dense`:

    In [16]: sparr.to_dense()
    Out[16]:
    array([-1.9557, -1.6589,     nan,     nan,     nan,  1.1589,  0.1453,
               nan,  0.606 ,  1.3342])

## SparseList

`SparseList` is a list-like data structure for managing a dynamic collection of SparseArrays. To create one, simply call the `SparseList` constructor with a `fill_value` (defaulting to `NaN`):

    In [17]: spl = pd.SparseList()

    In [18]: spl
    Out[18]: <pandas.sparse.list.SparseList object at 0x7f093f75ca50>

The two important methods are `append` and `to_array`. `append` can accept scalar values or any 1-dimensional sequence:

    In [19]: spl.append(np.array([1., np.nan, np.nan, 2., 3.]))

    In [20]: spl.append(5)

    In [21]: spl.append(sparr)

    In [22]: spl
    Out[22]:
    <pandas.sparse.list.SparseList object at 0x7f093f75ca50>
    [1.0, nan, nan, 2.0, 3.0]
    Fill: nan
    IntIndex
    Indices: array([0, 3, 4], dtype=int32)

    [5.0]
    Fill: nan
    IntIndex
    Indices: array([0], dtype=int32)

    [-1.95566352972, -1.6588664276, nan, nan, nan, 1.15893288864, 0.145297113733, nan, 0.606027190513, 1.33421134013]
    Fill: nan
    IntIndex
    Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)

As you can see, all of the contents are stored internally as a list of memory-efficient `SparseArray` objects. Once you’ve accumulated all of the data, you can call `to_array` to get a single `SparseArray` with all the data:

    In [23]: spl.to_array()
    Out[23]:
    [1.0, nan, nan, 2.0, 3.0, 5.0, -1.95566352972, -1.6588664276, nan, nan, nan, 1.15893288864, 0.145297113733, nan, 0.606027190513, 1.33421134013]
    Fill: nan
    IntIndex
    Indices: array([ 0,  3,  4,  5,  6,  7, 11, 12, 14, 15], dtype=int32)

## SparseIndex objects

Two kinds of `SparseIndex` are implemented, `block` and `integer`. We recommend using `block` as it’s more memory efficient. The `integer` format keeps an arrays of all of the locations where the data are not equal to the fill value. The `block` format tracks only the locations and sizes of blocks of data.

## Interaction with scipy.sparse

Experimental api to transform between sparse pandas and scipy.sparse structures.

A `SparseSeries.to_coo()` method is implemented for transforming a `SparseSeries` indexed by a `MultiIndex` to a `scipy.sparse.coo_matrix`.

The method requires a `MultiIndex` with two or more levels.

    In [24]: s = pd.Series([3.0, np.nan, 1.0, 3.0, np.nan, np.nan])

    In [25]: s.index = pd.MultiIndex.from_tuples([(1, 2, 'a', 0),
       ....:                                      (1, 2, 'a', 1),
       ....:                                      (1, 1, 'b', 0),
       ....:                                      (1, 1, 'b', 1),
       ....:                                      (2, 1, 'b', 0),
       ....:                                      (2, 1, 'b', 1)],
       ....:                                      names=['A', 'B', 'C', 'D'])
       ....:

    In [26]: s
    Out[26]:
    A  B  C  D
    1  2  a  0     3
             1   NaN
       1  b  0     1
             1     3
    2  1  b  0   NaN
             1   NaN
    dtype: float64

    # SparseSeries
    In [27]: ss = s.to_sparse()

    In [28]: ss
    Out[28]:
    A  B  C  D
    1  2  a  0     3
             1   NaN
       1  b  0     1
             1     3
    2  1  b  0   NaN
             1   NaN
    dtype: float64
    BlockIndex
    Block locations: array([0, 2], dtype=int32)
    Block lengths: array([1, 2], dtype=int32)

In the example below, we transform the `SparseSeries` to a sparse representation of a 2-d array by specifying that the first and second `MultiIndex` levels define labels for the rows and the third and fourth levels define labels for the columns. We also specify that the column and row labels should be sorted in the final sparse representation.

    In [29]: A, rows, columns = ss.to_coo(row_levels=['A', 'B'],
       ....:                              column_levels=['C', 'D'],
       ....:                              sort_labels=True)
       ....:

    In [30]: A
    Out[30]:
    <3x4 sparse matrix of type '<type 'numpy.float64'>'
    	with 3 stored elements in COOrdinate format>

    In [31]: A.todense()
    Out[31]:
    matrix([[ 0.,  0.,  1.,  3.],
            [ 3.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.]])

    In [32]: rows
    Out[32]: [(1, 1), (1, 2), (2, 1)]

    In [33]: columns
    Out[33]: [('a', 0), ('a', 1), ('b', 0), ('b', 1)]

Specifying different row and column labels (and not sorting them) yields a different sparse matrix:

    In [34]: A, rows, columns = ss.to_coo(row_levels=['A', 'B', 'C'],
       ....:                              column_levels=['D'],
       ....:                              sort_labels=False)
       ....:

    In [35]: A
    Out[35]:
    <3x2 sparse matrix of type '<type 'numpy.float64'>'
    	with 3 stored elements in COOrdinate format>

    In [36]: A.todense()
    Out[36]:
    matrix([[ 3.,  0.],
            [ 1.,  3.],
            [ 0.,  0.]])

    In [37]: rows
    Out[37]: [(1, 2, 'a'), (1, 1, 'b'), (2, 1, 'b')]

    In [38]: columns
    Out[38]: [0, 1]

A convenience method `SparseSeries.from_coo()` is implemented for creating a `SparseSeries` from a `scipy.sparse.coo_matrix`.

    In [39]: from scipy import sparse

    In [40]: A = sparse.coo_matrix(([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])),
       ....:                       shape=(3, 4))
       ....:

    In [41]: A
    Out[41]:
    <3x4 sparse matrix of type '<type 'numpy.float64'>'
    	with 3 stored elements in COOrdinate format>

    In [42]: A.todense()
    Out[42]:
    matrix([[ 0.,  0.,  1.,  2.],
            [ 3.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.]])

The default behaviour (with `dense_index=False`) simply returns a `SparseSeries` containing only the non-null entries.

    In [43]: ss = pd.SparseSeries.from_coo(A)

    In [44]: ss
    Out[44]:
    0  2    1
       3    2
    1  0    3
    dtype: float64
    BlockIndex
    Block locations: array([0], dtype=int32)
    Block lengths: array([3], dtype=int32)

Specifying `dense_index=True` will result in an index that is the Cartesian product of the row and columns coordinates of the matrix. Note that this will consume a significant amount of memory (relative to `dense_index=False`) if the sparse matrix is large (and sparse) enough.

    In [45]: ss_dense = pd.SparseSeries.from_coo(A, dense_index=True)

    In [46]: ss_dense
    Out[46]:
    0  0   NaN
       1   NaN
       2     1
       3     2
    1  0     3
       1   NaN
       2   NaN
       3   NaN
    2  0   NaN
       1   NaN
       2   NaN
       3   NaN
    dtype: float64
    BlockIndex
    Block locations: array([2], dtype=int32)
    Block lengths: array([3], dtype=int32)