# v0.11.0 (April 22, 2013)

This is a major release from 0.10.1 and includes many new features and enhancements along with a large number of bug fixes. The methods of Selecting Data have had quite a number of additions, and Dtype support is now full-fledged. There are also a number of important API changes that long-time pandas users should pay close attention to.

There is a new section in the documentation, 10 Minutes to Pandas, primarily geared to new users.

There is a new section in the documentation, Cookbook, a collection of useful recipes in pandas (and that we want contributions!).

There are several libraries that are now Recommended Dependencies

## Selection choices

Starting in 0.11.0, object selection has had a number of user-requested additions in order to support more explicit location based indexing. Pandas now supports three types of multi-axis indexing.

- `.loc` is strictly label based, will raise `KeyError` when the items are not found, allowed inputs are:

  - A single label, e.g. `5` or `'a'`, (note that `5` is interpreted as a label of the index. This use is not an integer position along the index)
  - A list or array of labels `['a', 'b', 'c']`
  - A slice object with labels `'a':'f'`, (note that contrary to usual python slices, both the start and the stop are included!)
  - A boolean array

  See more at Selection by Label

- `.iloc` is strictly integer position based (from `0` to `length-1` of the axis), will raise `IndexError` when the requested indices are out of bounds. Allowed inputs are:

  - An integer e.g. `5`
  - A list or array of integers `[4, 3, 0]`
  - A slice object with ints `1:7`
  - A boolean array

  See more at Selection by Position

- `.ix` supports mixed integer and label based access. It is primarily label based, but will fallback to integer positional access. `.ix` is the most general and will support any of the inputs to `.loc` and `.iloc`, as well as support for floating point label schemes. `.ix` is especially useful when dealing with mixed positional and label based hierarchical indexes.

  As using integer slices with `.ix` have different behavior depending on whether the slice is interpreted as position based or label based, it’s usually better to be explicit and use `.iloc` or `.loc`.

  See more at Advanced Indexing and Advanced Hierarchical.

## Selection deprecations

Starting in version 0.11.0, these methods may be deprecated in future versions.

- `irow`
- `icol`
- `iget_value`

See the section Selection by Position for substitutes.

## Dtypes

Numeric dtypes will propagate and can coexist in DataFrames. If a dtype is passed (either directly via the `dtype` keyword, a passed `ndarray`, or a passed `Series`, then it will be preserved in DataFrame operations. Furthermore, different numeric dtypes will NOT be combined. The following example will give you a taste.

    In [1]: df1 = pd.DataFrame(np.random.randn(8, 1), columns=['A'], dtype='float32')

    In [2]: df1
    Out[2]:
              A
    0  0.469112
    1 -0.282863
    2 -1.509058
    3 -1.135632
    4  1.212112
    5 -0.173215
    6  0.119209
    7 -1.044236

    In [3]: df1.dtypes
    Out[3]:
    A    float32
    dtype: object

    In [4]: df2 = pd.DataFrame({'A': pd.Series(np.random.randn(8), dtype='float16'),
       ...:                     'B': pd.Series(np.random.randn(8)),
       ...:                     'C': pd.Series(range(8), dtype='uint8')})
       ...:

    In [5]: df2
    Out[5]:
              A         B  C
    0 -0.861816 -0.424972  0
    1 -2.105469  0.567020  1
    2 -0.494873  0.276232  2
    3  1.072266 -1.087401  3
    4  0.721680 -0.673690  4
    5 -0.706543  0.113648  5
    6 -1.040039 -1.478427  6
    7  0.271973  0.524988  7

    In [6]: df2.dtypes
    Out[6]:
    A    float16
    B    float64
    C      uint8
    dtype: object

    # here you get some upcasting
    In [7]: df3 = df1.reindex_like(df2).fillna(value=0.0) + df2

    In [8]: df3
    Out[8]:
              A         B    C
    0 -0.392704 -0.424972  0.0
    1 -2.388332  0.567020  1.0
    2 -2.003932  0.276232  2.0
    3 -0.063367 -1.087401  3.0
    4  1.933792 -0.673690  4.0
    5 -0.879758  0.113648  5.0
    6 -0.920830 -1.478427  6.0
    7 -0.772263  0.524988  7.0

    In [9]: df3.dtypes
    Out[9]:
    A    float32
    B    float64
    C    float64
    dtype: object

## Dtype conversion

This is lower-common-denominator upcasting, meaning you get the dtype which can accommodate all of the types

    In [10]: df3.values.dtype
    Out[10]: dtype('float64')

Conversion

    In [11]: df3.astype('float32').dtypes
    Out[11]:
    A    float32
    B    float32
    C    float32
    dtype: object

Mixed conversion

    In [12]: df3['D'] = '1.'

    In [13]: df3['E'] = '1'

    In [14]: df3.convert_objects(convert_numeric=True).dtypes
    Out[14]:
    A    float32
    B    float64
    C    float64
    D    float64
    E      int64
    dtype: object

    # same, but specific dtype conversion
    In [15]: df3['D'] = df3['D'].astype('float16')

    In [16]: df3['E'] = df3['E'].astype('int32')

    In [17]: df3.dtypes
    Out[17]:
    A    float32
    B    float64
    C    float64
    D    float16
    E      int32
    dtype: object

Forcing date coercion (and setting `NaT` when not datelike)

    In [18]: import datetime

    In [19]: s = pd.Series([datetime.datetime(2001, 1, 1, 0, 0), 'foo', 1.0, 1,
       ....:                pd.Timestamp('20010104'), '20010105'], dtype='O')
       ....:

    In [20]: s.convert_objects(convert_dates='coerce')
    Out[20]:
    0   2001-01-01
    1          NaT
    2          NaT
    3          NaT
    4   2001-01-04
    5   2001-01-05
    dtype: datetime64[ns]

## Dtype gotchas

Platform gotchas

Starting in 0.11.0, construction of DataFrame/Series will use default dtypes of `int64` and `float64`, regardless of platform. This is not an apparent change from earlier versions of pandas. If you specify dtypes, they WILL be respected, however (GH2837)

The following will all result in `int64` dtypes

    In [21]: pd.DataFrame([1, 2], columns=['a']).dtypes
    Out[21]:
    a    int64
    dtype: object

    In [22]: pd.DataFrame({'a': [1, 2]}).dtypes
    Out[22]:
    a    int64
    dtype: object

    In [23]: pd.DataFrame({'a': 1}, index=range(2)).dtypes
    Out[23]:
    a    int64
    dtype: object

Keep in mind that `DataFrame(np.array([1,2]))` WILL result in `int32` on 32-bit platforms!

Upcasting gotchas

Performing indexing operations on integer type data can easily upcast the data. The dtype of the input data will be preserved in cases where `nans` are not introduced.

    In [24]: dfi = df3.astype('int32')

    In [25]: dfi['D'] = dfi['D'].astype('int64')

    In [26]: dfi
    Out[26]:
      A  B  C  D  E
    0  0  0  0  1  1
    1 -2  0  1  1  1
    2 -2  0  2  1  1
    3  0 -1  3  1  1
    4  1  0  4  1  1
    5  0  0  5  1  1
    6  0 -1  6  1  1
    7  0  0  7  1  1

    In [27]: dfi.dtypes
    Out[27]:
    A    int32
    B    int32
    C    int32
    D    int64
    E    int32
    dtype: object

    In [28]: casted = dfi[dfi > 0]

    In [29]: casted
    Out[29]:
        A   B    C  D  E
    0  NaN NaN  NaN  1  1
    1  NaN NaN  1.0  1  1
    2  NaN NaN  2.0  1  1
    3  NaN NaN  3.0  1  1
    4  1.0 NaN  4.0  1  1
    5  NaN NaN  5.0  1  1
    6  NaN NaN  6.0  1  1
    7  NaN NaN  7.0  1  1

    In [30]: casted.dtypes
    Out[30]:
    A    float64
    B    float64
    C    float64
    D      int64
    E      int32
    dtype: object

While float dtypes are unchanged.

    In [31]: df4 = df3.copy()

    In [32]: df4['A'] = df4['A'].astype('float32')

    In [33]: df4.dtypes
    Out[33]:
    A    float32
    B    float64
    C    float64
    D    float16
    E      int32
    dtype: object

    In [34]: casted = df4[df4 > 0]

    In [35]: casted
    Out[35]:
              A         B    C    D  E
    0       NaN       NaN  NaN  1.0  1
    1       NaN  0.567020  1.0  1.0  1
    2       NaN  0.276232  2.0  1.0  1
    3       NaN       NaN  3.0  1.0  1
    4  1.933792       NaN  4.0  1.0  1
    5       NaN  0.113648  5.0  1.0  1
    6       NaN       NaN  6.0  1.0  1
    7       NaN  0.524988  7.0  1.0  1

    In [36]: casted.dtypes
    Out[36]:
    A    float32
    B    float64
    C    float64
    D    float16
    E      int32
    dtype: object

## Datetimes conversion

Datetime64[ns] columns in a DataFrame (or a Series) allow the use of `np.nan` to indicate a nan value, in addition to the traditional `NaT`, or not-a-time. This allows convenient nan setting in a generic way. Furthermore `datetime64[ns]` columns are created by default, when passed datetimelike objects (this change was introduced in 0.10.1) (GH2809, GH2810)

    In [12]: df = pd.DataFrame(np.random.randn(6, 2), pd.date_range('20010102', periods=6),
       ....:                   columns=['A', ' B'])
       ....:

    In [13]: df['timestamp'] = pd.Timestamp('20010103')

    In [14]: df
    Out[14]:
                       A         B  timestamp
    2001-01-02  0.404705  0.577046 2001-01-03
    2001-01-03 -1.715002 -1.039268 2001-01-03
    2001-01-04 -0.370647 -1.157892 2001-01-03
    2001-01-05 -1.344312  0.844885 2001-01-03
    2001-01-06  1.075770 -0.109050 2001-01-03
    2001-01-07  1.643563 -1.469388 2001-01-03

    # datetime64[ns] out of the box
    In [15]: df.dtypes.value_counts()
    Out[15]:
    float64           2
    datetime64[ns]    1
    dtype: int64

    # use the traditional nan, which is mapped to NaT internally
    In [16]: df.loc[df.index[2:4], ['A', 'timestamp']] = np.nan

    In [17]: df
    Out[17]:
                       A         B  timestamp
    2001-01-02  0.404705  0.577046 2001-01-03
    2001-01-03 -1.715002 -1.039268 2001-01-03
    2001-01-04       NaN -1.157892        NaT
    2001-01-05       NaN  0.844885        NaT
    2001-01-06  1.075770 -0.109050 2001-01-03
    2001-01-07  1.643563 -1.469388 2001-01-03

Astype conversion on `datetime64[ns]` to `object`, implicitly converts `NaT` to `np.nan`

    In [18]: s = pd.Series([datetime.datetime(2001, 1, 2, 0, 0) for i in range(3)])

    In [19]: s.dtype
    Out[19]: dtype('<M8[ns]')

    In [20]: s[1] = np.nan

    In [21]: s
    Out[21]:
    0   2001-01-02
    1          NaT
    2   2001-01-02
    dtype: datetime64[ns]

    In [22]: s.dtype
    Out[22]: dtype('<M8[ns]')

    In [23]: s = s.astype('O')

    In [24]: s
    Out[24]:
    0    2001-01-02 00:00:00
    1                    NaT
    2    2001-01-02 00:00:00
    dtype: object

    In [25]: s.dtype
    Out[25]: dtype('O')

## API changes

> - Added to_series() method to indices, to facilitate the creation of indexers (GH3275)
> - `HDFStore`
>   - added the method `select_column` to select a single column from a table as a Series.
>   - deprecated the `unique` method, can be replicated by `select_column(key,column).unique()`
>   - `min_itemsize` parameter to `append` will now automatically create data_columns for passed keys

## Enhancements

> - Improved performance of df.to_csv() by up to 10x in some cases. (GH3059)

> - Numexpr is now a Recommended Dependencies, to accelerate certain types of numerical and boolean operations

> - Bottleneck is now a Recommended Dependencies, to accelerate certain types of `nan` operations

> - `HDFStore`

>   - support `read_hdf/to_hdf` API similar to `read_csv/to_csv`

>         In [26]: df = pd.DataFrame({'A': range(5), 'B': range(5)})

>         In [27]: df.to_hdf('store.h5', 'table', append=True)

>         In [28]: pd.read_hdf('store.h5', 'table', where=['index > 2'])
>         Out[28]:
>            A  B
>         3  3  3
>         4  4  4

>   - provide dotted attribute access to `get` from stores, e.g. `store.df == store['df']`

>   - new keywords `iterator=boolean`, and `chunksize=number_in_a_chunk` are provided to support iteration on `select` and `select_as_multiple` (GH3076)

> - You can now select timestamps from an unordered timeseries similarly to an ordered timeseries (GH2437)

> - You can now select with a string from a DataFrame with a datelike index, in a similar way to a Series (GH3070)

>       In [29]: idx = pd.date_range("2001-10-1", periods=5, freq='M')

>       In [30]: ts = pd.Series(np.random.rand(len(idx)), index=idx)

>       In [31]: ts['2001']
>       Out[31]:
>       2001-10-31    0.117967
>       2001-11-30    0.702184
>       2001-12-31    0.414034
>       Freq: M, dtype: float64

>       In [32]: df = pd.DataFrame({'A': ts})

>       In [33]: df['2001']
>       Out[33]:
>                          A
>       2001-10-31  0.117967
>       2001-11-30  0.702184
>       2001-12-31  0.414034

> - `Squeeze` to possibly remove length 1 dimensions from an object.

>       >>> p = pd.Panel(np.random.randn(3, 4, 4), items=['ItemA', 'ItemB', 'ItemC'],
>       ...              major_axis=pd.date_range('20010102', periods=4),
>       ...              minor_axis=['A', 'B', 'C', 'D'])
>       >>> p
>       <class 'pandas.core.panel.Panel'>
>       Dimensions: 3 (items) x 4 (major_axis) x 4 (minor_axis)
>       Items axis: ItemA to ItemC
>       Major_axis axis: 2001-01-02 00:00:00 to 2001-01-05 00:00:00
>       Minor_axis axis: A to D

>       >>> p.reindex(items=['ItemA']).squeeze()
>                          A         B         C         D
>       2001-01-02  0.926089 -2.026458  0.501277 -0.204683
>       2001-01-03 -0.076524  1.081161  1.141361  0.479243
>       2001-01-04  0.641817 -0.185352  1.824568  0.809152
>       2001-01-05  0.575237  0.669934  1.398014 -0.399338

>       >>> p.reindex(items=['ItemA'], minor=['B']).squeeze()
>       2001-01-02   -2.026458
>       2001-01-03    1.081161
>       2001-01-04   -0.185352
>       2001-01-05    0.669934
>       Freq: D, Name: B, dtype: float64

> - In `pd.io.data.Options`,

>   - Fix bug when trying to fetch data for the current month when already past expiry.
>   - Now using lxml to scrape html instead of BeautifulSoup (lxml was faster).
>   - New instance variables for calls and puts are automatically created when a method that creates them is called. This works for current month where the instance variables are simply `calls` and `puts`. Also works for future expiry months and save the instance variable as `callsMMYY` or `putsMMYY`, where `MMYY` are, respectively, the month and year of the option’s expiry.
>   - `Options.get_near_stock_price` now allows the user to specify the month for which to get relevant options data.
>   - `Options.get_forward_data` now has optional kwargs `near` and `above_below`. This allows the user to specify if they would like to only return forward looking data for options near the current stock price. This just obtains the data from Options.get_near_stock_price instead of Options.get_xxx_data() (GH2758).

> - Cursor coordinate information is now displayed in time-series plots.

> - added option display.max_seq_items to control the number of elements printed per sequence pprinting it. (GH2979)

> - added option display.chop_threshold to control display of small numerical values. (GH2739)

> - added option display.max_info_rows to prevent verbose_info from being calculated for frames above 1M rows (configurable). (GH2807, GH2918)

> - value_counts() now accepts a “normalize” argument, for normalized histograms. (GH2710).

> - DataFrame.from_records now accepts not only dicts but any instance of the collections.Mapping ABC.

> - added option display.mpl_style providing a sleeker visual style for plots. Based on https://gist.github.com/huyng/816622 (GH3075).

> - Treat boolean values as integers (values 1 and 0) for numeric operations. (GH2641)

> - to_html() now accepts an optional “escape” argument to control reserved HTML character escaping (enabled by default) and escapes `&`, in addition to `<` and `>`. (GH2919)

See the full release notes or issue tracker on GitHub for a complete list.

## Contributors

For contributors, please see /usr/share/doc/contributors_list.txt or https://github.com/pandas-dev/pandas/graphs/contributors
Scroll To Top