# v0.15.0 (October 18, 2014)

This is a major release from 0.14.1 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.

Warning

pandas >= 0.15.0 will no longer support compatibility with NumPy versions < 1.7.0. If you want to use the latest versions of pandas, please upgrade to NumPy >= 1.7.0 (GH7711)

- Highlights include:
  - The `Categorical` type was integrated as a first-class pandas type, see here
  - New scalar type `Timedelta`, and a new index type `TimedeltaIndex`, see here
  - New datetimelike properties accessor `.dt` for Series, see Datetimelike Properties
  - New DataFrame default display for `df.info()` to include memory usage, see Memory Usage
  - `read_csv` will now by default ignore blank lines when parsing, see here
  - API change in using Indexes in set operations, see here
  - Enhancements in the handling of timezones, see here
  - A lot of improvements to the rolling and expanding moment functions, see here
  - Internal refactoring of the `Index` class to no longer sub-class `ndarray`, see Internal Refactoring
  - dropping support for `PyTables` less than version 3.0.0, and `numexpr` less than version 2.1 (GH7990)
  - Split indexing documentation into Indexing and Selecting Data and MultiIndex / Advanced Indexing
  - Split out string methods documentation into Working with Text Data
- Check the API Changes and deprecations before updating
- Other Enhancements
- Performance Improvements
- Bug Fixes

Warning

In 0.15.0 `Index` has internally been refactored to no longer sub-class `ndarray` but instead subclass `PandasObject`, similarly to the rest of the pandas objects. This change allows very easy sub-classing and creation of new index types. This should be a transparent change with only very limited API implications (See the Internal Refactoring)

Warning

The refactoring in `Categorical` changed the two argument constructor from “codes/labels and levels” to “values and levels (now called ‘categories’)”. This can lead to subtle bugs. If you use `Categorical` directly, please audit your code before updating to this pandas version and change it to use the `from_codes()` constructor. See more on `Categorical` here

## New features

### Categoricals in Series/DataFrame

`Categorical` can now be included in Series and DataFrames and gained new methods to manipulate. Thanks to Jan Schulz for much of this API/implementation. (GH3943, GH5313, GH5314, GH7444, GH7839, GH7848, GH7864, GH7914, GH7768, GH8006, GH3678, GH8075, GH8076, GH8143, GH8453, GH8518).

For full docs, see the categorical introduction and the API documentation.

    In [1]: df = pd.DataFrame({"id": [1, 2, 3, 4, 5, 6],
       ...:                    "raw_grade": ['a', 'b', 'b', 'a', 'a', 'e']})
       ...:

    In [2]: df["grade"] = df["raw_grade"].astype("category")

    In [3]: df["grade"]
    Out[3]:
    0    a
    1    b
    2    b
    3    a
    4    a
    5    e
    Name: grade, Length: 6, dtype: category
    Categories (3, object): [a, b, e]

    # Rename the categories
    In [4]: df["grade"].cat.categories = ["very good", "good", "very bad"]

    # Reorder the categories and simultaneously add the missing categories
    In [5]: df["grade"] = df["grade"].cat.set_categories(["very bad", "bad",
       ...:                                               "medium", "good", "very good"])
       ...:

    In [6]: df["grade"]
    Out[6]:
    0    very good
    1         good
    2         good
    3    very good
    4    very good
    5     very bad
    Name: grade, Length: 6, dtype: category
    Categories (5, object): [very bad, bad, medium, good, very good]

    In [7]: df.sort_values("grade")
    Out[7]:
       id raw_grade      grade
    5   6         e   very bad
    1   2         b       good
    2   3         b       good
    0   1         a  very good
    3   4         a  very good
    4   5         a  very good

    [6 rows x 3 columns]

    In [8]: df.groupby("grade").size()
    Out[8]:
    grade
    very bad     1
    bad          0
    medium       0
    good         2
    very good    3
    Length: 5, dtype: int64

- `pandas.core.group_agg` and `pandas.core.factor_agg` were removed. As an alternative, construct a dataframe and use `df.groupby(<group>).agg(<func>)`.
- Supplying “codes/labels and levels” to the `Categorical` constructor is not supported anymore. Supplying two arguments to the constructor is now interpreted as “values and levels (now called ‘categories’)”. Please change your code to use the `from_codes()` constructor.
- The `Categorical.labels` attribute was renamed to `Categorical.codes` and is read only. If you want to manipulate codes, please use one of the API methods on Categoricals.
- The `Categorical.levels` attribute is renamed to `Categorical.categories`.

### TimedeltaIndex/Scalar

We introduce a new scalar type `Timedelta`, which is a subclass of `datetime.timedelta`, and behaves in a similar manner, but allows compatibility with `np.timedelta64` types as well as a host of custom representation, parsing, and attributes. This type is very similar to how `Timestamp` works for `datetimes`. It is a nice-API box for the type. See the docs. (GH3009, GH4533, GH8209, GH8187, GH8190, GH7869, GH7661, GH8345, GH8471)

Warning

`Timedelta` scalars (and `TimedeltaIndex`) component fields are not the same as the component fields on a `datetime.timedelta` object. For example, `.seconds` on a `datetime.timedelta` object returns the total number of seconds combined between `hours`, `minutes` and `seconds`. In contrast, the pandas `Timedelta` breaks out hours, minutes, microseconds and nanoseconds separately.

    # Timedelta accessor
    In [9]: tds = pd.Timedelta('31 days 5 min 3 sec')

    In [10]: tds.minutes
    Out[10]: 5L

    In [11]: tds.seconds
    Out[11]: 3L

    # datetime.timedelta accessor
    # this is 5 minutes * 60 + 3 seconds
    In [12]: tds.to_pytimedelta().seconds
    Out[12]: 303

Note: this is no longer true starting from v0.16.0, where full compatibility with `datetime.timedelta` is introduced. See the 0.16.0 whatsnew entry

Warning

Prior to 0.15.0 `pd.to_timedelta` would return a `Series` for list-like/Series input, and a `np.timedelta64` for scalar input. It will now return a `TimedeltaIndex` for list-like input, `Series` for Series input, and `Timedelta` for scalar input.

The arguments to `pd.to_timedelta` are now `(arg,unit='ns',box=True,coerce=False)`, previously were `(arg,box=True,unit='ns')` as these are more logical.

Construct a scalar

    In [9]: pd.Timedelta('1 days 06:05:01.00003')
    Out[9]: Timedelta('1 days 06:05:01.000030')

    In [10]: pd.Timedelta('15.5us')
    Out[10]: Timedelta('0 days 00:00:00.000015')

    In [11]: pd.Timedelta('1 hour 15.5us')
    Out[11]: Timedelta('0 days 01:00:00.000015')

    # negative Timedeltas have this string repr
    # to be more consistent with datetime.timedelta conventions
    In [12]: pd.Timedelta('-1us')
    Out[12]: Timedelta('-1 days +23:59:59.999999')

    # a NaT
    In [13]: pd.Timedelta('nan')
    Out[13]: NaT

Access fields for a `Timedelta`

    In [14]: td = pd.Timedelta('1 hour 3m 15.5us')

    In [15]: td.seconds
    Out[15]: 3780

    In [16]: td.microseconds
    Out[16]: 16

    In [17]: td.nanoseconds
    Out[17]: 500

Construct a `TimedeltaIndex`

    In [18]: pd.TimedeltaIndex(['1 days', '1 days, 00:00:05',
       ....:                    np.timedelta64(2, 'D'),
       ....:                    datetime.timedelta(days=2, seconds=2)])
       ....:
    Out[18]:
    TimedeltaIndex(['1 days 00:00:00', '1 days 00:00:05', '2 days 00:00:00',
                    '2 days 00:00:02'],
                   dtype='timedelta64[ns]', freq=None)

Constructing a `TimedeltaIndex` with a regular range

    In [19]: pd.timedelta_range('1 days', periods=5, freq='D')
    Out[19]: TimedeltaIndex(['1 days', '2 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]', freq='D')

    In [20]: pd.timedelta_range(start='1 days', end='2 days', freq='30T')
    Out[20]:
    TimedeltaIndex(['1 days 00:00:00', '1 days 00:30:00', '1 days 01:00:00',
                    '1 days 01:30:00', '1 days 02:00:00', '1 days 02:30:00',
                    '1 days 03:00:00', '1 days 03:30:00', '1 days 04:00:00',
                    '1 days 04:30:00', '1 days 05:00:00', '1 days 05:30:00',
                    '1 days 06:00:00', '1 days 06:30:00', '1 days 07:00:00',
                    '1 days 07:30:00', '1 days 08:00:00', '1 days 08:30:00',
                    '1 days 09:00:00', '1 days 09:30:00', '1 days 10:00:00',
                    '1 days 10:30:00', '1 days 11:00:00', '1 days 11:30:00',
                    '1 days 12:00:00', '1 days 12:30:00', '1 days 13:00:00',
                    '1 days 13:30:00', '1 days 14:00:00', '1 days 14:30:00',
                    '1 days 15:00:00', '1 days 15:30:00', '1 days 16:00:00',
                    '1 days 16:30:00', '1 days 17:00:00', '1 days 17:30:00',
                    '1 days 18:00:00', '1 days 18:30:00', '1 days 19:00:00',
                    '1 days 19:30:00', '1 days 20:00:00', '1 days 20:30:00',
                    '1 days 21:00:00', '1 days 21:30:00', '1 days 22:00:00',
                    '1 days 22:30:00', '1 days 23:00:00', '1 days 23:30:00',
                    '2 days 00:00:00'],
                   dtype='timedelta64[ns]', freq='30T')

You can now use a `TimedeltaIndex` as the index of a pandas object

    In [21]: s = pd.Series(np.arange(5),
       ....:               index=pd.timedelta_range('1 days', periods=5, freq='s'))
       ....:

    In [22]: s
    Out[22]:
    1 days 00:00:00    0
    1 days 00:00:01    1
    1 days 00:00:02    2
    1 days 00:00:03    3
    1 days 00:00:04    4
    Freq: S, Length: 5, dtype: int64

You can select with partial string selections

    In [23]: s['1 day 00:00:02']
    Out[23]: 2

    In [24]: s['1 day':'1 day 00:00:02']
    Out[24]:
    1 days 00:00:00    0
    1 days 00:00:01    1
    1 days 00:00:02    2
    Freq: S, Length: 3, dtype: int64

Finally, the combination of `TimedeltaIndex` with `DatetimeIndex` allow certain combination operations that are `NaT` preserving:

    In [25]: tdi = pd.TimedeltaIndex(['1 days', pd.NaT, '2 days'])

    In [26]: tdi.tolist()
    Out[26]: [Timedelta('1 days 00:00:00'), NaT, Timedelta('2 days 00:00:00')]

    In [27]: dti = pd.date_range('20130101', periods=3)

    In [28]: dti.tolist()
    Out[28]:
    [Timestamp('2013-01-01 00:00:00', freq='D'),
     Timestamp('2013-01-02 00:00:00', freq='D'),
     Timestamp('2013-01-03 00:00:00', freq='D')]

    In [29]: (dti + tdi).tolist()
    Out[29]: [Timestamp('2013-01-02 00:00:00'), NaT, Timestamp('2013-01-05 00:00:00')]

    In [30]: (dti - tdi).tolist()
    Out[30]: [Timestamp('2012-12-31 00:00:00'), NaT, Timestamp('2013-01-01 00:00:00')]

- iteration of a `Series` e.g. `list(Series(...))` of `timedelta64[ns]` would prior to v0.15.0 return `np.timedelta64` for each element. These will now be wrapped in `Timedelta`.

### Memory usage

Implemented methods to find memory usage of a DataFrame. See the FAQ for more. (GH6852).

A new display option `display.memory_usage` (see Options and settings) sets the default behavior of the `memory_usage` argument in the `df.info()` method. By default `display.memory_usage` is `True`.

    In [31]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
       ....:           'complex128', 'object', 'bool']
       ....:

    In [32]: n = 5000

    In [33]: data = {t: np.random.randint(100, size=n).astype(t) for t in dtypes}

    In [34]: df = pd.DataFrame(data)

    In [35]: df['categorical'] = df['object'].astype('category')

    In [36]: df.info()
    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 5000 entries, 0 to 4999
    Data columns (total 8 columns):
    int64              5000 non-null int64
    float64            5000 non-null float64
    datetime64[ns]     5000 non-null datetime64[ns]
    timedelta64[ns]    5000 non-null timedelta64[ns]
    complex128         5000 non-null complex128
    object             5000 non-null object
    bool               5000 non-null bool
    categorical        5000 non-null category
    dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
    memory usage: 289.1+ KB

Additionally `memory_usage()` is an available method for a dataframe object which returns the memory usage of each column.

    In [37]: df.memory_usage(index=True)
    Out[37]:
    Index                128
    int64              40000
    float64            40000
    datetime64[ns]     40000
    timedelta64[ns]    40000
    complex128         80000
    object             40000
    bool                5000
    categorical        10920
    Length: 9, dtype: int64

### .dt accessor

`Series` has gained an accessor to succinctly return datetime like properties for the values of the Series, if its a datetime/period like Series. (GH7207) This will return a Series, indexed like the existing Series. See the docs

    # datetime
    In [38]: s = pd.Series(pd.date_range('20130101 09:10:12', periods=4))

    In [39]: s
    Out[39]:
    0   2013-01-01 09:10:12
    1   2013-01-02 09:10:12
    2   2013-01-03 09:10:12
    3   2013-01-04 09:10:12
    Length: 4, dtype: datetime64[ns]

    In [40]: s.dt.hour
    Out[40]:
    0    9
    1    9
    2    9
    3    9
    Length: 4, dtype: int64

    In [41]: s.dt.second
    Out[41]:
    0    12
    1    12
    2    12
    3    12
    Length: 4, dtype: int64

    In [42]: s.dt.day
    Out[42]:
    0    1
    1    2
    2    3
    3    4
    Length: 4, dtype: int64

    In [43]: s.dt.freq
    Out[43]: 'D'

This enables nice expressions like this:

    In [44]: s[s.dt.day == 2]
    Out[44]:
    1   2013-01-02 09:10:12
    Length: 1, dtype: datetime64[ns]

You can easily produce tz aware transformations:

    In [45]: stz = s.dt.tz_localize('US/Eastern')

    In [46]: stz
    Out[46]:
    0   2013-01-01 09:10:12-05:00
    1   2013-01-02 09:10:12-05:00
    2   2013-01-03 09:10:12-05:00
    3   2013-01-04 09:10:12-05:00
    Length: 4, dtype: datetime64[ns, US/Eastern]

    In [47]: stz.dt.tz
    Out[47]: <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>

You can also chain these types of operations:

    In [48]: s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')
    Out[48]:
    0   2013-01-01 04:10:12-05:00
    1   2013-01-02 04:10:12-05:00
    2   2013-01-03 04:10:12-05:00
    3   2013-01-04 04:10:12-05:00
    Length: 4, dtype: datetime64[ns, US/Eastern]

The `.dt` accessor works for period and timedelta dtypes.

    # period
    In [49]: s = pd.Series(pd.period_range('20130101', periods=4, freq='D'))

    In [50]: s
    Out[50]:
    0    2013-01-01
    1    2013-01-02
    2    2013-01-03
    3    2013-01-04
    Length: 4, dtype: period[D]

    In [51]: s.dt.year
    Out[51]:
    0    2013
    1    2013
    2    2013
    3    2013
    Length: 4, dtype: int64

    In [52]: s.dt.day
    Out[52]:
    0    1
    1    2
    2    3
    3    4
    Length: 4, dtype: int64

    # timedelta
    In [53]: s = pd.Series(pd.timedelta_range('1 day 00:00:05', periods=4, freq='s'))

    In [54]: s
    Out[54]:
    0   1 days 00:00:05
    1   1 days 00:00:06
    2   1 days 00:00:07
    3   1 days 00:00:08
    Length: 4, dtype: timedelta64[ns]

    In [55]: s.dt.days
    Out[55]:
    0    1
    1    1
    2    1
    3    1
    Length: 4, dtype: int64

    In [56]: s.dt.seconds
    Out[56]:
    0    5
    1    6
    2    7
    3    8
    Length: 4, dtype: int64

    In [57]: s.dt.components
    Out[57]:
       days  hours  minutes  seconds  milliseconds  microseconds  nanoseconds
    0     1      0        0        5             0             0            0
    1     1      0        0        6             0             0            0
    2     1      0        0        7             0             0            0
    3     1      0        0        8             0             0            0

    [4 rows x 7 columns]

### Timezone handling improvements

- `tz_localize(None)` for tz-aware `Timestamp` and `DatetimeIndex` now removes timezone holding local time, previously this resulted in `Exception` or `TypeError` (GH7812)

      In [58]: ts = pd.Timestamp('2014-08-01 09:00', tz='US/Eastern')

      In [59]: ts
      Out[59]: Timestamp('2014-08-01 09:00:00-0400', tz='US/Eastern')

      In [60]: ts.tz_localize(None)
      Out[60]: Timestamp('2014-08-01 09:00:00')

      In [61]: didx = pd.DatetimeIndex(start='2014-08-01 09:00', freq='H',
         ....:                         periods=10, tz='US/Eastern')
         ....:

      In [62]: didx
      Out[62]:
      DatetimeIndex(['2014-08-01 09:00:00-04:00', '2014-08-01 10:00:00-04:00',
                     '2014-08-01 11:00:00-04:00', '2014-08-01 12:00:00-04:00',
                     '2014-08-01 13:00:00-04:00', '2014-08-01 14:00:00-04:00',
                     '2014-08-01 15:00:00-04:00', '2014-08-01 16:00:00-04:00',
                     '2014-08-01 17:00:00-04:00', '2014-08-01 18:00:00-04:00'],
                    dtype='datetime64[ns, US/Eastern]', freq='H')

      In [63]: didx.tz_localize(None)
      Out[63]:
      DatetimeIndex(['2014-08-01 09:00:00', '2014-08-01 10:00:00',
                     '2014-08-01 11:00:00', '2014-08-01 12:00:00',
                     '2014-08-01 13:00:00', '2014-08-01 14:00:00',
                     '2014-08-01 15:00:00', '2014-08-01 16:00:00',
                     '2014-08-01 17:00:00', '2014-08-01 18:00:00'],
                    dtype='datetime64[ns]', freq='H')

- `tz_localize` now accepts the `ambiguous` keyword which allows for passing an array of bools indicating whether the date belongs in DST or not, ‘NaT’ for setting transition times to NaT, ‘infer’ for inferring DST/non-DST, and ‘raise’ (default) for an `AmbiguousTimeError` to be raised. See the docs for more details (GH7943)

- `DataFrame.tz_localize` and `DataFrame.tz_convert` now accepts an optional `level` argument for localizing a specific level of a MultiIndex (GH7846)

- `Timestamp.tz_localize` and `Timestamp.tz_convert` now raise `TypeError` in error cases, rather than `Exception` (GH8025)

- a timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone (rather than being a naive `datetime64[ns]`) as `object` dtype (GH8411)

- `Timestamp.__repr__` displays `dateutil.tz.tzoffset` info (GH7907)

### Rolling/expanding moments improvements

- `rolling_min()`, `rolling_max()`, `rolling_cov()`, and `rolling_corr()` now return objects with all `NaN` when `len(arg) < min_periods <= window` rather than raising. (This makes all rolling functions consistent in this behavior). (GH7766)

  Prior to 0.15.0

      In [64]: s = pd.Series([10, 11, 12, 13])

      In [15]: pd.rolling_min(s, window=10, min_periods=5)
      ValueError: min_periods (5) must be <= window (4)

  New behavior

      In [4]: pd.rolling_min(s, window=10, min_periods=5)
      Out[4]:
      0   NaN
      1   NaN
      2   NaN
      3   NaN
      dtype: float64

- `rolling_max()`, `rolling_min()`, `rolling_sum()`, `rolling_mean()`, `rolling_median()`, `rolling_std()`, `rolling_var()`, `rolling_skew()`, `rolling_kurt()`, `rolling_quantile()`, `rolling_cov()`, `rolling_corr()`, `rolling_corr_pairwise()`, `rolling_window()`, and `rolling_apply()` with `center=True` previously would return a result of the same structure as the input `arg` with `NaN` in the final `(window-1)/2` entries.

  Now the final `(window-1)/2` entries of the result are calculated as if the input `arg` were followed by `(window-1)/2` `NaN` values (or with shrinking windows, in the case of `rolling_apply()`). (GH7925, GH8269)

  Prior behavior (note final value is `NaN`):

      In [7]: pd.rolling_sum(Series(range(4)), window=3, min_periods=0, center=True)
      Out[7]:
      0     1
      1     3
      2     6
      3   NaN
      dtype: float64

  New behavior (note final value is `5 = sum([2, 3, NaN])`):

      In [7]: pd.rolling_sum(pd.Series(range(4)), window=3,
        ....:                min_periods=0, center=True)
      Out[7]:
      0    1
      1    3
      2    6
      3    5
      dtype: float64

- `rolling_window()` now normalizes the weights properly in rolling mean mode (mean=True) so that the calculated weighted means (e.g. ‘triang’, ‘gaussian’) are distributed about the same means as those calculated without weighting (i.e. ‘boxcar’). See the note on normalization for further details. (GH7618)

      In [65]: s = pd.Series([10.5, 8.8, 11.4, 9.7, 9.3])

  Behavior prior to 0.15.0:

      In [39]: pd.rolling_window(s, window=3, win_type='triang', center=True)
      Out[39]:
      0         NaN
      1    6.583333
      2    6.883333
      3    6.683333
      4         NaN
      dtype: float64

  New behavior

      In [10]: pd.rolling_window(s, window=3, win_type='triang', center=True)
      Out[10]:
      0       NaN
      1     9.875
      2    10.325
      3    10.025
      4       NaN
      dtype: float64

- Removed `center` argument from all `expanding_` functions (see list), as the results produced when `center=True` did not make much sense. (GH7925)

- Added optional `ddof` argument to `expanding_cov()` and `rolling_cov()`. The default value of `1` is backwards-compatible. (GH8279)

- Documented the `ddof` argument to `expanding_var()`, `expanding_std()`, `rolling_var()`, and `rolling_std()`. These functions’ support of a `ddof` argument (with a default value of `1`) was previously undocumented. (GH8064)

- `ewma()`, `ewmstd()`, `ewmvol()`, `ewmvar()`, `ewmcov()`, and `ewmcorr()` now interpret `min_periods` in the same manner that the `rolling_*()` and `expanding_*()` functions do: a given result entry will be `NaN` if the (expanding, in this case) window does not contain at least `min_periods` values. The previous behavior was to set to `NaN` the `min_periods` entries starting with the first non- `NaN` value. (GH7977)

  Prior behavior (note values start at index `2`, which is `min_periods` after index `0` (the index of the first non-empty value)):

      In [66]: s  = pd.Series([1, None, None, None, 2, 3])

      In [51]: ewma(s, com=3., min_periods=2)
      Out[51]:
      0         NaN
      1         NaN
      2    1.000000
      3    1.000000
      4    1.571429
      5    2.189189
      dtype: float64

  New behavior (note values start at index `4`, the location of the 2nd (since `min_periods=2`) non-empty value):

      In [2]: pd.ewma(s, com=3., min_periods=2)
      Out[2]:
      0         NaN
      1         NaN
      2         NaN
      3         NaN
      4    1.759644
      5    2.383784
      dtype: float64

- `ewmstd()`, `ewmvol()`, `ewmvar()`, `ewmcov()`, and `ewmcorr()` now have an optional `adjust` argument, just like `ewma()` does, affecting how the weights are calculated. The default value of `adjust` is `True`, which is backwards-compatible. See Exponentially weighted moment functions for details. (GH7911)

- `ewma()`, `ewmstd()`, `ewmvol()`, `ewmvar()`, `ewmcov()`, and `ewmcorr()` now have an optional `ignore_na` argument. When `ignore_na=False` (the default), missing values are taken into account in the weights calculation. When `ignore_na=True` (which reproduces the pre-0.15.0 behavior), missing values are ignored in the weights calculation. (GH7543)

      In [7]: pd.ewma(pd.Series([None, 1., 8.]), com=2.)
      Out[7]:
      0    NaN
      1    1.0
      2    5.2
      dtype: float64

      In [8]: pd.ewma(pd.Series([1., None, 8.]), com=2.,
        ....:         ignore_na=True)  # pre-0.15.0 behavior
      Out[8]:
      0    1.0
      1    1.0
      2    5.2
      dtype: float64

      In [9]: pd.ewma(pd.Series([1., None, 8.]), com=2.,
        ....:         ignore_na=False)  # new default
      Out[9]:
      0    1.000000
      1    1.000000
      2    5.846154
      dtype: float64

  Warning

  By default (`ignore_na=False`) the `ewm*()` functions’ weights calculation in the presence of missing values is different than in pre-0.15.0 versions. To reproduce the pre-0.15.0 calculation of weights in the presence of missing values one must specify explicitly `ignore_na=True`.

- Bug in `expanding_cov()`, `expanding_corr()`, `rolling_cov()`, `rolling_cor()`, `ewmcov()`, and `ewmcorr()` returning results with columns sorted by name and producing an error for non-unique columns; now handles non-unique columns and returns columns in original order (except for the case of two DataFrames with `pairwise=False`, where behavior is unchanged) (GH7542)

- Bug in `rolling_count()` and `expanding_*()` functions unnecessarily producing error message for zero-length data (GH8056)

- Bug in `rolling_apply()` and `expanding_apply()` interpreting `min_periods=0` as `min_periods=1` (GH8080)

- Bug in `expanding_std()` and `expanding_var()` for a single value producing a confusing error message (GH7900)

- Bug in `rolling_std()` and `rolling_var()` for a single value producing `0` rather than `NaN` (GH7900)

- Bug in `ewmstd()`, `ewmvol()`, `ewmvar()`, and `ewmcov()` calculation of de-biasing factors when `bias=False` (the default). Previously an incorrect constant factor was used, based on `adjust=True`, `ignore_na=True`, and an infinite number of observations. Now a different factor is used for each entry, based on the actual weights (analogous to the usual `N/(N-1)` factor). In particular, for a single point a value of `NaN` is returned when `bias=False`, whereas previously a value of (approximately) `0` was returned.

  For example, consider the following pre-0.15.0 results for `ewmvar(..., bias=False)`, and the corresponding debiasing factors:

      In [67]: s = pd.Series([1., 2., 0., 4.])

      In [89]: ewmvar(s, com=2., bias=False)
      Out[89]:
      0   -2.775558e-16
      1    3.000000e-01
      2    9.556787e-01
      3    3.585799e+00
      dtype: float64

      In [90]: ewmvar(s, com=2., bias=False) / ewmvar(s, com=2., bias=True)
      Out[90]:
      0    1.25
      1    1.25
      2    1.25
      3    1.25
      dtype: float64

  Note that entry `0` is approximately 0, and the debiasing factors are a constant 1.25. By comparison, the following 0.15.0 results have a `NaN` for entry `0`, and the debiasing factors are decreasing (towards 1.25):

      In [14]: pd.ewmvar(s, com=2., bias=False)
      Out[14]:
      0         NaN
      1    0.500000
      2    1.210526
      3    4.089069
      dtype: float64

      In [15]: pd.ewmvar(s, com=2., bias=False) / pd.ewmvar(s, com=2., bias=True)
      Out[15]:
      0         NaN
      1    2.083333
      2    1.583333
      3    1.425439
      dtype: float64

  See Exponentially weighted moment functions for details. (GH7912)

### Improvements in the sql io module

- Added support for a `chunksize` parameter to `to_sql` function. This allows DataFrame to be written in chunks and avoid packet-size overflow errors (GH8062).

- Added support for a `chunksize` parameter to `read_sql` function. Specifying this argument will return an iterator through chunks of the query result (GH2908).

- Added support for writing `datetime.date` and `datetime.time` object columns with `to_sql` (GH6932).

- Added support for specifying a `schema` to read from/write to with `read_sql_table` and `to_sql` (GH7441, GH7952). For example:

      df.to_sql('table', engine, schema='other_schema')  # noqa F821
      pd.read_sql_table('table', engine, schema='other_schema')  # noqa F821

- Added support for writing `NaN` values with `to_sql` (GH2754).

- Added support for writing datetime64 columns with `to_sql` for all database flavors (GH7103).

## Backwards incompatible API changes

### Breaking changes

API changes related to `Categorical` (see here for more details):

- The `Categorical` constructor with two arguments changed from “codes/labels and levels” to “values and levels (now called ‘categories’)”. This can lead to subtle bugs. If you use `Categorical` directly, please audit your code by changing it to use the `from_codes()` constructor.

  An old function call like (prior to 0.15.0):

      pd.Categorical([0,1,0,2,1], levels=['a', 'b', 'c'])

  will have to adapted to the following to keep the same behaviour:

      In [2]: pd.Categorical.from_codes([0,1,0,2,1], categories=['a', 'b', 'c'])
      Out[2]:
      [a, b, a, c, b]
      Categories (3, object): [a, b, c]

API changes related to the introduction of the `Timedelta` scalar (see above for more details):

- Prior to 0.15.0 `to_timedelta()` would return a `Series` for list-like/Series input, and a `np.timedelta64` for scalar input. It will now return a `TimedeltaIndex` for list-like input, `Series` for Series input, and `Timedelta` for scalar input.

For API changes related to the rolling and expanding functions, see detailed overview above.

Other notable API changes:

- Consistency when indexing with `.loc` and a list-like indexer when no values are found.

      In [68]: df = pd.DataFrame([['a'], ['b']], index=[1, 2])

      In [69]: df
      Out[69]:
         0
      1  a
      2  b

      [2 rows x 1 columns]

  In prior versions there was a difference in these two constructs:

  - `df.loc[[3]]` would return a frame reindexed by 3 (with all `np.nan` values)
  - `df.loc[[3],:]` would raise `KeyError`.

  Both will now raise a `KeyError`. The rule is that at least 1 indexer must be found when using a list-like and `.loc` (GH7999)

  Furthermore in prior versions these were also different:

  - `df.loc[[1,3]]` would return a frame reindexed by [1,3]
  - `df.loc[[1,3],:]` would raise `KeyError`.

  Both will now return a frame reindex by [1,3]. E.g.

      In [3]: df.loc[[1, 3]]
      Out[3]:
           0
      1    a
      3  NaN

      In [4]: df.loc[[1, 3], :]
      Out[4]:
           0
      1    a
      3  NaN

  This can also be seen in multi-axis indexing with a `Panel`.

      >>> p = pd.Panel(np.arange(2 * 3 * 4).reshape(2, 3, 4),
      ...              items=['ItemA', 'ItemB'],
      ...              major_axis=[1, 2, 3],
      ...              minor_axis=['A', 'B', 'C', 'D'])
      >>> p
      <class 'pandas.core.panel.Panel'>
      Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
      Items axis: ItemA to ItemB
      Major_axis axis: 1 to 3
      Minor_axis axis: A to D

  The following would raise `KeyError` prior to 0.15.0:

      In [5]:
      Out[5]:
         ItemA  ItemD
      1      3    NaN
      2      7    NaN
      3     11    NaN

  Furthermore, `.loc` will raise If no values are found in a MultiIndex with a list-like indexer:

      In [70]: s = pd.Series(np.arange(3, dtype='int64'),
         ....:               index=pd.MultiIndex.from_product([['A'],
         ....:                                                ['foo', 'bar', 'baz']],
         ....:                                                names=['one', 'two'])
         ....:               ).sort_index()
         ....:

      In [71]: s
      Out[71]:
      one  two
      A    bar    1
           baz    2
           foo    0
      Length: 3, dtype: int64

      In [72]: try:
         ....:     s.loc[['D']]
         ....: except KeyError as e:
         ....:     print("KeyError: " + str(e))
         ....:

- Assigning values to `None` now considers the dtype when choosing an ‘empty’ value (GH7941).

  Previously, assigning to `None` in numeric containers changed the dtype to object (or errored, depending on the call). It now uses `NaN`:

      In [73]: s = pd.Series([1, 2, 3])

      In [74]: s.loc[0] = None

      In [75]: s
      Out[75]:
      0    NaN
      1    2.0
      2    3.0
      Length: 3, dtype: float64

  `NaT` is now used similarly for datetime containers.

  For object containers, we now preserve `None` values (previously these were converted to `NaN` values).

      In [76]: s = pd.Series(["a", "b", "c"])

      In [77]: s.loc[0] = None

      In [78]: s
      Out[78]:
      0    None
      1       b
      2       c
      Length: 3, dtype: object

  To insert a `NaN`, you must explicitly use `np.nan`. See the docs.

- In prior versions, updating a pandas object inplace would not reflect in other python references to this object. (GH8511, GH5104)

      In [79]: s = pd.Series([1, 2, 3])

      In [80]: s2 = s

      In [81]: s += 1.5

  Behavior prior to v0.15.0

      # the original object
      In [5]: s
      Out[5]:
      0    2.5
      1    3.5
      2    4.5
      dtype: float64

      # a reference to the original object
      In [7]: s2
      Out[7]:
      0    1
      1    2
      2    3
      dtype: int64

  This is now the correct behavior

      # the original object
      In [82]: s
      Out[82]:
      0    2.5
      1    3.5
      2    4.5
      Length: 3, dtype: float64

      # a reference to the original object
      In [83]: s2
      Out[83]:
      0    2.5
      1    3.5
      2    4.5
      Length: 3, dtype: float64

- Made both the C-based and Python engines for read_csv and read_table ignore empty lines in input as well as white space-filled lines, as long as `sep` is not white space. This is an API change that can be controlled by the keyword parameter `skip_blank_lines`. See the docs (GH4466)

- A timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone and inserted as `object` dtype rather than being converted to a naive `datetime64[ns]` (GH8411).

- Bug in passing a `DatetimeIndex` with a timezone that was not being retained in DataFrame construction from a dict (GH7822)

  In prior versions this would drop the timezone, now it retains the timezone, but gives a column of `object` dtype:

      In [84]: i = pd.date_range('1/1/2011', periods=3, freq='10s', tz='US/Eastern')

      In [85]: i
      Out[85]:
      DatetimeIndex(['2011-01-01 00:00:00-05:00', '2011-01-01 00:00:10-05:00',
                     '2011-01-01 00:00:20-05:00'],
                    dtype='datetime64[ns, US/Eastern]', freq='10S')

      In [86]: df = pd.DataFrame({'a': i})

      In [87]: df
      Out[87]:
                                a
      0 2011-01-01 00:00:00-05:00
      1 2011-01-01 00:00:10-05:00
      2 2011-01-01 00:00:20-05:00

      [3 rows x 1 columns]

      In [88]: df.dtypes
      Out[88]:
      a    datetime64[ns, US/Eastern]
      Length: 1, dtype: object

  Previously this would have yielded a column of `datetime64` dtype, but without timezone info.

  The behaviour of assigning a column to an existing dataframe as df[‘a’] = i remains unchanged (this already returned an `object` column with a timezone).

- When passing multiple levels to `stack()`, it will now raise a `ValueError` when the levels aren’t all level names or all level numbers (GH7660). See Reshaping by stacking and unstacking.

- Raise a `ValueError` in `df.to_hdf` with ‘fixed’ format, if `df` has non-unique columns as the resulting file will be broken (GH7761)

- `SettingWithCopy` raise/warnings (according to the option `mode.chained_assignment`) will now be issued when setting a value on a sliced mixed-dtype DataFrame using chained-assignment. (GH7845, GH7950)

      In [1]: df = pd.DataFrame(np.arange(0, 9), columns=['count'])

      In [2]: df['group'] = 'b'

      In [3]: df.iloc[0:5]['group'] = 'a'
      /usr/local/bin/ipython:1: SettingWithCopyWarning:
      A value is trying to be set on a copy of a slice from a DataFrame.
      Try using .loc[row_indexer,col_indexer] = value instead

      See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

- `merge`, `DataFrame.merge`, and `ordered_merge` now return the same type as the `left` argument (GH7737).

- Previously an enlargement with a mixed-dtype frame would act unlike `.append` which will preserve dtypes (related GH2578, GH8176):

      In [89]: df = pd.DataFrame([[True, 1], [False, 2]],
         ....:                   columns=["female", "fitness"])
         ....:

      In [90]: df
      Out[90]:
         female  fitness
      0    True        1
      1   False        2

      [2 rows x 2 columns]

      In [91]: df.dtypes
      Out[91]:
      female      bool
      fitness    int64
      Length: 2, dtype: object

      # dtypes are now preserved
      In [92]: df.loc[2] = df.loc[1]

      In [93]: df
      Out[93]:
         female  fitness
      0    True        1
      1   False        2
      2   False        2

      [3 rows x 2 columns]

      In [94]: df.dtypes
      Out[94]:
      female      bool
      fitness    int64
      Length: 2, dtype: object

- `Series.to_csv()` now returns a string when `path=None`, matching the behaviour of `DataFrame.to_csv()` (GH8215).

- `read_hdf` now raises `IOError` when a file that doesn’t exist is passed in. Previously, a new, empty file was created, and a `KeyError` raised (GH7715).

- `DataFrame.info()` now ends its output with a newline character (GH8114)

- Concatenating no objects will now raise a `ValueError` rather than a bare `Exception`.

- Merge errors will now be sub-classes of `ValueError` rather than raw `Exception` (GH8501)

- `DataFrame.plot` and `Series.plot` keywords are now have consistent orders (GH8037)

### Internal refactoring

In 0.15.0 `Index` has internally been refactored to no longer sub-class `ndarray` but instead subclass `PandasObject`, similarly to the rest of the pandas objects. This change allows very easy sub-classing and creation of new index types. This should be a transparent change with only very limited API implications (GH5080, GH7439, GH7796, GH8024, GH8367, GH7997, GH8522):

- you may need to unpickle pandas version < 0.15.0 pickles using `pd.read_pickle` rather than `pickle.load`. See pickle docs
- when plotting with a `PeriodIndex`, the matplotlib internal axes will now be arrays of `Period` rather than a `PeriodIndex` (this is similar to how a `DatetimeIndex` passes arrays of `datetimes` now)
- MultiIndexes will now raise similarly to other pandas objects w.r.t. truth testing, see here (GH7897).
- When plotting a DatetimeIndex directly with matplotlib’s plot function, the axis labels will no longer be formatted as dates but as integers (the internal representation of a `datetime64`). UPDATE This is fixed in 0.15.1, see here.

### Deprecations

- The attributes `Categorical` `labels` and `levels` attributes are deprecated and renamed to `codes` and `categories`.
- The `outtype` argument to `pd.DataFrame.to_dict` has been deprecated in favor of `orient`. (GH7840)
- The `convert_dummies` method has been deprecated in favor of `get_dummies` (GH8140)
- The `infer_dst` argument in `tz_localize` will be deprecated in favor of `ambiguous` to allow for more flexibility in dealing with DST transitions. Replace `infer_dst=True` with `ambiguous='infer'` for the same behavior (GH7943). See the docs for more details.
- The top-level `pd.value_range` has been deprecated and can be replaced by `.describe()` (GH8481)

- The `Index` set operations `+` and `-` were deprecated in order to provide these for numeric type operations on certain index types. `+` can be replaced by `.union()` or `|`, and `-` by `.difference()`. Further the method name `Index.diff()` is deprecated and can be replaced by `Index.difference()` (GH8226)

      # +
      pd.Index(['a', 'b', 'c']) + pd.Index(['b', 'c', 'd'])

      # should be replaced by
      pd.Index(['a', 'b', 'c']).union(pd.Index(['b', 'c', 'd']))

      # -
      pd.Index(['a', 'b', 'c']) - pd.Index(['b', 'c', 'd'])

      # should be replaced by
      pd.Index(['a', 'b', 'c']).difference(pd.Index(['b', 'c', 'd']))

- The `infer_types` argument to `read_html()` now has no effect and is deprecated (GH7762, GH7032).

### Removal of prior version deprecations/changes

- Remove `DataFrame.delevel` method in favor of `DataFrame.reset_index`

## Enhancements

Enhancements in the importing/exporting of Stata files:

- Added support for bool, uint8, uint16 and uint32 data types in `to_stata` (GH7097, GH7365)
- Added conversion option when importing Stata files (GH8527)
- `DataFrame.to_stata` and `StataWriter` check string length for compatibility with limitations imposed in dta files where fixed-width strings must contain 244 or fewer characters. Attempting to write Stata dta files with strings longer than 244 characters raises a `ValueError`. (GH7858)
- `read_stata` and `StataReader` can import missing data information into a `DataFrame` by setting the argument `convert_missing` to `True`. When using this options, missing values are returned as `StataMissingValue` objects and columns containing missing values have `object` data type. (GH8045)

Enhancements in the plotting functions:

- Added `layout` keyword to `DataFrame.plot`. You can pass a tuple of `(rows, columns)`, one of which can be `-1` to automatically infer (GH6667, GH8071).
- Allow to pass multiple axes to `DataFrame.plot`, `hist` and `boxplot` (GH5353, GH6970, GH7069)
- Added support for `c`, `colormap` and `colorbar` arguments for `DataFrame.plot` with `kind='scatter'` (GH7780)
- Histogram from `DataFrame.plot` with `kind='hist'` (GH7809), See the docs.
- Boxplot from `DataFrame.plot` with `kind='box'` (GH7998), See the docs.

Other:

- `read_csv` now has a keyword parameter `float_precision` which specifies which floating-point converter the C engine should use during parsing, see here (GH8002, GH8044)

- Added `searchsorted` method to `Series` objects (GH7447)

- `describe()` on mixed-types DataFrames is more flexible. Type-based column filtering is now possible via the `include`/`exclude` arguments. See the docs (GH8164).

      In [95]: df = pd.DataFrame({'catA': ['foo', 'foo', 'bar'] * 8,
         ....:                    'catB': ['a', 'b', 'c', 'd'] * 6,
         ....:                    'numC': np.arange(24),
         ....:                    'numD': np.arange(24.) + .5})
         ....:

      In [96]: df.describe(include=["object"])
      Out[96]:
             catA catB
      count    24   24
      unique    2    4
      top     foo    c
      freq     16    6

      [4 rows x 2 columns]

      In [97]: df.describe(include=["number", "object"], exclude=["float"])
      Out[97]:
             catA catB       numC
      count    24   24  24.000000
      unique    2    4        NaN
      top     foo    c        NaN
      freq     16    6        NaN
      mean    NaN  NaN  11.500000
      std     NaN  NaN   7.071068
      min     NaN  NaN   0.000000
      25%     NaN  NaN   5.750000
      50%     NaN  NaN  11.500000
      75%     NaN  NaN  17.250000
      max     NaN  NaN  23.000000

      [11 rows x 3 columns]

  Requesting all columns is possible with the shorthand ‘all’

      In [98]: df.describe(include='all')
      Out[98]:
             catA catB       numC       numD
      count    24   24  24.000000  24.000000
      unique    2    4        NaN        NaN
      top     foo    c        NaN        NaN
      freq     16    6        NaN        NaN
      mean    NaN  NaN  11.500000  12.000000
      std     NaN  NaN   7.071068   7.071068
      min     NaN  NaN   0.000000   0.500000
      25%     NaN  NaN   5.750000   6.250000
      50%     NaN  NaN  11.500000  12.000000
      75%     NaN  NaN  17.250000  17.750000
      max     NaN  NaN  23.000000  23.500000

      [11 rows x 4 columns]

  Without those arguments, `describe` will behave as before, including only numerical columns or, if none are, only categorical columns. See also the docs

- Added `split` as an option to the `orient` argument in `pd.DataFrame.to_dict`. (GH7840)

- The `get_dummies` method can now be used on DataFrames. By default only categorical columns are encoded as 0’s and 1’s, while other columns are left untouched.

      In [99]: df = pd.DataFrame({'A': ['a', 'b', 'a'], 'B': ['c', 'c', 'b'],
         ....:                 'C': [1, 2, 3]})
         ....:

      In [100]: pd.get_dummies(df)
      Out[100]:
         C  A_a  A_b  B_b  B_c
      0  1    1    0    0    1
      1  2    0    1    0    1
      2  3    1    0    1    0

      [3 rows x 5 columns]

- `PeriodIndex` supports `resolution` as the same as `DatetimeIndex` (GH7708)

- `pandas.tseries.holiday` has added support for additional holidays and ways to observe holidays (GH7070)

- `pandas.tseries.holiday.Holiday` now supports a list of offsets in Python3 (GH7070)

- `pandas.tseries.holiday.Holiday` now supports a days_of_week parameter (GH7070)

- `GroupBy.nth()` now supports selecting multiple nth values (GH7910)

      In [101]: business_dates = pd.date_range(start='4/1/2014', end='6/30/2014', freq='B')

      In [102]: df = pd.DataFrame(1, index=business_dates, columns=['a', 'b'])

      # get the first, 4th, and last date index for each month
      In [103]: df.groupby([df.index.year, df.index.month]).nth([0, 3, -1])
      Out[103]:
              a  b
      2014 4  1  1
           4  1  1
           4  1  1
           5  1  1
           5  1  1
           5  1  1
           6  1  1
           6  1  1
           6  1  1

      [9 rows x 2 columns]

- `Period` and `PeriodIndex` supports addition/subtraction with `timedelta`-likes (GH7966)

  If `Period` freq is `D`, `H`, `T`, `S`, `L`, `U`, `N`, `Timedelta`-like can be added if the result can have same freq. Otherwise, only the same `offsets` can be added.

      In [104]: idx = pd.period_range('2014-07-01 09:00', periods=5, freq='H')

      In [105]: idx
      Out[105]:
      PeriodIndex(['2014-07-01 09:00', '2014-07-01 10:00', '2014-07-01 11:00',
                   '2014-07-01 12:00', '2014-07-01 13:00'],
                  dtype='period[H]', freq='H')

      In [106]: idx + pd.offsets.Hour(2)
      Out[106]:
      PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00',
                   '2014-07-01 14:00', '2014-07-01 15:00'],
                  dtype='period[H]', freq='H')

      In [107]: idx + pd.Timedelta('120m')
      Out[107]:
      PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00',
                   '2014-07-01 14:00', '2014-07-01 15:00'],
                  dtype='period[H]', freq='H')

      In [108]: idx = pd.period_range('2014-07', periods=5, freq='M')

      In [109]: idx
      Out[109]: PeriodIndex(['2014-07', '2014-08', '2014-09', '2014-10', '2014-11'], dtype='period[M]', freq='M')

      In [110]: idx + pd.offsets.MonthEnd(3)
      Out[110]: PeriodIndex(['2014-10', '2014-11', '2014-12', '2015-01', '2015-02'], dtype='period[M]', freq='M')

- Added experimental compatibility with `openpyxl` for versions >= 2.0. The `DataFrame.to_excel` method `engine` keyword now recognizes `openpyxl1` and `openpyxl2` which will explicitly require openpyxl v1 and v2 respectively, failing if the requested version is not available. The `openpyxl` engine is a now a meta-engine that automatically uses whichever version of openpyxl is installed. (GH7177)

- `DataFrame.fillna` can now accept a `DataFrame` as a fill value (GH8377)

- Passing multiple levels to `stack()` will now work when multiple level numbers are passed (GH7660). See Reshaping by stacking and unstacking.

- `set_names()`, `set_labels()`, and `set_levels()` methods now take an optional `level` keyword argument to all modification of specific level(s) of a MultiIndex. Additionally `set_names()` now accepts a scalar string value when operating on an `Index` or on a specific level of a `MultiIndex` (GH7792)

      In [111]: idx = pd.MultiIndex.from_product([['a'], range(3), list("pqr")],
         .....:                                  names=['foo', 'bar', 'baz'])
         .....:

      In [112]: idx.set_names('qux', level=0)
      Out[112]:
      MultiIndex([('a', 0, 'p'),
                  ('a', 0, 'q'),
                  ('a', 0, 'r'),
                  ('a', 1, 'p'),
                  ('a', 1, 'q'),
                  ('a', 1, 'r'),
                  ('a', 2, 'p'),
                  ('a', 2, 'q'),
                  ('a', 2, 'r')],
                 names=['qux', 'bar', 'baz'])

      In [113]: idx.set_names(['qux', 'corge'], level=[0, 1])
      Out[113]:
      MultiIndex([('a', 0, 'p'),
                  ('a', 0, 'q'),
                  ('a', 0, 'r'),
                  ('a', 1, 'p'),
                  ('a', 1, 'q'),
                  ('a', 1, 'r'),
                  ('a', 2, 'p'),
                  ('a', 2, 'q'),
                  ('a', 2, 'r')],
                 names=['qux', 'corge', 'baz'])

      In [114]: idx.set_levels(['a', 'b', 'c'], level='bar')
      Out[114]:
      MultiIndex([('a', 'a', 'p'),
                  ('a', 'a', 'q'),
                  ('a', 'a', 'r'),
                  ('a', 'b', 'p'),
                  ('a', 'b', 'q'),
                  ('a', 'b', 'r'),
                  ('a', 'c', 'p'),
                  ('a', 'c', 'q'),
                  ('a', 'c', 'r')],
                 names=['foo', 'bar', 'baz'])

      In [115]: idx.set_levels([['a', 'b', 'c'], [1, 2, 3]], level=[1, 2])
      Out[115]:
      MultiIndex([('a', 'a', 1),
                  ('a', 'a', 2),
                  ('a', 'a', 3),
                  ('a', 'b', 1),
                  ('a', 'b', 2),
                  ('a', 'b', 3),
                  ('a', 'c', 1),
                  ('a', 'c', 2),
                  ('a', 'c', 3)],
                 names=['foo', 'bar', 'baz'])

- `Index.isin` now supports a `level` argument to specify which index level to use for membership tests (GH7892, GH7890)

      In [1]: idx = pd.MultiIndex.from_product([[0, 1], ['a', 'b', 'c']])

      In [2]: idx.values
      Out[2]: array([(0, 'a'), (0, 'b'), (0, 'c'), (1, 'a'), (1, 'b'), (1, 'c')], dtype=object)

      In [3]: idx.isin(['a', 'c', 'e'], level=1)
      Out[3]: array([ True, False,  True,  True, False,  True], dtype=bool)

- `Index` now supports `duplicated` and `drop_duplicates`. (GH4060)

      In [116]: idx = pd.Index([1, 2, 3, 4, 1, 2])

      In [117]: idx
      Out[117]: Int64Index([1, 2, 3, 4, 1, 2], dtype='int64')

      In [118]: idx.duplicated()
      Out[118]: array([False, False, False, False,  True,  True])

      In [119]: idx.drop_duplicates()
      Out[119]: Int64Index([1, 2, 3, 4], dtype='int64')

- add `copy=True` argument to `pd.concat` to enable pass through of complete blocks (GH8252)

- Added support for numpy 1.8+ data types (`bool_`, `int_`, `float_`, `string_`) for conversion to R dataframe (GH8400)

## Performance

- Performance improvements in `DatetimeIndex.__iter__` to allow faster iteration (GH7683)
- Performance improvements in `Period` creation (and `PeriodIndex` setitem) (GH5155)
- Improvements in Series.transform for significant performance gains (revised) (GH6496)
- Performance improvements in `StataReader` when reading large files (GH8040, GH8073)
- Performance improvements in `StataWriter` when writing large files (GH8079)
- Performance and memory usage improvements in multi-key `groupby` (GH8128)
- Performance improvements in groupby `.agg` and `.apply` where builtins max/min were not mapped to numpy/cythonized versions (GH7722)
- Performance improvement in writing to sql (`to_sql`) of up to 50% (GH8208).
- Performance benchmarking of groupby for large value of ngroups (GH6787)
- Performance improvement in `CustomBusinessDay`, `CustomBusinessMonth` (GH8236)
- Performance improvement for `MultiIndex.values` for multi-level indexes containing datetimes (GH8543)

## Bug fixes

- Bug in pivot_table, when using margins and a dict aggfunc (GH8349)
- Bug in `read_csv` where `squeeze=True` would return a view (GH8217)
- Bug in checking of table name in `read_sql` in certain cases (GH7826).
- Bug in `DataFrame.groupby` where `Grouper` does not recognize level when frequency is specified (GH7885)
- Bug in multiindexes dtypes getting mixed up when DataFrame is saved to SQL table (GH8021)
- Bug in `Series` 0-division with a float and integer operand dtypes (GH7785)
- Bug in `Series.astype("unicode")` not calling `unicode` on the values correctly (GH7758)
- Bug in `DataFrame.as_matrix()` with mixed `datetime64[ns]` and `timedelta64[ns]` dtypes (GH7778)
- Bug in `HDFStore.select_column()` not preserving UTC timezone info when selecting a `DatetimeIndex` (GH7777)
- Bug in `to_datetime` when `format='%Y%m%d'` and `coerce=True` are specified, where previously an object array was returned (rather than a coerced time-series with `NaT`), (GH7930)
- Bug in `DatetimeIndex` and `PeriodIndex` in-place addition and subtraction cause different result from normal one (GH6527)
- Bug in adding and subtracting `PeriodIndex` with `PeriodIndex` raise `TypeError` (GH7741)
- Bug in `combine_first` with `PeriodIndex` data raises `TypeError` (GH3367)
- Bug in MultiIndex slicing with missing indexers (GH7866)
- Bug in MultiIndex slicing with various edge cases (GH8132)
- Regression in MultiIndex indexing with a non-scalar type object (GH7914)
- Bug in `Timestamp` comparisons with `==` and `int64` dtype (GH8058)
- Bug in pickles contains `DateOffset` may raise `AttributeError` when `normalize` attribute is referred internally (GH7748)
- Bug in `Panel` when using `major_xs` and `copy=False` is passed (deprecation warning fails because of missing `warnings`) (GH8152).
- Bug in pickle deserialization that failed for pre-0.14.1 containers with dup items trying to avoid ambiguity when matching block and manager items, when there’s only one block there’s no ambiguity (GH7794)
- Bug in putting a `PeriodIndex` into a `Series` would convert to `int64` dtype, rather than `object` of `Periods` (GH7932)
- Bug in `HDFStore` iteration when passing a where (GH8014)
- Bug in `DataFrameGroupby.transform` when transforming with a passed non-sorted key (GH8046, GH8430)
- Bug in repeated timeseries line and area plot may result in `ValueError` or incorrect kind (GH7733)
- Bug in inference in a `MultiIndex` with `datetime.date` inputs (GH7888)
- Bug in `get` where an `IndexError` would not cause the default value to be returned (GH7725)
- Bug in `offsets.apply`, `rollforward` and `rollback` may reset nanosecond (GH7697)
- Bug in `offsets.apply`, `rollforward` and `rollback` may raise `AttributeError` if `Timestamp` has `dateutil` tzinfo (GH7697)
- Bug in sorting a MultiIndex frame with a `Float64Index` (GH8017)
- Bug in inconsistent panel setitem with a rhs of a `DataFrame` for alignment (GH7763)
- Bug in `is_superperiod` and `is_subperiod` cannot handle higher frequencies than `S` (GH7760, GH7772, GH7803)
- Bug in 32-bit platforms with `Series.shift` (GH8129)
- Bug in `PeriodIndex.unique` returns int64 `np.ndarray` (GH7540)
- Bug in `groupby.apply` with a non-affecting mutation in the function (GH8467)
- Bug in `DataFrame.reset_index` which has `MultiIndex` contains `PeriodIndex` or `DatetimeIndex` with tz raises `ValueError` (GH7746, GH7793)
- Bug in `DataFrame.plot` with `subplots=True` may draw unnecessary minor xticks and yticks (GH7801)
- Bug in `StataReader` which did not read variable labels in 117 files due to difference between Stata documentation and implementation (GH7816)
- Bug in `StataReader` where strings were always converted to 244 characters-fixed width irrespective of underlying string size (GH7858)
- Bug in `DataFrame.plot` and `Series.plot` may ignore `rot` and `fontsize` keywords (GH7844)
- Bug in `DatetimeIndex.value_counts` doesn’t preserve tz (GH7735)
- Bug in `PeriodIndex.value_counts` results in `Int64Index` (GH7735)
- Bug in `DataFrame.join` when doing left join on index and there are multiple matches (GH5391)
- Bug in `GroupBy.transform()` where int groups with a transform that didn’t preserve the index were incorrectly truncated (GH7972).
- Bug in `groupby` where callable objects without name attributes would take the wrong path, and produce a `DataFrame` instead of a `Series` (GH7929)
- Bug in `groupby` error message when a DataFrame grouping column is duplicated (GH7511)
- Bug in `read_html` where the `infer_types` argument forced coercion of date-likes incorrectly (GH7762, GH7032).
- Bug in `Series.str.cat` with an index which was filtered as to not include the first item (GH7857)
- Bug in `Timestamp` cannot parse `nanosecond` from string (GH7878)
- Bug in `Timestamp` with string offset and `tz` results incorrect (GH7833)
- Bug in `tslib.tz_convert` and `tslib.tz_convert_single` may return different results (GH7798)
- Bug in `DatetimeIndex.intersection` of non-overlapping timestamps with tz raises `IndexError` (GH7880)
- Bug in alignment with TimeOps and non-unique indexes (GH8363)
- Bug in `GroupBy.filter()` where fast path vs. slow path made the filter return a non scalar value that appeared valid but wasn’t (GH7870).
- Bug in `date_range()`/`DatetimeIndex()` when the timezone was inferred from input dates yet incorrect times were returned when crossing DST boundaries (GH7835, GH7901).
- Bug in `to_excel()` where a negative sign was being prepended to positive infinity and was absent for negative infinity (GH7949)
- Bug in area plot draws legend with incorrect `alpha` when `stacked=True` (GH8027)
- `Period` and `PeriodIndex` addition/subtraction with `np.timedelta64` results in incorrect internal representations (GH7740)
- Bug in `Holiday` with no offset or observance (GH7987)
- Bug in `DataFrame.to_latex` formatting when columns or index is a `MultiIndex` (GH7982).
- Bug in `DateOffset` around Daylight Savings Time produces unexpected results (GH5175).
- Bug in `DataFrame.shift` where empty columns would throw `ZeroDivisionError` on numpy 1.7 (GH8019)
- Bug in installation where `html_encoding/*.html` wasn’t installed and therefore some tests were not running correctly (GH7927).
- Bug in `read_html` where `bytes` objects were not tested for in `_read` (GH7927).
- Bug in `DataFrame.stack()` when one of the column levels was a datelike (GH8039)
- Bug in broadcasting numpy scalars with `DataFrame` (GH8116)
- Bug in `pivot_table` performed with nameless `index` and `columns` raises `KeyError` (GH8103)
- Bug in `DataFrame.plot(kind='scatter')` draws points and errorbars with different colors when the color is specified by `c` keyword (GH8081)
- Bug in `Float64Index` where `iat` and `at` were not testing and were failing (GH8092).
- Bug in `DataFrame.boxplot()` where y-limits were not set correctly when producing multiple axes (GH7528, GH5517).
- Bug in `read_csv` where line comments were not handled correctly given a custom line terminator or `delim_whitespace=True` (GH8122).
- Bug in `read_html` where empty tables caused a `StopIteration` (GH7575)
- Bug in casting when setting a column in a same-dtype block (GH7704)
- Bug in accessing groups from a `GroupBy` when the original grouper was a tuple (GH8121).
- Bug in `.at` that would accept integer indexers on a non-integer index and do fallback (GH7814)
- Bug with kde plot and NaNs (GH8182)
- Bug in `GroupBy.count` with float32 data type were nan values were not excluded (GH8169).
- Bug with stacked barplots and NaNs (GH8175).
- Bug in resample with non evenly divisible offsets (e.g. ‘7s’) (GH8371)
- Bug in interpolation methods with the `limit` keyword when no values needed interpolating (GH7173).
- Bug where `col_space` was ignored in `DataFrame.to_string()` when `header=False` (GH8230).
- Bug with `DatetimeIndex.asof` incorrectly matching partial strings and returning the wrong date (GH8245).
- Bug in plotting methods modifying the global matplotlib rcParams (GH8242).
- Bug in `DataFrame.__setitem__` that caused errors when setting a dataframe column to a sparse array (GH8131)
- Bug where `Dataframe.boxplot()` failed when entire column was empty (GH8181).
- Bug with messed variables in `radviz` visualization (GH8199).
- Bug in interpolation methods with the `limit` keyword when no values needed interpolating (GH7173).
- Bug where `col_space` was ignored in `DataFrame.to_string()` when `header=False` (GH8230).
- Bug in `to_clipboard` that would clip long column data (GH8305)
- Bug in `DataFrame` terminal display: Setting max_column/max_rows to zero did not trigger auto-resizing of dfs to fit terminal width/height (GH7180).
- Bug in OLS where running with “cluster” and “nw_lags” parameters did not work correctly, but also did not throw an error (GH5884).
- Bug in `DataFrame.dropna` that interpreted non-existent columns in the subset argument as the ‘last column’ (GH8303)
- Bug in `Index.intersection` on non-monotonic non-unique indexes (GH8362).
- Bug in masked series assignment where mismatching types would break alignment (GH8387)
- Bug in `NDFrame.equals` gives false negatives with dtype=object (GH8437)
- Bug in assignment with indexer where type diversity would break alignment (GH8258)
- Bug in `NDFrame.loc` indexing when row/column names were lost when target was a list/ndarray (GH6552)
- Regression in `NDFrame.loc` indexing when rows/columns were converted to Float64Index if target was an empty list/ndarray (GH7774)
- Bug in `Series` that allows it to be indexed by a `DataFrame` which has unexpected results. Such indexing is no longer permitted (GH8444)
- Bug in item assignment of a `DataFrame` with MultiIndex columns where right-hand-side columns were not aligned (GH7655)
- Suppress FutureWarning generated by NumPy when comparing object arrays containing NaN for equality (GH7065)
- Bug in `DataFrame.eval()` where the dtype of the `not` operator (`~`) was not correctly inferred as `bool`.

## Contributors

For contributors, please see /usr/share/doc/contributors_list.txt or https://github.com/pandas-dev/pandas/graphs/contributors
Scroll To Top