# Comparison with spreadsheets

Since many potential pandas users have some familiarity with spreadsheet programs like Excel, this page is meant to provide some examples of how various spreadsheet operations would be performed using pandas. This page will use terminology and link to documentation for Excel, but much will be the same/similar in Google Sheets, LibreOffice Calc, Apple Numbers, and other Excel-compatible spreadsheet software.

If you’re new to pandas, you might want to first read through 10 Minutes to pandas to familiarize yourself with the library.

As is customary, we import pandas and NumPy as follows:

    In [1]: import pandas as pd

    In [2]: import numpy as np

## Data structures

### General terminology translation

pandas | Excel
--- | ---
`DataFrame` | worksheet
`Series` | column
`Index` | row headings
row | row
`NaN` | empty cell

### `DataFrame`

A `DataFrame` in pandas is analogous to an Excel worksheet. While an Excel workbook can contain multiple worksheets, pandas `DataFrame`s exist independently.

### `Series`

A `Series` is the data structure that represents one column of a `DataFrame`. Working with a `Series` is analogous to referencing a column of a spreadsheet.

### `Index`

Every `DataFrame` and `Series` has an `Index`, which are labels on the rows of the data. In pandas, if no index is specified, a `RangeIndex` is used by default (first row = 0, second row = 1, and so on), analogous to row headings/numbers in spreadsheets.

In pandas, indexes can be set to one (or multiple) unique values, which is like having a column that is used as the row identifier in a worksheet. Unlike most spreadsheets, these `Index` values can actually be used to reference the rows. (Note that this can be done in Excel with structured references.) For example, in spreadsheets, you would reference the first row as `A1:Z1`, while in pandas you could use `populations.loc['Chicago']`.

Index values are also persistent, so if you re-order the rows in a `DataFrame`, the label for a particular row don’t change.

See the indexing documentation for much more on how to use an `Index` effectively.

### Copies vs. in place operations

Most pandas operations return copies of the `Series`/`DataFrame`. To make the changes “stick”, you’ll need to either assign to a new variable:

>     sorted_df = df.sort_values("col1")

or overwrite the original one:

>     df = df.sort_values("col1")

Note

You will see an `inplace=True` keyword argument available for some methods:

    df.sort_values("col1", inplace=True)

Its use is discouraged. More information.

## Data input / output

### Constructing a DataFrame from values

In a spreadsheet, values can be typed directly into cells.

A pandas `DataFrame` can be constructed in many different ways, but for a small number of values, it is often convenient to specify it as a Python dictionary, where the keys are the column names and the values are the data.

    In [3]: df = pd.DataFrame({"x": [1, 3, 5], "y": [2, 4, 6]})

    In [4]: df
    Out[4]:
       x  y
    0  1  2
    1  3  4
    2  5  6

### Reading external data

Both Excel and pandas can import data from various sources in various formats.

#### CSV

Let’s load and display the tips dataset from the pandas tests, which is a CSV file. In Excel, you would download and then open the CSV. In pandas, you pass the URL or local path of the CSV file to `read_csv()`:

    In [5]: url = (
       ...:     "https://raw.github.com/pandas-dev"
       ...:     "/pandas/master/pandas/tests/io/data/csv/tips.csv"
       ...: )
       ...:

    In [6]: tips = pd.read_csv(url)
    ---------------------------------------------------------------------------
    ConnectionRefusedError                    Traceback (most recent call last)
    /usr/lib/python3.10/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
       1347             try:
    -> 1348                 h.request(req.get_method(), req.selector, req.data, headers,
       1349                           encode_chunked=req.has_header('Transfer-encoding'))

    /usr/lib/python3.10/http/client.py in request(self, method, url, body, headers, encode_chunked)
       1281         """Send a complete request to the server."""
    -> 1282         self._send_request(method, url, body, headers, encode_chunked)
       1283

    /usr/lib/python3.10/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
       1327             body = _encode(body, 'body')
    -> 1328         self.endheaders(body, encode_chunked=encode_chunked)
       1329

    /usr/lib/python3.10/http/client.py in endheaders(self, message_body, encode_chunked)
       1276             raise CannotSendHeader()
    -> 1277         self._send_output(message_body, encode_chunked=encode_chunked)
       1278

    /usr/lib/python3.10/http/client.py in _send_output(self, message_body, encode_chunked)
       1036         del self._buffer[:]
    -> 1037         self.send(msg)
       1038

    /usr/lib/python3.10/http/client.py in send(self, data)
        974             if self.auto_open:
    --> 975                 self.connect()
        976             else:

    /usr/lib/python3.10/http/client.py in connect(self)
       1446
    -> 1447             super().connect()
       1448

    /usr/lib/python3.10/http/client.py in connect(self)
        940         sys.audit("http.client.connect", self, self.host, self.port)
    --> 941         self.sock = self._create_connection(
        942             (self.host,self.port), self.timeout, self.source_address)

    /usr/lib/python3.10/socket.py in create_connection(address, timeout, source_address)
        844         try:
    --> 845             raise err
        846         finally:

    /usr/lib/python3.10/socket.py in create_connection(address, timeout, source_address)
        832                 sock.bind(source_address)
    --> 833             sock.connect(sa)
        834             # Break explicitly a reference cycle

    ConnectionRefusedError: [Errno 111] Connection refused

    During handling of the above exception, another exception occurred:

    URLError                                  Traceback (most recent call last)
    <ipython-input-6-8ab2297b7141> in <module>
    ----> 1 tips = pd.read_csv(url)

    /usr/lib/python3/dist-packages/pandas/util/_decorators.py in wrapper(*args, **kwargs)
        309                     stacklevel=stacklevel,
        310                 )
    --> 311             return func(*args, **kwargs)
        312
        313         return wrapper

    /usr/lib/python3/dist-packages/pandas/io/parsers/readers.py in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options)
        584     kwds.update(kwds_defaults)
        585
    --> 586     return _read(filepath_or_buffer, kwds)
        587
        588

    /usr/lib/python3/dist-packages/pandas/io/parsers/readers.py in _read(filepath_or_buffer, kwds)
        480
        481     # Create the parser.
    --> 482     parser = TextFileReader(filepath_or_buffer, **kwds)
        483
        484     if chunksize or iterator:

    /usr/lib/python3/dist-packages/pandas/io/parsers/readers.py in __init__(self, f, engine, **kwds)
        809             self.options["has_index_names"] = kwds["has_index_names"]
        810
    --> 811         self._engine = self._make_engine(self.engine)
        812
        813     def close(self):

    /usr/lib/python3/dist-packages/pandas/io/parsers/readers.py in _make_engine(self, engine)
       1038             )
       1039         # error: Too many arguments for "ParserBase"
    -> 1040         return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
       1041
       1042     def _failover_to_python(self):

    /usr/lib/python3/dist-packages/pandas/io/parsers/c_parser_wrapper.py in __init__(self, src, **kwds)
         49
         50         # open handles
    ---> 51         self._open_handles(src, kwds)
         52         assert self.handles is not None
         53

    /usr/lib/python3/dist-packages/pandas/io/parsers/base_parser.py in _open_handles(self, src, kwds)
        220         Let the readers open IOHandles after they are done with their potential raises.
        221         """
    --> 222         self.handles = get_handle(
        223             src,
        224             "r",

    /usr/lib/python3/dist-packages/pandas/io/common.py in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
        607
        608     # open URLs
    --> 609     ioargs = _get_filepath_or_buffer(
        610         path_or_buf,
        611         encoding=encoding,

    /usr/lib/python3/dist-packages/pandas/io/common.py in _get_filepath_or_buffer(filepath_or_buffer, encoding, compression, mode, storage_options)
        310         # assuming storage_options is to be interpreted as headers
        311         req_info = urllib.request.Request(filepath_or_buffer, headers=storage_options)
    --> 312         with urlopen(req_info) as req:
        313             content_encoding = req.headers.get("Content-Encoding", None)
        314             if content_encoding == "gzip":

    /usr/lib/python3/dist-packages/pandas/io/common.py in urlopen(*args, **kwargs)
        210     import urllib.request
        211
    --> 212     return urllib.request.urlopen(*args, **kwargs)
        213
        214

    /usr/lib/python3.10/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
        214     else:
        215         opener = _opener
    --> 216     return opener.open(url, data, timeout)
        217
        218 def install_opener(opener):

    /usr/lib/python3.10/urllib/request.py in open(self, fullurl, data, timeout)
        517
        518         sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
    --> 519         response = self._open(req, data)
        520
        521         # post-process response

    /usr/lib/python3.10/urllib/request.py in _open(self, req, data)
        534
        535         protocol = req.type
    --> 536         result = self._call_chain(self.handle_open, protocol, protocol +
        537                                   '_open', req)
        538         if result:

    /usr/lib/python3.10/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
        494         for handler in handlers:
        495             func = getattr(handler, meth_name)
    --> 496             result = func(*args)
        497             if result is not None:
        498                 return result

    /usr/lib/python3.10/urllib/request.py in https_open(self, req)
       1389
       1390         def https_open(self, req):
    -> 1391             return self.do_open(http.client.HTTPSConnection, req,
       1392                 context=self._context, check_hostname=self._check_hostname)
       1393

    /usr/lib/python3.10/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
       1349                           encode_chunked=req.has_header('Transfer-encoding'))
       1350             except OSError as err: # timeout error
    -> 1351                 raise URLError(err)
       1352             r = h.getresponse()
       1353         except:

    URLError: <urlopen error [Errno 111] Connection refused>

    In [7]: tips
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-7-9034fb5f8272> in <module>
    ----> 1 tips

    NameError: name 'tips' is not defined

Like Excel’s Text Import Wizard, `read_csv` can take a number of parameters to specify how the data should be parsed. For example, if the data was instead tab delimited, and did not have column names, the pandas command would be:

    tips = pd.read_csv("tips.csv", sep="\t", header=None)

    # alternatively, read_table is an alias to read_csv with tab delimiter
    tips = pd.read_table("tips.csv", header=None)

#### Excel files

Excel opens various Excel file formats by double-clicking them, or using the Open menu. In pandas, you use special methods for reading and writing from/to Excel files.

Let’s first create a new Excel file based on the `tips` dataframe in the above example:

    tips.to_excel("./tips.xlsx")

Should you wish to subsequently access the data in the `tips.xlsx` file, you can read it into your module using

    tips_df = pd.read_excel("./tips.xlsx", index_col=0)

You have just read in an Excel file using pandas!

### Limiting output

Spreadsheet programs will only show one screenful of data at a time and then allow you to scroll, so there isn’t really a need to limit output. In pandas, you’ll need to put a little more thought into controlling how your `DataFrame`s are displayed.

By default, pandas will truncate output of large `DataFrame`s to show the first and last rows. This can be overridden by changing the pandas options, or using `DataFrame.head()` or `DataFrame.tail()`.

    In [8]: tips.head(5)
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-8-e2693e4d5ad6> in <module>
    ----> 1 tips.head(5)

    NameError: name 'tips' is not defined

### Exporting data

By default, desktop spreadsheet software will save to its respective file format (`.xlsx`, `.ods`, etc). You can, however, save to other file formats.

pandas can create Excel files, CSV, or a number of other formats.

## Data operations

### Operations on columns

In spreadsheets, formulas are often created in individual cells and then dragged into other cells to compute them for other columns. In pandas, you’re able to do operations on whole columns directly.

pandas provides vectorized operations by specifying the individual `Series` in the `DataFrame`. New columns can be assigned in the same way. The `DataFrame.drop()` method drops a column from the `DataFrame`.

    In [9]: tips["total_bill"] = tips["total_bill"] - 2
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-9-30a6f4d646c7> in <module>
    ----> 1 tips["total_bill"] = tips["total_bill"] - 2

    NameError: name 'tips' is not defined

    In [10]: tips["new_bill"] = tips["total_bill"] / 2
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-10-7e2a96df4894> in <module>
    ----> 1 tips["new_bill"] = tips["total_bill"] / 2

    NameError: name 'tips' is not defined

    In [11]: tips
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-11-9034fb5f8272> in <module>
    ----> 1 tips

    NameError: name 'tips' is not defined

    In [12]: tips = tips.drop("new_bill", axis=1)
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-12-2bfa6fa8e9bc> in <module>
    ----> 1 tips = tips.drop("new_bill", axis=1)

    NameError: name 'tips' is not defined

Note that we aren’t having to tell it to do that subtraction cell-by-cell — pandas handles that for us. See how to create new columns derived from existing columns.

### Filtering

In Excel, filtering is done through a graphical menu.

DataFrames can be filtered in multiple ways; the most intuitive of which is using boolean indexing.

    In [13]: tips[tips["total_bill"] > 10]
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-13-4e5df2b556a0> in <module>
    ----> 1 tips[tips["total_bill"] > 10]

    NameError: name 'tips' is not defined

The above statement is simply passing a `Series` of `True`/`False` objects to the DataFrame, returning all rows with `True`.

    In [14]: is_dinner = tips["time"] == "Dinner"
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-14-a38011ae4c7d> in <module>
    ----> 1 is_dinner = tips["time"] == "Dinner"

    NameError: name 'tips' is not defined

    In [15]: is_dinner
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-15-49a5e8deec4a> in <module>
    ----> 1 is_dinner

    NameError: name 'is_dinner' is not defined

    In [16]: is_dinner.value_counts()
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-16-900d99c3802f> in <module>
    ----> 1 is_dinner.value_counts()

    NameError: name 'is_dinner' is not defined

    In [17]: tips[is_dinner]
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-17-ed12d6ad643e> in <module>
    ----> 1 tips[is_dinner]

    NameError: name 'tips' is not defined

### If/then logic

Let’s say we want to make a `bucket` column with values of `low` and `high`, based on whether the `total_bill` is less or more than $10.

In spreadsheets, logical comparison can be done with conditional formulas. We’d use a formula of `=IF(A2 < 10, "low", "high")`, dragged to all cells in a new `bucket` column.

The same operation in pandas can be accomplished using the `where` method from `numpy`.

    In [18]: tips["bucket"] = np.where(tips["total_bill"] < 10, "low", "high")
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-18-a8d4661e6618> in <module>
    ----> 1 tips["bucket"] = np.where(tips["total_bill"] < 10, "low", "high")

    NameError: name 'tips' is not defined

    In [19]: tips
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-19-9034fb5f8272> in <module>
    ----> 1 tips

    NameError: name 'tips' is not defined

### Date functionality

This section will refer to “dates”, but timestamps are handled similarly.

We can think of date functionality in two parts: parsing, and output. In spreadsheets, date values are generally parsed automatically, though there is a DATEVALUE function if you need it. In pandas, you need to explicitly convert plain text to datetime objects, either while reading from a CSV or once in a DataFrame.

Once parsed, spreadsheets display the dates in a default format, though the format can be changed. In pandas, you’ll generally want to keep dates as `datetime` objects while you’re doing calculations with them. Outputting parts of dates (such as the year) is done through date functions in spreadsheets, and datetime properties in pandas.

Given `date1` and `date2` in columns `A` and `B` of a spreadsheet, you might have these formulas:

column | formula
--- | ---
`date1_year` | `=YEAR(A2)`
`date2_month` | `=MONTH(B2)`
`date1_next` | `=DATE(YEAR(A2),MONTH(A2)+1,1)`
`months_between` | `=DATEDIF(A2,B2,"M")`

The equivalent pandas operations are shown below.

    In [20]: tips["date1"] = pd.Timestamp("2013-01-15")
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-20-33d7f805e8ca> in <module>
    ----> 1 tips["date1"] = pd.Timestamp("2013-01-15")

    NameError: name 'tips' is not defined

    In [21]: tips["date2"] = pd.Timestamp("2015-02-15")
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-21-40a3130f91b4> in <module>
    ----> 1 tips["date2"] = pd.Timestamp("2015-02-15")

    NameError: name 'tips' is not defined

    In [22]: tips["date1_year"] = tips["date1"].dt.year
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-22-c9e20f269b76> in <module>
    ----> 1 tips["date1_year"] = tips["date1"].dt.year

    NameError: name 'tips' is not defined

    In [23]: tips["date2_month"] = tips["date2"].dt.month
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-23-ef1fc2805b92> in <module>
    ----> 1 tips["date2_month"] = tips["date2"].dt.month

    NameError: name 'tips' is not defined

    In [24]: tips["date1_next"] = tips["date1"] + pd.offsets.MonthBegin()
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-24-60c45d997a52> in <module>
    ----> 1 tips["date1_next"] = tips["date1"] + pd.offsets.MonthBegin()

    NameError: name 'tips' is not defined

    In [25]: tips["months_between"] = tips["date2"].dt.to_period("M") - tips[
       ....:     "date1"
       ....: ].dt.to_period("M")
       ....:
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-25-a4bc4ed24519> in <module>
    ----> 1 tips["months_between"] = tips["date2"].dt.to_period("M") - tips[
          2     "date1"
          3 ].dt.to_period("M")

    NameError: name 'tips' is not defined

    In [26]: tips[
       ....:     ["date1", "date2", "date1_year", "date2_month", "date1_next", "months_between"]
       ....: ]
       ....:
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-26-606cac654d39> in <module>
    ----> 1 tips[
          2     ["date1", "date2", "date1_year", "date2_month", "date1_next", "months_between"]
          3 ]

    NameError: name 'tips' is not defined

See Time series / date functionality for more details.

### Selection of columns

In spreadsheets, you can select columns you want by:

- Hiding columns

- Deleting columns

- Referencing a range from one worksheet into another

Since spreadsheet columns are typically named in a header row, renaming a column is simply a matter of changing the text in that first cell.

The same operations are expressed in pandas below.

#### Keep certain columns

    In [27]: tips[["sex", "total_bill", "tip"]]
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-27-88ea27a409ff> in <module>
    ----> 1 tips[["sex", "total_bill", "tip"]]

    NameError: name 'tips' is not defined

#### Drop a column

    In [28]: tips.drop("sex", axis=1)
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-28-e6cea0c561ee> in <module>
    ----> 1 tips.drop("sex", axis=1)

    NameError: name 'tips' is not defined

#### Rename a column

    In [29]: tips.rename(columns={"total_bill": "total_bill_2"})
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-29-89e2303a1fc4> in <module>
    ----> 1 tips.rename(columns={"total_bill": "total_bill_2"})

    NameError: name 'tips' is not defined

### Sorting by values

Sorting in spreadsheets is accomplished via the sort dialog.

pandas has a `DataFrame.sort_values()` method, which takes a list of columns to sort by.

    In [30]: tips = tips.sort_values(["sex", "total_bill"])
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-30-9d59129f8932> in <module>
    ----> 1 tips = tips.sort_values(["sex", "total_bill"])

    NameError: name 'tips' is not defined

    In [31]: tips
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-31-9034fb5f8272> in <module>
    ----> 1 tips

    NameError: name 'tips' is not defined

## String processing

### Finding length of string

In spreadsheets, the number of characters in text can be found with the LEN function. This can be used with the TRIM function to remove extra whitespace.

    =LEN(TRIM(A2))

You can find the length of a character string with `Series.str.len()`. In Python 3, all strings are Unicode strings. `len` includes trailing blanks. Use `len` and `rstrip` to exclude trailing blanks.

    In [32]: tips["time"].str.len()
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-32-ef639d164546> in <module>
    ----> 1 tips["time"].str.len()

    NameError: name 'tips' is not defined

    In [33]: tips["time"].str.rstrip().str.len()
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-33-3ffb003cc2cc> in <module>
    ----> 1 tips["time"].str.rstrip().str.len()

    NameError: name 'tips' is not defined

Note this will still include multiple spaces within the string, so isn’t 100% equivalent.

### Finding position of substring

The FIND spreadsheet function returns the position of a substring, with the first character being `1`.

You can find the position of a character in a column of strings with the `Series.str.find()` method. `find` searches for the first position of the substring. If the substring is found, the method returns its position. If not found, it returns `-1`. Keep in mind that Python indexes are zero-based.

    In [34]: tips["sex"].str.find("ale")
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-34-ce45837e4fed> in <module>
    ----> 1 tips["sex"].str.find("ale")

    NameError: name 'tips' is not defined

### Extracting substring by position

Spreadsheets have a MID formula for extracting a substring from a given position. To get the first character:

    =MID(A2,1,1)

With pandas you can use `[]` notation to extract a substring from a string by position locations. Keep in mind that Python indexes are zero-based.

    In [35]: tips["sex"].str[0:1]
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-35-e5ebda285fba> in <module>
    ----> 1 tips["sex"].str[0:1]

    NameError: name 'tips' is not defined

### Extracting nth word

In Excel, you might use the Text to Columns Wizard for splitting text and retrieving a specific column. (Note it’s possible to do so through a formula as well.)

The simplest way to extract words in pandas is to split the strings by spaces, then reference the word by index. Note there are more powerful approaches should you need them.

    In [36]: firstlast = pd.DataFrame({"String": ["John Smith", "Jane Cook"]})

    In [37]: firstlast["First_Name"] = firstlast["String"].str.split(" ", expand=True)[0]

    In [38]: firstlast["Last_Name"] = firstlast["String"].str.rsplit(" ", expand=True)[0]

    In [39]: firstlast
    Out[39]:
           String First_Name Last_Name
    0  John Smith       John      John
    1   Jane Cook       Jane      Jane

### Changing case

Spreadsheets provide UPPER, LOWER, and PROPER functions for converting text to upper, lower, and title case, respectively.

The equivalent pandas methods are `Series.str.upper()`, `Series.str.lower()`, and `Series.str.title()`.

    In [40]: firstlast = pd.DataFrame({"string": ["John Smith", "Jane Cook"]})

    In [41]: firstlast["upper"] = firstlast["string"].str.upper()

    In [42]: firstlast["lower"] = firstlast["string"].str.lower()

    In [43]: firstlast["title"] = firstlast["string"].str.title()

    In [44]: firstlast
    Out[44]:
           string       upper       lower       title
    0  John Smith  JOHN SMITH  john smith  John Smith
    1   Jane Cook   JANE COOK   jane cook   Jane Cook

## Merging

The following tables will be used in the merge examples:

    In [45]: df1 = pd.DataFrame({"key": ["A", "B", "C", "D"], "value": np.random.randn(4)})

    In [46]: df1
    Out[46]:
      key     value
    0   A  0.469112
    1   B -0.282863
    2   C -1.509059
    3   D -1.135632

    In [47]: df2 = pd.DataFrame({"key": ["B", "D", "D", "E"], "value": np.random.randn(4)})

    In [48]: df2
    Out[48]:
      key     value
    0   B  1.212112
    1   D -0.173215
    2   D  0.119209
    3   E -1.044236

In Excel, there are merging of tables can be done through a VLOOKUP.

pandas DataFrames have a `merge()` method, which provides similar functionality. The data does not have to be sorted ahead of time, and different join types are accomplished via the `how` keyword.

    In [49]: inner_join = df1.merge(df2, on=["key"], how="inner")

    In [50]: inner_join
    Out[50]:
      key   value_x   value_y
    0   B -0.282863  1.212112
    1   D -1.135632 -0.173215
    2   D -1.135632  0.119209

    In [51]: left_join = df1.merge(df2, on=["key"], how="left")

    In [52]: left_join
    Out[52]:
      key   value_x   value_y
    0   A  0.469112       NaN
    1   B -0.282863  1.212112
    2   C -1.509059       NaN
    3   D -1.135632 -0.173215
    4   D -1.135632  0.119209

    In [53]: right_join = df1.merge(df2, on=["key"], how="right")

    In [54]: right_join
    Out[54]:
      key   value_x   value_y
    0   B -0.282863  1.212112
    1   D -1.135632 -0.173215
    2   D -1.135632  0.119209
    3   E       NaN -1.044236

    In [55]: outer_join = df1.merge(df2, on=["key"], how="outer")

    In [56]: outer_join
    Out[56]:
      key   value_x   value_y
    0   A  0.469112       NaN
    1   B -0.282863  1.212112
    2   C -1.509059       NaN
    3   D -1.135632 -0.173215
    4   D -1.135632  0.119209
    5   E       NaN -1.044236

`merge` has a number of advantages over `VLOOKUP`:

- The lookup value doesn’t need to be the first column of the lookup table

- If multiple rows are matched, there will be one row for each match, instead of just the first

- It will include all columns from the lookup table, instead of just a single specified column

- It supports more complex join operations

## Other considerations

### Fill Handle

Create a series of numbers following a set pattern in a certain set of cells. In a spreadsheet, this would be done by shift+drag after entering the first number or by entering the first two or three values and then dragging.

This can be achieved by creating a series and assigning it to the desired cells.

    In [57]: df = pd.DataFrame({"AAA": [1] * 8, "BBB": list(range(0, 8))})

    In [58]: df
    Out[58]:
       AAA  BBB
    0    1    0
    1    1    1
    2    1    2
    3    1    3
    4    1    4
    5    1    5
    6    1    6
    7    1    7

    In [59]: series = list(range(1, 5))

    In [60]: series
    Out[60]: [1, 2, 3, 4]

    In [61]: df.loc[2:5, "AAA"] = series

    In [62]: df
    Out[62]:
       AAA  BBB
    0    1    0
    1    1    1
    2    1    2
    3    2    3
    4    3    4
    5    4    5
    6    1    6
    7    1    7

### Drop Duplicates

Excel has built-in functionality for removing duplicate values. This is supported in pandas via `drop_duplicates()`.

    In [63]: df = pd.DataFrame(
       ....:     {
       ....:         "class": ["A", "A", "A", "B", "C", "D"],
       ....:         "student_count": [42, 35, 42, 50, 47, 45],
       ....:         "all_pass": ["Yes", "Yes", "Yes", "No", "No", "Yes"],
       ....:     }
       ....: )
       ....:

    In [64]: df.drop_duplicates()
    Out[64]:
      class  student_count all_pass
    0     A             42      Yes
    1     A             35      Yes
    3     B             50       No
    4     C             47       No
    5     D             45      Yes

    In [65]: df.drop_duplicates(["class", "student_count"])
    Out[65]:
      class  student_count all_pass
    0     A             42      Yes
    1     A             35      Yes
    3     B             50       No
    4     C             47       No
    5     D             45      Yes

### Pivot Tables

PivotTables from spreadsheets can be replicated in pandas through Reshaping and pivot tables. Using the `tips` dataset again, let’s find the average gratuity by size of the party and sex of the server.

In Excel, we use the following configuration for the PivotTable:

The equivalent in pandas:

    In [66]: pd.pivot_table(
       ....:     tips, values="tip", index=["size"], columns=["sex"], aggfunc=np.average
       ....: )
       ....:
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-66-03e8c72d8ade> in <module>
          1 pd.pivot_table(
    ----> 2     tips, values="tip", index=["size"], columns=["sex"], aggfunc=np.average
          3 )

    NameError: name 'tips' is not defined

### Adding a row

Assuming we are using a `RangeIndex` (numbered `0`, `1`, etc.), we can use `DataFrame.append()` to add a row to the bottom of a `DataFrame`.

    In [67]: df
    Out[67]:
      class  student_count all_pass
    0     A             42      Yes
    1     A             35      Yes
    2     A             42      Yes
    3     B             50       No
    4     C             47       No
    5     D             45      Yes

    In [68]: new_row = {"class": "E", "student_count": 51, "all_pass": True}

    In [69]: df.append(new_row, ignore_index=True)
    Out[69]:
      class  student_count all_pass
    0     A             42      Yes
    1     A             35      Yes
    2     A             42      Yes
    3     B             50       No
    4     C             47       No
    5     D             45      Yes
    6     E             51     True

### Find and Replace

Excel’s Find dialog takes you to cells that match, one by one. In pandas, this operation is generally done for an entire column or `DataFrame` at once through conditional expressions.

    In [70]: tips
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-70-9034fb5f8272> in <module>
    ----> 1 tips

    NameError: name 'tips' is not defined

    In [71]: tips == "Sun"
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-71-4aeef4a79fb3> in <module>
    ----> 1 tips == "Sun"

    NameError: name 'tips' is not defined

    In [72]: tips["day"].str.contains("S")
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-72-9d3cde7cf446> in <module>
    ----> 1 tips["day"].str.contains("S")

    NameError: name 'tips' is not defined

pandas’ `replace()` is comparable to Excel’s `Replace All`.

    In [73]: tips.replace("Thu", "Thursday")
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-73-d66dd2975b21> in <module>
    ----> 1 tips.replace("Thu", "Thursday")

    NameError: name 'tips' is not defined