Method Chaining in Pandas
Method Chaining (or Flow Programming in Pandas)¶
This tutorial illustrates how to use method chaining or flow programming in pandas.
Motivation¶
As an R programmer (or someone who uses R almost exclusively for data science at work), I have always been curious to find a dplyr-like technique in Python. But I had no luck until about three weeks ago when I saw Matt Harrison's talk on PyData SLC (Salt Lake City). Matt explains the advantages of using method chaining in this talk, and his explanations got me sold on this technique. Soon after watching the video, I purchased his book, Pandas 1. x Cookbook, 2nd. And I decided to share what I have learned and read so far with those who would love to implement pandas method chaining in their data science workflow.
With that said, let's get started!!¶
Load the Libraries¶
First, we will load numpy, pandas and YouTubeVideo from Ipython.display.
import pandas as pd
import numpy as np
from IPython.display import YouTubeVideo
Next, we will load the dataset. And since our prime objective is to illustrate how to implement method chaining in pandas, we'll do less explanation, as the steps are self-explanatory.
# Import the dataset
ss_census_2008_df = pd.read_csv('00_Data/ss_2008_census_data_raw.csv')
# Inspect the first 5 rows
ss_census_2008_df.head()
# Inspect the last 5 rows
ss_census_2008_df.tail()
Checking for missing values¶
Here we'll check for nas using .isna() and sum methods.
# Check for missing values
ss_census_2008_df.isna().sum()
Transform the data by chaining methods¶
Here we begin by selecting our columns of interest followed by method chaining to wrangling the data. Next, we drop nas with .dropna() method and rename the columns with .rename() method and add a dictionary of old and new column names to it.
To add new columns or modify the existing columns, we use the .assign() method. In the below code chunk, we use the lambda function to split the gender column and retain the value at index 1 (the second value). Next, we create new age categories and use the .replace() method to map them to the old values, convert the population column to the integer type and use the .query() method to exclude rows with the Total in them using gender and age_group columns. Last, we group the data by state, gender and _agegroup columns, compute the sum using the agg() function and .reset_index() method to reset the index.
cols = ['Region Name', 'Variable Name', 'Age Name', '2008']
master_ss_census_2008_df = (ss_census_2008_df
[cols]
.dropna()
.rename(columns = {'Region Name':'state',
'Variable Name':'gender',
'Age Name':'age_group',
'2008':'population'})
.assign(gender = lambda x: x['gender'].str.split('\s+').str[1],
age_group = lambda x:x.age_group.replace(['0 to 4', '5 to 9', '10 to 14',
'15 to 19', '20 to 24', '25 to 29',
'30 to 34', '35 to 39', '40 to 44',
'45 to 49', '50 to 54', '55 to 59',
'60 to 64', '65+'],
['0-14', '0-14', '0-14',
'15-29', '15-29', '15-29',
'30-49', '30-49', '30-49',
'30-49', '50-64', '50-64',
'50-64', '>= 65']),
population = lambda x:x['population'].astype('int')
)
.query('gender != "Total" & age_group != "Total"')
.groupby(['state', 'gender', 'age_group']).agg(population = ('population', 'sum'))
.reset_index()
)
# Inspect the first few rows
master_ss_census_2008_df.head()
Find below the tutorial video¶
YouTubeVideo('WsXGrF6VWqM', width=1000, height=300)
Modify the code for better readability¶
As you can see in the above code, we've condensed the entire transformation steps on this dataset to a single code chunk; however, as we can see in the .rename() and .replace() methods, things can quickly get out of hand, especially if we've several columns and many column values to map. So, we will rewrite our code as follow.
# Create new age categories
new_age_cats = {'0 to 4':'0-14',
'5 to 9':'0-14',
'10 to 14':'0-14',
'15 to 19':'15-29',
'20 to 24':'15-29',
'25 to 29':'15-29',
'30 to 34':'30-49',
'35 to 39':'30-49',
'40 to 44':'30-49',
'45 to 49':'30-49',
'50 to 54':'50-64',
'55 to 59':'50-64',
'60 to 64':'50-64',
'65+':'>= 65'
}
# Rename columns
new_col_names = {'Region Name':'state', 'Variable Name':'gender', 'Age Name':'age_group', '2008':'population'}
modified_master_ss_census_2008_df = (ss_census_2008_df
[cols]
.dropna()
.rename(columns = new_col_names)
.assign(gender = lambda x: x['gender'].str.split('\s+').str[1],
age_group = lambda df:df['age_group'].replace(new_age_cats),
population = lambda x:x['population'].astype('int')
)
.query('gender != "Total" & age_group != "Total"')
.groupby(['state', 'gender', 'age_group']).agg(population = ('population', 'sum'))
.reset_index()
)
# Inspect the first 5 rows
modified_master_ss_census_2008_df.head()
modified_master_ss_census_2008_df.shape
Modify .query line¶
filter_criteria = "gender not in 'Total' and age_group not in 'Total'"
df = (ss_census_2008_df
[cols]
.dropna()
.rename(columns = new_col_names)
.assign(gender = lambda x: x['gender'].str.split('\s+').str[1],
age_group = lambda df:df['age_group'].replace(new_age_cats),
population = lambda x:x['population'].astype('int')
)
.query(filter_criteria)
.groupby(['state', 'gender', 'age_group']).agg(population = ('population', 'sum'))
.reset_index()
)
# Inspect the first 5 rows
df.shape
Convert the code into a function¶
Now let's convert our code into a function that we can use every time we use this dataset.
def transform_ss_census_2008_data(df):
'''A function for transforming South Sudan 2008 Census Data.
'''
# Select columns of interest
cols = ['Region Name', 'Variable Name', 'Age Name', '2008']
# Return a clean DataFrame
return(df
[cols]
.dropna()
.rename(columns = new_col_names)
.assign(gender = lambda x: x['gender'].str.split('\s+').str[1],
age_group = lambda df:df['age_group'].replace(new_age_cats),
population = lambda x:x['population'].astype('int')
)
.query('gender != "Total" & age_group != "Total"')
.groupby(['state', 'gender', 'age_group']).agg(population = ('population', 'sum'))
.reset_index()
)
Test your new function¶
data_raw = pd.read_csv('00_Data/ss_2008_census_data_raw.csv')
df = transform_ss_census_2008_data(data_raw)
df.head()
df.tail()
Compute the states' totals¶
state_df = (df
.groupby(['state'])
.sum()
.sort_values('population', ascending = False)
)
state_df
Compute states' totals by gender¶
state_by_gender_df = (df
.groupby(['state', 'gender'])
.sum()
.sort_values(['population', 'state'], ascending = False)
)
state_by_gender_df
state_by_gender_n_age_category = (df
.groupby(['state', 'gender', 'age_group'])
.sum()
.sort_values('population', ascending = False)
)
state_by_gender_n_age_category.head()
state_by_gender_n_age_category.tail()
Closing Remarks¶
This tutorial demonstrates how to implement method chaining in pandas. Pandas method chaining is similar to the dplyr pipe (%>%); it allows us to write concise code and avoid getting pandas warnings such as SettingWithCopyWarning; the code is easy to read. I hope you will find this tutorial beneficial.
Thank you for taking the time to read it!
References¶
Harrison, M., & Petrou, T. (Author) (2020). Pandas 1.x Cookbook: Practical recipes for scientific computing, time series analysis, and exploratory data analysis using Python, 2nd Edition.
Tag:method chaining, pandas
You may also like
Marrying-R-with-Python-1
Data Wrangling with PySpark
