Data analysis with pandas
Load a table, clean its missing values, summarize it by group, and communicate the result without misleading a reader.
By the end, you’ll be able to…
- Read and inspect a DataFrame before changing it.
- Clean missing values, duplicates, and column types deliberately.
- Use groupby, aggregation, and a labeled chart to explain a result.
01A DataFrame is a table with an index
pandas has two central objects. A Series is a one-dimensional labeled array; a DataFrame is a table of aligned Series with row and column labels. read_csv is the usual entry point for a file. Inspect shape, columns, dtypes, and a few rows before assuming anything about the data. The index identifies rows but is not automatically a meaningful business key.
Keep the first inspection small. head() shows the first rows, info() shows column types and non-null counts, and describe() summarizes numeric columns. These are not decoration; they reveal whether a value column arrived as text, whether an identifier lost leading zeros, or whether a date needs parsing. Fix those boundaries before calculating totals.
import pandas as pdrecords = [ {"region": "North", "sales": 120, "month": "2026-01"}, {"region": "South", "sales": 80, "month": "2026-01"}, {"region": "North", "sales": 100, "month": "2026-02"}, {"region": "South", "sales": 140, "month": "2026-02"},]df = pd.DataFrame(records)print(df.to_string(index=False))print(df.shape)print(df.dtypes)region sales month North 120 2026-01 South 80 2026-01 North 100 2026-02 South 140 2026-02 (4, 3) region object sales int64 month object dtype: object
02Clean before you calculate
Missing data is not one problem. A missing sale may mean zero, an unknown value, or a failed import; filling it with zero without a decision can change the answer. pd.to_numeric(errors='coerce') turns unparseable text into NaN so it becomes visible. Then choose whether to drop, fill, or investigate those rows.
drop_duplicates removes repeated records, but only after you decide which columns define identity. A duplicate order line may be legitimate. astype and parse_dates make the type match the question. Record these choices in the analysis so another person can reproduce the result. Cleaning is part of the argument, not a hidden preprocessing step.
import pandas as pdraw = [ {"region": "North", "sales": "120"}, {"region": "South", "sales": "n/a"}, {"region": "North", "sales": "120"}, {"region": "East", "sales": "95"},]df = pd.DataFrame(raw)df["sales"] = pd.to_numeric(df["sales"], errors="coerce")clean = df.dropna(subset=["sales"]).drop_duplicates()print(clean.to_string(index=False))region sales North 120.0 East 95.0
03Group, join, and summarize
groupby splits rows into groups, an aggregation turns each group into one value, and a merge combines tables that share a key. Always make the grouping keys and the aggregation explicit. `as_index=False` keeps the result as ordinary columns when you want to sort or export it. A summary should answer a question, not merely display every operation the library can perform.
Before joining, check that the key is unique on the side you expect. A duplicate key can multiply rows and inflate totals. After joining, compare row counts and missing matches. Sort the final table by the measure that matters to the reader, and round only at the presentation boundary so the calculation keeps its precision.
import pandas as pddf = pd.DataFrame({ "region": ["North", "South", "North", "East"], "sales": [120, 80, 100, 95],})summary = (df.groupby("region", as_index=False)["sales"] .sum() .sort_values("sales", ascending=False))print(summary.to_string(index=False))region sales North 220 East 95 South 80
04Tell the truth with a chart
A chart is an argument. Start the value axis at zero for bar charts, label the axes, and put the unit in the label when it is not obvious. A line chart implies continuity over time, so use one only when the x-axis is ordered. A pie chart with many slices is usually harder to read than a sorted bar chart.
Matplotlib gives you explicit control over titles, labels, and output. Save the figure at a useful resolution and keep the script that produced it. The final report should include the data source, the cleaning decisions, the calculation, and the chart. A chart that cannot be regenerated from the repository is a screenshot, not an analysis.
import matplotlib.pyplot as pltimport pandas as pddf = pd.DataFrame({"region": ["North", "East", "South"], "sales": [220, 95, 80]})ax = df.plot.bar(x="region", y="sales", legend=False)ax.set_title("Sales by region")ax.set_xlabel("Region")ax.set_ylabel("Sales")plt.tight_layout()plt.savefig("sales.png", dpi=150)print("saved sales.png")saved sales.png
The ideas worth keeping.
- Inspect shape, columns, dtypes, and missing values before calculating.
- Make cleaning decisions explicit and reproducible; do not silently replace missing values.
- A useful analysis ends with a labeled, reproducible summary rather than an unexplained table or chart.
Make it yours.
Type and test your code right here in the interactive workspace. Run your solution, check the output, and compare with the worked guide when ready.
Inspect and total
Create a DataFrame with region and sales columns, print its shape, and calculate the total sales. Use a reproducible column operation rather than adding the numbers by hand.
Explore a worked solution
import pandas as pddf = pd.DataFrame({ "region": ["North", "South", "North", "South"], "sales": [120, 80, 100, 140],})print(df.shape)print(df["sales"].sum())The column operation returns a Series; sum reduces it to one value. Keeping the data in a DataFrame makes the same code work when the records come from a file.
Clean a missing score
Given a table of team and score values with one missing score, drop the missing row, group by team, and print the mean score rounded to one decimal place.
Explore a worked solution
import pandas as pdraw = pd.DataFrame({ "team": ["A", "A", "B", "B"], "score": [10, None, 8, 12],})clean = raw.dropna(subset=["score"])print(clean.groupby("team")["score"].mean().round(1).to_dict())The missing value is dropped before grouping because there is no justified replacement. The result shows one average per team.
What stayed with you?
Check the ideas while they’re fresh. Your answers stay in this page and feedback appears as you go.
Inspect and total
Create a DataFrame with region and sales columns, print its shape, and calculate the total sales. Use a reproducible column operation rather than adding the numbers by hand.
df.shape df["sales"].sum()
Keep exploring.
The official references behind this chapter, when you want a second angle or the full details.
One chapter closer.
Reading is a start. Trying it yourself makes the connection.