Pandas Basics

A self-help guide to understanding, practising, and revising pandas.

Python for data work · Learning guide 01 · Updated 18 October 2026

A table becomes useful when I can answer a question with it: which rows match a condition, what needs cleaning, or how a group compares with another. pandas gives these recurring tasks a small, consistent vocabulary.

The guide follows a practical sequence: understand the table, select data, make changes, clean carefully, then summarize. Read it once to build the ideas; return to the parameter tables and worked exercises when you need to revise or solve a similar problem.

Before you begin: basic Python lists, dictionaries, functions, and comparisons. No NumPy, SQL, or machine learning background is required. Read sections 1–5 first; return to the exercises and reference tables over several short sessions.

Before choosing a method, ask what one row represents. That tells you whether an operation should retain individual records, combine them, or turn them into summaries.

What is pandas?

pandas is an open-source Python library for working with labelled data, particularly tables. It provides tools to read files, inspect columns, select records, handle missing values, calculate summaries, and combine datasets.

Its two central objects are a DataFrame, a two-dimensional table, and a Series, a one-dimensional collection of values with an index. A DataFrame can hold different types in different columns: names, integer counts, decimal measurements, dates, or categories.

A useful comparison:

These tools often work together. You might query PostgreSQL, inspect the result in pandas, and pass numeric features to a model.

pandas usually works with data in memory. A CSV's file size is not a reliable estimate of the memory its parsed table will consume. If a dataset is large, read fewer columns, process suitable operations in chunks, or push work into a database. The official beginner tutorials are a useful companion to this guide.

Set up your environment

Use a project-specific virtual environment so experiments do not interfere with other projects. A compatible Python installation is required; if Python is missing, begin with the official Python downloads.

macOS or Linux

python3 -m venv .venv
source .venv/bin/activate
python -m pip install pandas

Windows PowerShell

py -m venv .venv
.\.venv\Scripts\python.exe -m pip install pandas

On Windows you can use the environment's interpreter directly, without changing PowerShell's script policy:

.\.venv\Scripts\python.exe your_script.py

Check the interpreter you intend to use:

python -c "import pandas as pd; print(pd.__version__)"

For Windows without an activated environment, replace python with .\.venv\Scripts\python.exe. In a notebook, select the kernel belonging to that same environment.

Import pandas using its conventional short name:

import pandas as pd

Version note. The accompanying examples were checked with pandas 2.2.3; assignment examples were also checked with its Copy-on-Write option enabled. Current pandas 3.x uses Copy-on-Write by default. The patterns here avoid chained assignment, but exact inferred string dtypes and printed formatting can vary between versions. See the installation guide and Copy-on-Write guide.

Read a DataFrame

One small table to learn with

Imagine Harry, Ron, Hermione, and Neville at a coding workshop. The scores and ages in these examples are invented practice data; the characters simply make the rows easier to remember.

Use this setup for the examples in sections 3–6. Each later example starts from this table unless it defines another input. Examples that modify a table make a copy so you can try them independently.

students = pd.DataFrame({
    "student_id": [101, 102, 103, 104],
    "name": ["Harry", "Ron", "Hermione", "Neville"],
    "track": ["SDE", "ML", "SDE", "ML"],
    "score": [82, 65, 91, 74],
}, index=[10, 20, 30, 40])
print(students)

    student_id      name track  score
10         101     Harry   SDE     82
20         102       Ron    ML     65
30         103  Hermione   SDE     91
40         104   Neville    ML     74

The leftmost labels, 10, 20, 30, 40, are the index. They are not a fifth data column. Here, a row represents one student; student_id is an ordinary column containing the application's identifier.

The vocabulary

Filtering removes rows, not their labels. If label 20 disappears, the remaining labels are still 10, 30, 40; the third row is reached by position 2 or label 40. Resetting the index is a separate choice.

The index is not automatically a database primary key. If uniqueness matters, check it with students.index.is_unique, or validate an explicit ID column.

Inspect before changing anything

print(students.shape)           # (4, 4): rows, columns
print(students.size)            # 16: total cells, including missing cells
print(len(students))            # 4: rows
print(students.columns.tolist())
print(students.head(2))
print(students.dtypes)
students.info()                 # prints structure and non-missing counts
print(students.describe())      # numeric summary by default

info() prints its report and returns None; use students.info(), rather than print(students.info()). describe() does not establish that the data is valid. Look for impossible values, duplicate identifiers, and unexpected units yourself.

Read and write files

from io import StringIO

csv_text = "student_id,name,score\n101,Harry,82\n102,Ron,65\n"
loaded = pd.read_csv(StringIO(csv_text))
print(loaded.shape)             # (2, 3)

With a real file, replace StringIO(csv_text) with a path. These lines expect an input file and write an output file:

loaded = pd.read_csv("students.csv", usecols=["student_id", "name", "score"])
loaded.to_csv("students_clean.csv", index=False)

index=False leaves pandas' row labels out of the CSV. Include an index deliberately if it carries useful information. For identifiers such as "0012", specify a string dtype while reading so leading zeros survive.

Useful CSV arguments when moving beyond the toy input:

These options describe an input contract; they do not establish that the values are valid. See read_csv parameters.

pandas also supports JSON, Excel, Parquet, and SQL inputs. Some formats need optional dependencies; CSV is enough to begin. See the reading and writing tutorial.

Select and transform

Labels and positions answer different questions

Use .loc[row_selector, column_selector] when labels or conditions express the task. Use .iloc when positions express it. For a single cell, .at[label, column] and .iat[row_position, column_position] are also available. Details: indexing and selection.

A filter is a sequence of decisions

mask = students["score"] >= 80
selected = students.loc[mask, ["name", "score"]]
print(selected)

        name  score
10     Harry     82
30  Hermione     91

Combine conditions with & for AND, | for OR, and ~ for NOT. Parenthesize each comparison:

selected = students.loc[
    (students["track"] == "SDE") & (students["score"] >= 85),
    ["name", "score"],
]
print(selected["name"].tolist())   # ['Hermione']

Python's and and or expect a single truth decision; a Series contains many. To test membership, use .isin(["SDE", "ML"]).

Work on whole columns

scored = students.copy()
scored["score_fraction"] = scored["score"] / 100
scored["passed"] = scored["score"] >= 70
print(scored["passed"].tolist())   # [True, False, True, True]

This style expresses a column operation directly instead of writing an explicit row loop. Prefer built-in arithmetic, comparisons, .str operations, and aggregations before reaching for .apply().

Vectorized syntax is not a universal speed guarantee; memory, dtype, and the particular operation matter. For custom logic, .apply() can be useful, but axis=1 often invokes Python once per row.

Make assignments explicit

updated = students.copy()
updated.loc[updated["student_id"] == 102, "score"] = 70

Avoid writing through an intermediate selection such as updated[updated["student_id"] == 102]["score"] = 70. Under Copy-on-Write it does not update the original table. Select the intended rows and column in a single .loc assignment.

Sort, rename, and chain

ranked = (
    students
    .sort_values(["score", "student_id"], ascending=[False, True])
    .rename(columns={"score": "marks"})
    .reset_index(drop=True)
)
print(ranked["name"].tolist())   # ['Hermione', 'Harry', 'Neville', 'Ron']

A secondary sort key makes ties predictable. reset_index(drop=True) gives the result consecutive row labels; it does not sort anything. Chaining reads as a sequence of transformations, but naming intermediate results is often easier while debugging.

Most transformations return an object you must use or assign. inplace=True is not a general promise of lower memory use or better performance.

Clean without losing meaning

Real tables contain inconsistent text, missing entries, duplicate records, and numbers stored as strings. Cleaning requires decisions about what those entries mean.

Missing is different from zero

raw = pd.DataFrame({
    "name": [" Harry ", "Ron", None],
    "score": ["82", "absent", "91"],
})
raw["name"] = raw["name"].astype("string").str.strip()
raw["score"] = pd.to_numeric(raw["score"], errors="coerce").astype("Int64")
print(raw["score"].isna().tolist())    # [False, True, False]
print(raw["score"].mean())            # 86.5: missing values are skipped

The two known scores, 82 and 91, average to 86.5. Filling the missing score with zero changes that average to 57.67. The operation succeeds either way; the meaning of the result changes.

errors="coerce" converts an unparseable number into a missing entry. That is convenient, but it can hide bad input: count or preserve the original invalid values if quality matters.

Int64, with a capital I, is pandas' nullable integer dtype; it can hold integers and pd.NA. Plain int64 cannot directly represent missing integers. Other dtypes can use NaN, NaT, or None-like values. Use .isna() and .notna() to detect missing entries rather than equality comparisons. The missing-data guide explains the representations.

Choose an action based on the task:

with_names = raw.dropna(subset=["name"])
labelled = raw.assign(name=raw["name"].fillna("Unknown"))

Dropping a missing name may be appropriate for a name directory. Replacing an absent exam score with zero changes its meaning. Filling a feature with the mean might be reasonable in an ML pipeline, but the fill value must be learned from the training split to avoid leakage.

For a group, .size() counts rows while .count() counts non-missing values in the selected column. Report both when missingness matters.

Duplicate according to which key?

contacts = pd.DataFrame({
    "name": ["Harry", "Ron", "Harry revised"],
    "email": ["harry@example.test", "ron@example.test", "harry@example.test"],
})
unique = contacts.drop_duplicates(subset=["email"], keep="first")
print(len(unique))   # 2

Two rows can be duplicates by email even when their names differ. Define the business key first. If you intend to retain the latest record, sort by a trustworthy timestamp before using keep="last".

Text, dates, and types

astype() converts to a type; it does not validate your domain. A negative age may be a perfectly valid integer and still be invalid data. An object dtype can also hold mixed Python values; it does not mean “definitely a clean string column.”

Summarize, combine, and reshape

Group by a question

“What is the average score in each track?” changes our unit of analysis: one student per row becomes one track per row.

summary = (
    students.groupby("track", as_index=False)
    .agg(students=("student_id", "size"), mean_score=("score", "mean"))
)
print(summary)

  track  students  mean_score
0    ML         2        69.5
1   SDE         2        86.5

The field named students counts rows using size; it does not count distinct IDs. Use nunique when the question is about distinct students. See groupby parameters.

An aggregation reduces a group to a summary. If you need the group mean beside every original row, use students.groupby("track")["score"].transform("mean"). By default, missing grouping keys are excluded; specify dropna=False if they should form a group. See the grouping tutorial.

Concatenate batches; merge related tables

concat stacks compatible batches. merge matches rows using keys.

first_batch = students.iloc[:2]
second_batch = students.iloc[2:]
combined = pd.concat([first_batch, second_batch], ignore_index=True)

Concatenation does not automatically remove duplicates. Collect batches and concatenate once instead of repeatedly growing a DataFrame inside a loop.

For a join, attach track information to each student:

tracks = pd.DataFrame({
    "track": ["SDE", "ML"],
    "focus": ["Systems", "Models"],
})
enriched = students.merge(tracks, on="track", how="left", validate="many_to_one")
print(enriched["focus"].tolist())   # ['Systems', 'Models', 'Systems', 'Models']

how="left" retains every student. validate="many_to_one" checks that the right-side key is unique, helping catch joins that unexpectedly multiply rows. An inner join retains matched keys; an outer join retains keys from both sides. With duplicate keys on both sides, matches can form a Cartesian expansion within each key.

See merge parameters when specifying a join contract.

Unlike typical SQL joins, pandas can match missing join keys with other missing keys. Decide how to handle them before merging. Details: merge and concatenate guide.

Wide and long are two representations

scores = pd.DataFrame({
    "name": ["Harry", "Harry", "Ron", "Ron"],
    "topic": ["SQL", "OS", "SQL", "OS"],
    "score": [80, 90, 70, 60],
})
wide = scores.pivot(index="name", columns="topic", values="score")
long_again = wide.reset_index().melt(
    id_vars="name", var_name="topic", value_name="score"
)

In the wide table, Harry has OS 90 and SQL 80; Ron has OS 60 and SQL 70. melt returns one name/topic observation per row; the row ordering may differ from the original.

pivot requires a unique value for each index/column pair. If the same student has two scores for SQL, decide what the repetitions mean. To intentionally average them, use pivot_table(..., aggfunc="mean"). Do not aggregate accidentally just to silence a duplicate-key error. See the reshaping tutorial.

Labels also control arithmetic

left = pd.Series([10, 20], index=["a", "b"])
right = pd.Series([1, 2], index=["b", "a"])
print((left + right).to_dict())   # {'a': 12, 'b': 21}

pandas aligns these values by label, not by visual position. Unexpected missing values after arithmetic may mean your labels do not match. Positional operations are possible, but first confirm that ignoring labels is actually correct.

Worked example: summarize benchmark runs

Suppose I have saved timings from two implementations. I want a report that shows successful latencies and keeps failed attempts visible.

The following timings are synthetic. They are used only to illustrate the reporting operations.

runs = pd.DataFrame({
    "engine": ["baseline", "baseline", "candidate", "candidate", "candidate", "candidate"],
    "case": ["scan", "scan", "scan", "scan", "scan", "scan"],
    "latency_ms": [10, 14, 8, 12, None, 100],
    "status": ["ok", "ok", "ok", "ok", "timeout", "ok"],
})
runs["latency_ms"] = pd.to_numeric(runs["latency_ms"], errors="coerce")
runs["valid"] = (runs["status"] == "ok") & runs["latency_ms"].notna()

counts = (
    runs.groupby(["engine", "case"], as_index=False)
    .agg(attempts=("status", "size"), valid=("valid", "sum"))
)
counts["failed_or_missing"] = counts["attempts"] - counts["valid"]

latencies = (
    runs.loc[runs["valid"]]
    .groupby(["engine", "case"], as_index=False)
    .agg(mean_ms=("latency_ms", "mean"),
         p95_ms=("latency_ms", lambda s: s.quantile(0.95)))
)
report = counts.merge(latencies, on=["engine", "case"], how="left", validate="one_to_one")
print(report.round(2).to_string(index=False))

The candidate has three usable timings out of four attempts. Its mean of 40 ms describes those three successes; it says nothing about how long the failed request waited. Keep the failure count beside the latency summary.

The failed attempt stays visible. Successful latencies are summarized separately. With only two or three observations, the interpolated p95 is a calculation, not a stable estimate of tail latency under load.

With real data, also retain workload size, seed, concurrency, cache conditions, source revision, environment, and timeout policy. Compare like-for-like workloads. For model evaluation, retain the evaluation unit and split identifiers; a tidy table cannot repair data leakage.

Try changing it: add a baseline timeout. The attempt and failure counts should increase while its successful-latency mean remains 12.0.

Fifteen exercises, with hidden solutions

These fifteen independent exercises move from creating a table to building a short analysis pipeline. Each worked solution explains the approach, the useful arguments, the result, and a mistake to watch for. The parameter tables are selective quick references, rather than complete method signatures.

Run the shared setup below once. Each exercise is independent: start from these original tables rather than carrying mutations from a previous answer. Try each task before opening its solution.

import pandas as pd

people = pd.DataFrame({
    "id": [11, 12, 13, 14],
    "name": ["Harry", "Ron", "Hermione", "Neville"],
    "age": [24, 25, 23, 26],
    "score": [82, 65, 91, 74],
})
dirty = pd.DataFrame({
    "id": [11, 12, 13, 14],
    "name": ["Harry", "Ron", None, "Neville"],
    "email": ["a@example.test", "b@example.test", "a@example.test", "d@example.test"],
    "score": [82.0, None, 91.0, 74.0],
})

01 · Build a table

Create pairs from [[11, 82], [12, 65], [13, 91]], with columns id and score. Expected shape: (3, 2).

pairs = pd.DataFrame([[11, 82], [12, 65], [13, 91]], columns=["id", "score"])

02 · Measure the table

Return the row count and column count of people as a list. Expected: [4, 4].

answer = list(people.shape)

03 · Preview records

Return the first three rows of people. Their IDs should be [11, 12, 13].

answer = people.head(3)

04 · Filter and project

Select the name and age columns for ID 13. Expected values: Hermione, 23, in a one-row DataFrame.

answer = people.loc[people["id"] == 13, ["name", "age"]]

05 · Derive a column

Create a new table with double_score, twice each score. Expected: [164, 130, 182, 148].

answer = people.assign(double_score=people["score"] * 2)

06 · Remove repeated keys

From dirty, retain the first row for each email. Expected retained IDs: [11, 12, 14].

answer = dirty.drop_duplicates(subset=["email"], keep="first")

07 · Require one field

Remove rows with a missing name, retaining rows even if their score is missing. Expected IDs: [11, 12, 14].

answer = dirty.dropna(subset=["name"])

08 · Modify values

Create a copy of people with each score increased by 5. Expected: [87, 70, 96, 79].

answer = people.copy()
answer["score"] = answer["score"] + 5

09 · Rename fields

Rename id to student_id and score to marks. Expected columns: student_id, name, age, marks.

answer = people.rename(columns={"id": "student_id", "score": "marks"})

10 · Convert a dtype

A table typed has scores "82" and "65", both strings. Convert them to nullable integers. Expected values: [82, 65]; dtype: Int64.

typed = pd.DataFrame({"score": ["82", "65"]})
answer = typed.astype({"score": "Int64"})

11 · Fill a missing value

For this exercise only, the reporting rule is to display missing scores as zero. Apply that rule to a new table based on dirty. Expected scores: [82.0, 0.0, 91.0, 74.0].

answer = dirty.assign(score=dirty["score"].fillna(0))

12 · Stack two batches

Concatenate people.iloc[:2] and people.iloc[2:], giving the result a fresh consecutive index. Expected IDs: [11, 12, 13, 14].

answer = pd.concat([people.iloc[:2], people.iloc[2:]], ignore_index=True)

13 · Turn categories into columns

Using the scores table below, create one row per name and one column per topic. Expected: Harry has OS 90 and SQL 80; Ron has OS 60 and SQL 70.

scores = pd.DataFrame({
    "name": ["Harry", "Harry", "Ron", "Ron"],
    "topic": ["SQL", "OS", "SQL", "OS"],
    "score": [80, 90, 70, 60],
})

answer = scores.pivot(index="name", columns="topic", values="score")

14 · Turn columns into observations

For wide_scores below, produce columns name, topic, score, with four observations.

wide_scores = pd.DataFrame({"name": ["Harry", "Ron"], "OS": [90, 60], "SQL": [80, 70]})

answer = wide_scores.melt(id_vars="name", var_name="topic", value_name="score")

15 · Express a short pipeline

Return only names with scores at least 75, sorted by score descending. Keep the result as a DataFrame. Expected names: Hermione, then Harry.

answer = (
    people.loc[people["score"] >= 75]
    .sort_values("score", ascending=False)
    .loc[:, ["name"]]
    .reset_index(drop=True)
)

Interview questions and quick reference

Explain these without opening an editor

Keep this beside your editor

A final self-check

Given a CSV of requests, can you produce one row per endpoint containing total attempts, failed requests, and the p95 latency of successful requests? Can you explain how missing timings affect the report, why different workloads must stay separate, and what sample size limits your conclusions?

If you can explain those choices as well as write the code, you have moved from memorizing commands to using pandas to answer questions.

Sources and next steps

Use the pandas documentation to check API details. For additional guided practice after the exercises here, the LeetCode study plan is listed separately below.

• pandas: getting started — installation and the beginner tutorial sequence.
• Introductory data structures — Series, DataFrame, labels, and alignment.
• Indexing and selecting — .loc, .iloc, masks, and assignment.
• Working with missing data — missing markers, nullable dtypes, and fill/drop operations.
• GroupBy guide — aggregations and transforms.
• Combining tables — join cardinality, validation, and concatenation.
• Copy-on-Write — modern mutation semantics.
• Scaling to larger datasets — memory and chunking considerations.

Further guided practice

• LeetCode: Introduction to Pandas — an additional guided exercise sequence.
• LeetCode: 30 Days of Pandas — a broader practice plan after the basics.