Working with Data at Scale

Jr Codex Data Science Notes

Level: Advanced Prerequisites: Chapter 1: The Data Science Workflow Time to complete: ~25 minutes

Table of Contents

1. When Pandas Stops Being Enough
2. First Steps Before Reaching for "Big Data" Tools
3. Chunking — Processing Data in Pieces
4. More Efficient File Formats
5. The Idea Behind Distributed Computing
6. A First Look at PySpark
7. Choosing the Right Tool for the Data's Size
8. Summary & Next Steps

1. When Pandas Stops Being Enough

Every technique across Modules 1-5 has assumed data comfortably fits in memory as a pandas DataFrame. Real datasets sometimes don't — a pd.read_csv() call that simply hangs, or crashes with a MemoryError, is the first sign this chapter's concerns have become relevant.

Rough Rule of Thumb
─────────────────────────────────────────
  Data fits comfortably in RAM (roughly < 20-30% of available memory)
    → pandas is fine, no changes needed

  Data is APPROACHING or EXCEEDING available RAM
    → this chapter's techniques become necessary
─────────────────────────────────────────

2. First Steps Before Reaching for "Big Data" Tools

Before adopting an entirely new toolchain, several simpler fixes often solve the problem outright:

import pandas as pd
 
# 1. Only load the columns you actually need
df = pd.read_csv("huge_file.csv", usecols=["customer_id", "purchase_amount", "date"])
 
# 2. Use more memory-efficient dtypes
print(df.memory_usage(deep=True))      # see where memory is actually going, column by column
 
df["customer_id"] = df["customer_id"].astype("int32")      # instead of the default int64, if values fit
df["category"] = df["category"].astype("category")           # HUGE savings for low-cardinality string columns
 
# 3. Filter with SQL (Module 2, Ch.2) BEFORE loading into pandas at all
import sqlite3
connection = sqlite3.connect("data.db")
df_filtered = pd.read_sql("SELECT * FROM sales WHERE year = 2024", connection)

# Checking how much a dtype change actually saves
import pandas as pd
 
df = pd.DataFrame({"category": ["A", "B", "A", "C", "B"] * 200_000})
print(f"As object: {df['category'].memory_usage(deep=True) / 1_000_000:.1f} MB")
 
df["category"] = df["category"].astype("category")
print(f"As category: {df['category'].memory_usage(deep=True) / 1_000_000:.1f} MB")
# Often a 5-10x reduction for repetitive string columns

These simple fixes solve the majority of "my data barely doesn't fit" problems — reaching for distributed computing tools (Sections 5-6) is worth doing only once these cheaper options are exhausted.

3. Chunking — Processing Data in Pieces

When a file is too large to load at once, process it in smaller pieces sequentially:

import pandas as pd
 
chunk_size = 100_000
total_revenue = 0
region_totals = {}
 
for chunk in pd.read_csv("huge_sales_file.csv", chunksize=chunk_size):
    total_revenue += chunk["revenue"].sum()
 
    chunk_region_totals = chunk.groupby("region")["revenue"].sum()
    for region, total in chunk_region_totals.items():
        region_totals[region] = region_totals.get(region, 0) + total
 
print(f"Total revenue: {total_revenue:,.0f}")
print(region_totals)

Chunking Trade-off
─────────────────────────────────────────
  Only ONE chunk sits in memory at a time — never the whole file.
  Works well for operations that can be computed INCREMENTALLY
  (sums, counts, group totals) — but operations needing the FULL
  dataset at once (like sorting the entire file, or certain
  merges) are awkward or impossible to chunk this way.
─────────────────────────────────────────

4. More Efficient File Formats

CSV is human-readable but inefficient — column-oriented binary formats are dramatically faster and smaller for large tabular data.

import pandas as pd
 
df = pd.read_csv("large_file.csv")
 
# Parquet — the standard choice for large tabular data in the data science/engineering world
df.to_parquet("large_file.parquet")
df_from_parquet = pd.read_parquet("large_file.parquet")      # much faster to read than CSV

CSV vs Parquet
─────────────────────────────────────────
  CSV:      row-oriented, plain text, human-readable,
              no compression, no stored dtype information
  Parquet:    column-oriented, BINARY, compressed, stores dtypes —
                often 5-10x smaller AND faster to read, especially
                when you only need a FEW columns (it can skip
                reading the rest entirely)
─────────────────────────────────────────

Practical habit: once a dataset is cleaned (Module 2) and you'll be reloading it repeatedly during EDA (Module 3) or modeling, save it as Parquet — the repeated load-time savings add up fast, and dtype information (which CSV loses) is preserved automatically.

5. The Idea Behind Distributed Computing

When data is too large for even chunking on a single machine to be practical, distributed computing splits both the data and the computation across multiple machines, working in parallel.

Single Machine (pandas, even with chunking)      Distributed (Spark, Dask)
─────────────────────────────────────────       ─────────────────────────────────────────
  ONE computer's CPU and RAM handle                Data is SPLIT across MANY machines'
  everything, one chunk at a time                    memory; each processes its own piece
                                                        SIMULTANEOUSLY, results combined after
─────────────────────────────────────────       ─────────────────────────────────────────

The Same Fundamental Idea as Python Notes' Multiprocessing
─────────────────────────────────────────
  Python Notes, Module 5, Ch.6 covered multiprocessing —
  splitting CPU-bound work across multiple CORES on ONE machine.

  Distributed computing is the same core idea, scaled up:
  splitting work across multiple MACHINES instead of just cores.
─────────────────────────────────────────

6. A First Look at PySpark

Apache Spark (used from Python via PySpark) is the industry-standard framework for distributed data processing — its DataFrame API deliberately mirrors pandas closely, so the transition is mostly about new syntax, not new concepts.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, avg
 
spark = SparkSession.builder.appName("DataScienceExample").getOrCreate()
 
df = spark.read.csv("huge_sales_file.csv", header=True, inferSchema=True)
 
# Filtering — conceptually identical to pandas boolean masking (Python Notes' NumPy/Pandas primers)
filtered = df.filter(col("region") == "West")
 
# Grouping and aggregating — the same idea as pandas' groupby (Module 2, Ch.2)
result = df.groupBy("region").agg(avg("revenue").alias("avg_revenue"))
result.show()

pandas ↔ PySpark — Familiar Concepts, New Syntax
─────────────────────────────────────────
  pandas:   df[df["region"] == "West"]
  PySpark:  df.filter(col("region") == "West")

  pandas:   df.groupby("region")["revenue"].mean()
  PySpark:  df.groupBy("region").agg(avg("revenue"))
─────────────────────────────────────────

Key mental shift: Spark operations are lazy — code like .filter() and .groupBy() build up a plan rather than executing immediately; the actual computation only runs when you call an action like .show() or .collect(). This is conceptually related to the lazy generators from the Python Notes (Module 5, Chapter 1) — nothing computes until a result is actually demanded.

This is intentionally a brief orientation, not a full PySpark course — the goal is recognizing when this tool becomes relevant and roughly how its API maps onto the pandas skills already built throughout this curriculum.

7. Choosing the Right Tool for the Data's Size

Decision Guide
─────────────────────────────────────────
  Data fits comfortably in memory
      → pandas (everything in Modules 1-5)

  Data is close to memory limits, but the WORKFLOW is simple
  (sums, filters, incremental aggregation)
      → chunking (Section 3) + efficient dtypes/formats (Section 2, 4)

  Data genuinely exceeds what ONE machine can hold or process
  in reasonable time
      → distributed computing (Spark/Dask, Section 5-6)
─────────────────────────────────────────

Practical guidance: don't reach for Spark by default "because big data sounds impressive" — the setup and operational overhead is real, and most datasets a data scientist encounters day-to-day are handled perfectly well by pandas with the optimizations in Sections 2-4.

8. Summary & Next Steps

Key Takeaways

• Before adopting distributed tools, try cheaper fixes first: loading only needed columns, using efficient dtypes (especially category), and filtering with SQL before loading into pandas.
• Chunking processes a too-large file piece by piece, keeping only one chunk in memory at a time — works well for incremental aggregations, awkward for whole-dataset operations.
• Parquet is a column-oriented binary format that's typically far smaller and faster than CSV, and preserves dtype information CSV loses.
• Distributed computing (Spark/PySpark) splits both data and computation across multiple machines; its DataFrame API deliberately mirrors pandas, and its lazy execution model echoes the generators from the Python Notes.
• Match the tool to the data's actual size — most real-world datasets are handled well by pandas plus the simpler optimizations, without needing distributed infrastructure at all.

Concept Check

1. What are three cheaper alternatives worth trying before reaching for distributed computing?
2. Why is Parquet often preferred over CSV for large tabular datasets?
3. In what sense is Spark's lazy execution model similar to Python generators from the Python Notes?

Next Chapter

→ Chapter 3: Capstone Project

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index