Iterators & Generators

Jr Codex Python Notes

Level: Advanced Prerequisites: Module 4: Object-Oriented Programming Time to complete: ~30 minutes

Iterables vs Iterators
The Iterator Protocol: __iter__ and __next__
Building a Custom Iterator
Generator Functions: yield
Why Generators? Lazy Evaluation
yield vs return
Generator Expressions Revisited
Infinite Generators
Summary & Next Steps

1. Iterables vs Iterators

You've used for item in collection: since Module 1 without examining what makes that work. Two related but distinct concepts are involved:

Iterable vs Iterator
─────────────────────────────────────────
  Iterable: anything you CAN loop over (has __iter__)
            e.g. list, str, dict, set, range, file objects

  Iterator: the ACTIVE object doing the looping (has __next__)
            produced by calling iter() on an iterable
─────────────────────────────────────────

numbers = [1, 2, 3]              # numbers is an ITERABLE
iterator = iter(numbers)             # calling iter() produces an ITERATOR
 
print(next(iterator))      # 1
print(next(iterator))       # 2
print(next(iterator))        # 3
print(next(iterator))          # StopIteration! — no more items

A for loop is really just this process, automated:

# What "for x in numbers:" actually does under the hood
iterator = iter(numbers)
while True:
    try:
        item = next(iterator)
    except StopIteration:
        break
    print(item)

2. The Iterator Protocol: `iter` and `next`

Any object implementing both __iter__ (returns an iterator) and __next__ (returns the next value, or raises StopIteration) qualifies as an iterator — this connects directly to Module 4's dunder methods chapter.

class CountUp:
    """A custom iterator counting from `start` to `end`."""
    def __init__(self, start, end):
        self.current = start
        self.end = end
 
    def __iter__(self):
        return self          # an iterator returns ITSELF from __iter__
 
    def __next__(self):
        if self.current > self.end:
            raise StopIteration          # signals "no more items"
        value = self.current
        self.current += 1
        return value
 
counter = CountUp(1, 5)
for num in counter:              # for...in works automatically now!
    print(num)                     # 1 2 3 4 5
 
# Manual equivalent:
counter2 = CountUp(1, 3)
print(next(counter2))      # 1
print(next(counter2))        # 2
print(next(counter2))          # 3
print(next(counter2))            # StopIteration

3. Building a Custom Iterator

A more realistic example — iterating over a custom collection:

class Playlist:
    def __init__(self, songs):
        self.songs = songs
 
    def __iter__(self):
        return PlaylistIterator(self.songs)
 
class PlaylistIterator:
    def __init__(self, songs):
        self.songs = songs
        self.index = 0
 
    def __iter__(self):
        return self
 
    def __next__(self):
        if self.index >= len(self.songs):
            raise StopIteration
        song = self.songs[self.index]
        self.index += 1
        return song
 
playlist = Playlist(["Song A", "Song B", "Song C"])
for song in playlist:
    print(song)
 
# Each fresh iteration gets its OWN iterator/index — playlist itself isn't "used up"
for song in playlist:      # works again, from the beginning
    print(song)

Notice Playlist and PlaylistIterator are separate classes — Playlist.__iter__ creates a new iterator each time, so the same playlist can be looped over repeatedly. This boilerplate is exactly what generators (Section 4) eliminate.

4. Generator Functions: `yield`

A generator function looks like a regular function but uses yield instead of (or alongside) return — Python automatically builds the entire iterator protocol for you.

def count_up(start, end):
    current = start
    while current <= end:
        yield current           # pauses here, returns `current`, remembers where it left off
        current += 1
 
counter = count_up(1, 5)
print(type(counter))        # <class 'generator'>
 
for num in counter:
    print(num)                 # 1 2 3 4 5 — identical result to the CountUp class, far less code

What Happens When You Call count_up(1, 5)
─────────────────────────────────────────
  1. The function body does NOT run yet — calling a generator function
     just creates a generator OBJECT.
  2. Each call to next() runs the body UNTIL the next `yield`,
     then PAUSES — all local state (current, etc.) is preserved.
  3. Calling next() again RESUMES exactly where it paused.
  4. When the function finally returns (falls off the end),
     Python raises StopIteration automatically.
─────────────────────────────────────────

def simple_gen():
    print("First")
    yield 1
    print("Second")
    yield 2
    print("Third")
    yield 3
 
gen = simple_gen()
print(next(gen))      # First   → 1
print(next(gen))        # Second → 2
print(next(gen))          # Third → 3

5. Why Generators? Lazy Evaluation

Generators produce values one at a time, on demand — they never hold the entire sequence in memory at once. This matters enormously once data gets large (a theme that returns constantly in the ML/data modules later in this curriculum).

# Eager — builds a list of ALL 10 million items in memory immediately
def get_squares_list(n):
    return [i ** 2 for i in range(n)]
 
# Lazy — produces ONE value at a time, using almost no memory regardless of n
def get_squares_gen(n):
    for i in range(n):
        yield i ** 2
 
# Both produce the "same" sequence of values, but very differently:
total = sum(get_squares_gen(10_000_000))    # fine — never holds 10M numbers at once
# total = sum(get_squares_list(10_000_000))   # works, but allocates a huge list first

Memory Footprint
─────────────────────────────────────────
  List version:      [4, 9, 16, 25, ...]  ← ALL items exist in memory simultaneously
  Generator version:  4 → (discarded) → 9 → (discarded) → 16 → ...
                       ← only ONE value exists in memory at any moment
─────────────────────────────────────────

This is exactly the same lazy idea introduced briefly with range() (Module 1) and generator expressions (Module 2, Chapter 5) — a generator function is the general-purpose version of that pattern, for logic too complex for a single expression.

6. `yield` vs `return`

def with_return(n):
    result = []
    for i in range(n):
        result.append(i ** 2)
    return result            # runs ALL the way through, then hands back ONE list
 
def with_yield(n):
    for i in range(n):
        yield i ** 2          # PAUSES and hands back ONE value at a time, repeatedly
 
# with_return(5) → a list: [0, 1, 4, 9, 16]
# with_yield(5)  → a generator object — values are pulled one at a time

	`return`	`yield`
Function stops?	Yes, completely, after `return`	No — pauses, can resume on the next `next()` call
What you get	One final value (or collection)	A generator object producing values lazily
Called again	Starts fresh from the top	Resumes exactly where it paused

A single function can only be one or the other overall — if a function contains any yield, Python treats the whole function as a generator function, even if it also has a return (which then just means "stop iterating here," not "return this value" the normal way):

def limited_counter(n):
    for i in range(n):
        if i == 3:
            return              # stops the generator early — NOT a returned value
        yield i
 
for num in limited_counter(10):
    print(num)     # 0 1 2 — stops early because of the bare `return`

7. Generator Expressions Revisited

Module 2, Chapter 5 introduced the syntax (expr for x in iterable) briefly. Now that you understand generator functions fully, a generator expression is just their compact, single-expression cousin:

squares_gen_expr = (n ** 2 for n in range(5))
 
def squares_gen_func():
    for n in range(5):
        yield n ** 2
 
# Both produce IDENTICAL generator behavior — the expression form is just
# more concise when the logic fits in one line.
print(list(squares_gen_expr))       # [0, 1, 4, 9, 16]
print(list(squares_gen_func()))       # [0, 1, 4, 9, 16]

Rule of thumb: use a generator expression for simple, one-line transforms; switch to a full def-based generator function once you need multiple statements, conditionals across several lines, or state that's clearer to express step-by-step.

8. Infinite Generators

Because generators are lazy, they can represent infinite sequences — something a list could never do (it would try to allocate infinite memory and simply never finish).

def infinite_counter(start=0):
    n = start
    while True:              # no stopping condition — runs forever, in theory
        yield n
        n += 1
 
counter = infinite_counter()
print(next(counter))      # 0
print(next(counter))        # 1
print(next(counter))          # 2
# ... this could continue forever; nothing crashes because only ONE value
#     is ever computed and held at a time
 
# Practical use: combine with itertools.islice (Chapter 4) to take just a few
from itertools import islice
first_five = list(islice(infinite_counter(100), 5))
print(first_five)      # [100, 101, 102, 103, 104]

9. Summary & Next Steps

Key Takeaways

An iterable can produce an iterator (via iter()); an iterator produces values one at a time (via next()) and raises StopIteration when exhausted.
Implementing __iter__/__next__ manually works but is verbose — a generator function (using yield) gets Python to build the entire iterator protocol for you automatically.
Generators are lazy — they compute and hold only one value at a time, making them ideal for large or infinite sequences where a list would be wasteful or impossible.
A function containing any yield becomes a generator function entirely; return inside one just stops the generator early rather than returning a normal value.

Concept Check

What's the difference between an iterable and an iterator?
Why does calling a generator function not run any of its code immediately?
Why can a generator represent an infinite sequence, but a list comprehension cannot?

Next Chapter

→ Chapter 2: Decorators

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index