Generators

Generators are a simple and powerful tool for creating iterators in Python. They allow you to write functions that can pause and resume execution, making them perfect for creating sequences that are computed lazily (on-demand).

What are Generators?

A generator is a special type of iterator that is created using a function with yield statements instead of return. Generators are memory-efficient because they produce values one at a time rather than storing all values in memory.

Generator Functions

Generator functions are defined like regular functions but use yield instead of return. When called, they return a generator object (an iterator).

Basic Generator Function

1
def count_up(limit):
2
    current = 0
3
    while current < limit:
4
        yield current
5
        current += 1
6

7
# Usage
8
for num in count_up(5):
9
    print(num)  # 0, 1, 2, 3, 4
10

11
# Or manually
12
gen = count_up(3)
13
print(next(gen))  # 0
14
print(next(gen))  # 1
15
print(next(gen))  # 2

How `yield` Works

When Python encounters a yield statement:

It returns the value to the caller
It pauses the function’s execution
It saves the function’s state (local variables, execution position)
When next() is called again, execution resumes from where it left off

1
def simple_generator():
2
    print("Starting generator")
3
    yield 1
4
    print("After first yield")
5
    yield 2
6
    print("After second yield")
7
    yield 3
8
    print("Generator finished")
9

10
gen = simple_generator()
11
print("Created generator")
12

13
print(next(gen))  # Starting generator, then 1
14
print(next(gen))  # After first yield, then 2
15
print(next(gen))  # After second yield, then 3
16
# print(next(gen))  # Generator finished, then StopIteration

Generator vs Regular Function

Regular Function

1
def squares_list(n):
2
    result = []
3
    for i in range(n):
4
        result.append(i ** 2)
5
    return result  # Returns all values at once
6

7
# All values stored in memory
8
squares = squares_list(1000000)  # Uses significant memory

Generator Function

1
def squares_generator(n):
2
    for i in range(n):
3
        yield i ** 2  # Yields one value at a time
4

5
# Values generated on-demand
6
squares = squares_generator(1000000)  # Uses minimal memory
7
for square in squares:
8
    if square > 100:
9
        break  # Only computes what's needed

Common Generator Patterns

Fibonacci Generator

1
def fibonacci(limit=None):
2
    prev, curr = 0, 1
3
    count = 0
4

5
    while limit is None or count < limit:
6
        yield prev
7
        prev, curr = curr, prev + curr
8
        count += 1
9

10
# Generate first 10 Fibonacci numbers
11
for num in fibonacci(10):
12
    print(num)  # 0, 1, 1, 2, 3, 5, 8, 13, 21, 34

Infinite Generator

1
def natural_numbers():
2
    num = 1
3
    while True:
4
        yield num
5
        num += 1
6

7
# Generate natural numbers
8
gen = natural_numbers()
9
for i in range(5):
10
    print(next(gen))  # 1, 2, 3, 4, 5

Generator with Multiple Yields

1
def alternating():
2
    yield "even"
3
    yield "odd"
4
    yield "even"
5
    yield "odd"
6

7
for value in alternating():
8
    print(value)  # even, odd, even, odd

Generator with Conditions

1
def even_numbers(start, end):
2
    for num in range(start, end):
3
        if num % 2 == 0:
4
            yield num
5

6
for num in even_numbers(1, 10):
7
    print(num)  # 2, 4, 6, 8

Generator Expressions

Generator expressions are a concise way to create generators, similar to list comprehensions but with parentheses instead of square brackets.

Syntax

1
# List comprehension (creates a list)
2
squares_list = [x ** 2 for x in range(5)]
3
print(squares_list)  # [0, 1, 4, 9, 16]
4

5
# Generator expression (creates a generator)
6
squares_gen = (x ** 2 for x in range(5))
7
print(squares_gen)  # <generator object <genexpr> at 0x...>
8
print(list(squares_gen))  # [0, 1, 4, 9, 16]

Generator Expression Examples

1
# Simple generator expression
2
numbers = (x for x in range(10))
3
print(list(numbers))  # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
4

5
# With condition
6
evens = (x for x in range(10) if x % 2 == 0)
7
print(list(evens))  # [0, 2, 4, 6, 8]
8

9
# With transformation
10
squares = (x ** 2 for x in range(5))
11
print(list(squares))  # [0, 1, 4, 9, 16]
12

13
# Nested
14
pairs = ((x, y) for x in range(3) for y in range(3))
15
print(list(pairs))  # [(0,0), (0,1), (0,2), (1,0), (1,1), (1,2), (2,0), (2,1), (2,2)]

Generator Methods

Generators have several useful methods:

`send()`

Sends a value into the generator and resumes execution:

1
def number_generator():
2
    value = yield
3
    while True:
4
        value = yield value * 2
5

6
gen = number_generator()
7
next(gen)  # Start the generator
8

9
print(gen.send(5))   # 10
10
print(gen.send(10))  # 20
11
print(gen.send(3))   # 6

`throw()`

Raises an exception inside the generator:

1
def simple_gen():
2
    try:
3
        yield 1
4
        yield 2
5
    except ValueError:
6
        yield "Error caught"
7

8
gen = simple_gen()
9
print(next(gen))  # 1
10
print(gen.throw(ValueError))  # Error caught

`close()`

Closes the generator:

1
def count():
2
    try:
3
        num = 0
4
        while True:
5
            yield num
6
            num += 1
7
    except GeneratorExit:
8
        print("Generator closed")
9

10
gen = count()
11
print(next(gen))  # 0
12
print(next(gen))  # 1
13
gen.close()  # Generator closed

Generator Pipelines

Generators can be chained together to create powerful data processing pipelines:

1
def numbers():
2
    for i in range(10):
3
        yield i
4

5
def square(nums):
6
    for num in nums:
7
        yield num ** 2
8

9
def even_only(nums):
10
    for num in nums:
11
        if num % 2 == 0:
12
            yield num
13

14
# Chain generators
15
pipeline = even_only(square(numbers()))
16
print(list(pipeline))  # [0, 4, 16, 36, 64]

Generator Advantages

Memory Efficient: Only produces values when needed
Lazy Evaluation: Computes values on-demand
Clean Code: More readable than custom iterator classes
Infinite Sequences: Can represent infinite sequences
State Preservation: Automatically preserves function state

Memory Comparison

1
import sys
2

3
# List comprehension - stores all values
4
list_comp = [x ** 2 for x in range(1000000)]
5
print(sys.getsizeof(list_comp))  # Large size (millions of bytes)
6

7
# Generator expression - doesn't store values
8
gen_expr = (x ** 2 for x in range(1000000))
9
print(sys.getsizeof(gen_expr))  # Small size (few hundred bytes)

Generator vs Iterator Class

Using Iterator Class

1
class CountDown:
2
    def __init__(self, start):
3
        self.current = start
4

5
    def __iter__(self):
6
        return self
7

8
    def __next__(self):
9
        if self.current <= 0:
10
            raise StopIteration
11
        self.current -= 1
12
        return self.current + 1
13

14
for num in CountDown(5):
15
    print(num)  # 5, 4, 3, 2, 1

Using Generator Function

1
def count_down(start):
2
    current = start
3
    while current > 0:
4
        yield current
5
        current -= 1
6

7
for num in count_down(5):
8
    print(num)  # 5, 4, 3, 2, 1

The generator version is much simpler and more Pythonic!

Practical Examples

Reading Large Files

1
def read_large_file(filename):
2
    with open(filename, 'r') as file:
3
        for line in file:
4
            yield line.strip()
5

6
# Process file line by line without loading entire file
7
for line in read_large_file('large_file.txt'):
8
    process(line)  # Process one line at a time

Generating Combinations

1
def combinations(items, r):
2
    if r == 0:
3
        yield ()
4
    else:
5
        for i in range(len(items)):
6
            for combo in combinations(items[i+1:], r-1):
7
                yield (items[i],) + combo
8

9
items = ['a', 'b', 'c', 'd']
10
for combo in combinations(items, 2):
11
    print(combo)  # ('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'c'), ('b', 'd'), ('c', 'd')

Sliding Window

1
def sliding_window(sequence, window_size):
2
    for i in range(len(sequence) - window_size + 1):
3
        yield sequence[i:i + window_size]
4

5
text = "Python"
6
for window in sliding_window(text, 3):
7
    print(window)  # Pyt, yth, tho, hon

Don’t Reuse Exhausted Generators

1
gen = (x for x in range(3))
2

3
print(list(gen))  # [0, 1, 2]
4
print(list(gen))  # [] (generator is exhausted)
5

6
# Create a new generator instead
7
gen = (x for x in range(3))
8
print(list(gen))  # [0, 1, 2]

Exercises

Exercise 1: Basic Generator Function

Create a generator function called count_up that takes a limit and yields numbers from 0 to limit-1. Read the limit from input and print all generated values.

Answer:

1
def count_up(limit):
2
    current = 0
3
    while current < limit:
4
        yield current
5
        current += 1
6

7
limit = int(input())
8
for num in count_up(limit):
9
    print(num)

Exercise 2: Generator Expression

Create a generator expression that generates squares of numbers from 0 to n-1. Read n from input, convert the generator to a list, and print it.

Answer:

1
n = int(input())
2
squares = (x ** 2 for x in range(n))
3
print(list(squares))

Exercise 3: Even Numbers Generator

Create a generator function called even_numbers that takes start and end values, and yields all even numbers in that range (inclusive start, exclusive end). Read start and end, then print all even numbers.

Answer:

1
def even_numbers(start, end):
2
    for num in range(start, end):
3
        if num % 2 == 0:
4
            yield num
5

6
start = int(input())
7
end = int(input())
8

9
for num in even_numbers(start, end):
10
    print(num)

Exercise 4: Fibonacci Generator

Create a generator function called fibonacci that takes a limit and yields the first limit Fibonacci numbers. Read the limit and print all Fibonacci numbers.

Answer:

1
def fibonacci(limit):
2
    prev, curr = 0, 1
3
    count = 0
4
    while count < limit:
5
        yield prev
6
        prev, curr = curr, prev + curr
7
        count += 1
8

9
limit = int(input())
10
for num in fibonacci(limit):
11
    print(num)

Exercise 5: Generator with Condition

Create a generator expression that generates numbers from 0 to n-1, but only includes numbers divisible by 3. Read n from input, convert to list, and print.

Answer:

1
n = int(input())
2
divisible_by_3 = (x for x in range(n) if x % 3 == 0)
3
print(list(divisible_by_3))

Table of Contents

Table of Contents

Course Progress

Exercise 1: Basic Generator Function

Exercise 2: Generator Expression

Exercise 3: Even Numbers Generator

Exercise 4: Fibonacci Generator

Exercise 5: Generator with Condition

Course Progress