The “Generator” Pattern in C# — Produce Data On-Demand with yield and (A)sync Streams

Most systems don’t need all data right now. They need “the next item,” then the next, and so on. That’s the essence of the generator style in C#: produce a sequence lazily—only when the consumer asks—using yield and the iterator interfaces.

In formal pattern terms this is C#’s take on the Iterator pattern with lazy generation. You’ll see it called generators, iterators, yield iterators, (async) streams. Same spirit: pull-based, on-demand data production.

---

One-minute intuition

Analogy: A conveyor-belt sushi bar. Chefs don’t prepare every possible plate up front. They place the next plate when customers need it. If no one is eating, nothing gets made. That saves time, money, and fridge space.

Generators do the same for your app:

• They don’t precompute an entire result set.

• They emit one item at a time when the caller iterates.

• They pause between items, resuming exactly where they left off.

---

Core building blocks in C#

• IEnumerable<T> / IEnumerator<T> — synchronous pull-based iteration.

• yield return / yield break — compiler builds the state machine for you.

• IAsyncEnumerable<T> / IAsyncEnumerator<T> — asynchronous iteration (C# 8+), perfect for I/O and backpressure (await foreach).

---

Why use generators?

• Memory-efficient: stream millions of rows without loading them all.

• Responsive: start producing output immediately.

• Composable: chain with LINQ operators naturally.

• Cancelable: async streams can accept CancellationToken.

• Infinite or open-ended: natural for “live” sequences (retry loops, tailing logs).

---

1) The simplest generator: numbers on demand

public static IEnumerable<int> RangeInclusive(int start, int end)
{
    for (int i = start; i <= end; i++)
        yield return i;   // produce the next value only when requested
}

Usage:

foreach (var n in RangeInclusive(1, 5))
    Console.Write($”{n} “); // 1 2 3 4 5

No arrays, no lists—just values “from the chef to the belt.”

---

2) Real-world: streaming file lines safely

Analogy: Like a librarian handing you the next page—not the entire archive box.

public static IEnumerable<string> ReadLines(string path)
{
    using var reader = new StreamReader(path);   // disposed when iteration ends
    string? line;
    while ((line = reader.ReadLine()) is not null)
        yield return line;
}

• The file opens when enumeration starts.

• The file closes when enumeration finishes or breaks.

---

3) Infinite / computed sequences (Fibonacci)

Analogy: A barista making the next cup only when you order it.

public static IEnumerable<long> Fibonacci()
{
    long a = 0, b = 1;
    while (true)
    {
        yield return a;
        (a, b) = (b, a + b);
    }
}

// Take first 10
foreach (var f in Fibonacci().Take(10))
    Console.Write($”{f} “);

---

4) Paging APIs: pull a dataset, page by page

public static IEnumerable<Order> FetchOrders(Func<int, IReadOnlyList<Order>> fetchPage)
{
    int page = 1;
    while (true)
    {
        var items = fetchPage(page);
        if (items.Count == 0) yield break;

        foreach (var o in items) yield return o;
        page++;
    }
}

This pattern hides nasty pagination details behind a clean, lazy stream.

---

5) Async generators: IAsyncEnumerable<T> for I/O and backpressure

Analogy: A delivery driver bringing boxes as they’re ready. If the dock is busy, the driver waits (awaits).

public static async IAsyncEnumerable<string> TailLogAsync(
    string path,
    [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken ct = default)
{
    using var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
    using var reader = new StreamReader(fs);

    string? line;
    while (!ct.IsCancellationRequested)
    {
        line = await reader.ReadLineAsync();
        if (line is null) { await Task.Delay(200, ct); continue; } // wait for new lines
        yield return line;
    }
}

Consume:

await foreach (var line in TailLogAsync(”app.log”, ct))
{
    Console.WriteLine(line);
    if (line.Contains(”FATAL”)) break; // stop any time
}

---

6) Real-world: database chunking (async)

public static async IAsyncEnumerable<Customer> StreamCustomersAsync(
    DbConnection conn, int batchSize,
    [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken ct = default)
{
    int offset = 0;
    while (true)
    {
        using var cmd = conn.CreateCommand();
        cmd.CommandText = @”SELECT Id, Name FROM Customers ORDER BY Id
                            OFFSET @o ROWS FETCH NEXT @b ROWS ONLY;”;
        cmd.AddParameter(”@o”, offset);
        cmd.AddParameter(”@b”, batchSize);

        using var rdr = await cmd.ExecuteReaderAsync(ct);
        int count = 0;
        while (await rdr.ReadAsync(ct))
        {
            count++;
            yield return new Customer { Id = rdr.GetInt32(0), Name = rdr.GetString(1) };
        }
        if (count == 0) yield break;
        offset += count;
    }
}

You can now await foreach millions of rows without blowing memory.

---

7) Composing generators with LINQ

Generators shine when piped:

var errors =
    ReadLines(”app.log”)
    .Where(l => l.Contains(”ERROR”))
    .Select(l => ParseError(l)); // nothing executes until you iterate

foreach (var e in errors.Take(100))
    Console.WriteLine(e);

This is deferred execution: work is done only as you consume.

---

8) Subtle rules & pitfalls (worth knowing)

• Deferred execution: If the source changes, re-enumerating recomputes. Cache with .ToList() if you need a snapshot.

• Multiple enumeration: Don’t iterate a generator twice if it’s expensive or has side effects—materialize when necessary.

• yield and try/catch: C# disallows yield return inside try/catch, but allows yield inside try with a finally for cleanup. Example:

public static IEnumerable<string> SafeRead(string path)
{
    var reader = new StreamReader(path);
    try
    {
        string? line;
        while ((line = reader.ReadLine()) is not null)
            yield return line;
    }
    finally
    {
        reader.Dispose();
    }
}

• Resource lifetime: The using/finally pattern ensures files/connections close even if the consumer stops early.

• Threading: Generators resume on the caller’s thread of enumeration; async streams use the async context. Don’t assume parallelism unless you add it.

• Cancellation: Only IAsyncEnumerable<T> supports CancellationToken directly (via the special [EnumeratorCancellation] parameter).

---

9) Where generators shine in everyday systems

• ETL: stream lines/rows from sources, transform on the fly, write out.

• APIs: return paged or chunked responses; or server-push via SignalR after consuming async streams.

• Observability: tail logs, watch metrics feeds.

• Domain logic: rule engines that “yield” matches, schedulers that “yield” due jobs.

• Retry/backoff loops: endlessly “yield next attempt time.”

---

10) A tiny end-to-end example: CSV → filter → upload (mixed sync/async)

// 1) Generate rows lazily from a big CSV
public static IEnumerable<Order> ReadOrders(string path)
{
    using var r = new StreamReader(path);
    _ = r.ReadLine(); // skip header
    string? line;
    while ((line = r.ReadLine()) is not null)
        yield return ParseOrder(line);
}

// 2) Async consumer uploads qualifying orders one by one
public static async Task UploadHighValueAsync(IAsyncCollector<Order> sink, CancellationToken ct)
{
    foreach (var o in ReadOrders(”orders.csv”).Where(o => o.Total > 1000))
    {
        ct.ThrowIfCancellationRequested();
        await sink.AddAsync(o, ct); // network I/O → keep consumer async
    }
}

The CSV never fully loads in memory; work scales with demand.

---

Takeaways

• C# generators (yield) implement a lazy, pull-based data flow.

• Use IEnumerable<T> when computation is CPU-bound and local; use IAsyncEnumerable<T> for I/O or backpressure.

• They improve memory usage, startup time, and composability—perfect for files, DB pages, streams, and long-running feeds.

• Treat them like a conveyor belt: produce exactly what the consumer needs, no more, no less.