Iterator Adapters in Go

With the release of version 1.23, Go now has support for what they call “range-over-func”, more commonly known as iterators. Essentially, it allows a developer to pass a function to for-range loops. This post won’t be about the basics of iterators in Go, as the basics are covered ably by Ian Lance Taylor on the official Go blog.

Most of the posts I’ve seen with real-world examples of iterators focus on their use when paired with custom, generic collection types. To be sure, iterators are huge improvement in the ergonomics of custom collection types in Go. However, I’ve used them much more often in the context of what I’ll call “iterator adapters”. Because Go did not have first-class support for this concept prior to 1.23, there are a number of bespoke iterator implementations in the standard library. Common ones include iterating over CSV rows using encoding/csv, fetching entries in a tar file using archive/tar, and processing chunks of a multipart message using mime/multipart.

Typically, usage of these bespoke iterators involves an unconditional loop with clauses that break out of the loop upon receiving io.EOF. The classical example with encoding/csv is below:

// ...
cr := csv.NewReader(r)

for {
    rec, err := r.Read()

    if err == io.EOF {
        break
    }

    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(record)
}
// ...

This is rather annoying if the logic is any more complex than the simple example above. It forces you to mix iteration control with your business logic. Let’s look at how we might clean this up with an iterator adapter.

func allRecords(cr *csv.Reader) iter.Seq2[[]string, error] {
    return func(yield func([]string, error) bool) {
        for {
            rec, err := r.Read()
            if err == io.EOF {
                return
            }

            if !yield(rec, err) {
                return
            }
        }
    }
}

Now, we can use this new function to significantly clean up our code:

// ...
cr := csv.NewReader(r)

for rec, err := range allRecords(cr) {
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(rec)
}
// ...

Not only is the business logic now separate from the iteration, it also reads better. Anyone looking at this code will immediately know that you’re looping over all records in a CSV file: “range over allRecords”. However, there are other benefits too. What if we decide that we don’t want the header of the CSV file? No problem, we don’t have to modify our application code at all - only the iterator adapter needs to be changed.

func allRecords(cr *csv.Reader) iter.Seq2[[]string, error] {
    return func(yield func([]string, error) bool) {
        // read off the header
        // any errors will be caught by the subsequent call to r.Read()
        _, _ = r.Read()

        for {
            rec, err := r.Read()
            if err == io.EOF {
                return
            }

            if !yield(rec, err) {
                return
            }
        }
    }
}

Done! You could even have two functions, allRecords with the original behavior and allRecordsNoHeader with the updated code. You can apply this pattern anytime you see an unconditional loop with a call to Next() or a similar function. Perhaps the Go team will ultimately modify the standard library to include iterator adapters like this. However, you may want to handle errors differently or make subtle changes like our no-header CSV change. These adapters are so simple that it might make sense to allow each programmer to define them as they desire. Enjoy!