Justin Overfelt

Iterator Adapters in Go

Thu, 03 Jul 2025 13:51:41 -0400

Iterator Adapters in Go

With the release of version 1.23, Go now has support for what they call “range-over-func”, more commonly known as iterators. Essentially, it allows a developer to pass a function to for-range loops. This post won’t be about the basics of iterators in Go, as the basics are covered ably by Ian Lance Taylor on the official Go blog.

Most of the posts I’ve seen with real-world examples of iterators focus on their use when paired with custom, generic collection types. To be sure, iterators are huge improvement in the ergonomics of custom collection types in Go. However, I’ve used them much more often in the context of what I’ll call “iterator adapters”. Because Go did not have first-class support for this concept prior to 1.23, there are a number of bespoke iterator implementations in the standard library. Common ones include iterating over CSV rows using encoding/csv, fetching entries in a tar file using archive/tar, and processing chunks of a multipart message using mime/multipart.

Typically, usage of these bespoke iterators involves an unconditional loop with clauses that break out of the loop upon receiving io.EOF. The classical example with encoding/csv is below:

// ...
cr := csv.NewReader(r)

for {
    rec, err := r.Read()

    if err == io.EOF {
        break
    }

    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(record)
}
// ...

This is rather annoying if the logic is any more complex than the simple example above. It forces you to mix iteration control with your business logic. Let’s look at how we might clean this up with an iterator adapter.

func allRecords(cr *csv.Reader) iter.Seq2[[]string, error] {
	return func(yield func([]string, error) bool) {
		for {
			rec, err := r.Read()
			if err == io.EOF {
				return
			}

			if !yield(rec, err) {
				return
			}
		}
	}
}

Now, we can use this new function to significantly clean up our code:

// ...
cr := csv.NewReader(r)

for rec, err := range allRecords(cr) {
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(rec)
}
// ...

Not only is the business logic now separate from the iteration, it also reads better. Anyone looking at this code will immediately know that you’re looping over all records in a CSV file: “range over allRecords”. However, there are other benefits too. What if we decide that we don’t want the header of the CSV file? No problem, we don’t have to modify our application code at all - only the iterator adapter needs to be changed.

func allRecords(cr *csv.Reader) iter.Seq2[[]string, error] {
	return func(yield func([]string, error) bool) {
		// read off the header
		// any errors will be caught by the subsequent call to r.Read()
		_, _ = r.Read()

		for {
			rec, err := r.Read()
			if err == io.EOF {
				return
			}

			if !yield(rec, err) {
				return
			}
		}
	}
}

Done! You could even have two functions, allRecords with the original behavior and allRecordsNoHeader with the updated code. You can apply this pattern anytime you see an unconditional loop with a call to Next() or a similar function. Perhaps the Go team will ultimately modify the standard library to include iterator adapters like this. However, you may want to handle errors differently or make subtle changes like our no-header CSV change. These adapters are so simple that it might make sense to allow each programmer to define them as they desire. Enjoy!

Open Source In Amateur Radio

Tue, 17 Sep 2024 21:21:41 -0400

Open Source in Amateur Radio

When I first got my Technician license in 2019, I heard people call amateur radio “the hobby of experimentation”. I was told I had received a “license to learn”. Indeed, 47 CFR Part 97, the section of the Code of Federal Regulations that governs amateur radio in the United States says this in Subpart A under “Basis And Purpose”:

The rules and regulations in this part are designed to provide an amateur radio service having a fundamental purpose as expressed in the following principles:

…

Continuation and extension of the amateur’s proven ability to contribute to the advancement of the radio art.

Encouragement and improvement of the amateur service through rules which provide for advancing skills in both the communication and technical phases of the art.

I have found a great many fellow hams that live these principles in one way or another. One thing I was surprised to find though is that, as a community, we don’t uphold them when it comes to software. The vast majority of the software ever written for the amateur community is proprietary and Windows-only. Worse, it is Windows desktop only, with little to no consideration given to mobile or remote operation.

Yeah, but most computers run Windows

This was once true, and probably is still true if all you’re counting are desktops and laptops. However, smartphones are by far the most popular computing devices these days, and none currently on sale run Windows. In fact, today we enjoy a huge diversity of computing devices and processor architectures. You can run Linux on a RISC-V processor, Windows on an ARM processor, NetBSD on x64, and on and on. This is a very different state of affairs than the Windows-on-Intel-processors status quo that existed when amateur radio software gained popularity. The Raspberry Pi brought low cost ARM devices that ran Linux to the masses, and the Pis and their clones are still hugely popular among hams. In fact, many of the new entrants to the hobby come from the tech world now (I’m one of them!). They might want to run higher power in the 2.4GHz band, or maybe they played with Meshtastic and are looking for more. These people expect a thriving open source culture from the hobby of experimentation.

Perhaps you don’t come from this culture, and are wondering what all the fuss is about.

Why Open Source?

Many reasons are detailed elsewhere, ask your favorite LLM! I’ll focus on the reasons that are specific to amateur radio.

Be an experimenter

Open Source, and specifically Free Software, preserves your freedom. In amateur radio, it also preserves your ability to experiment. You can’t experiment and learn from closed source software. It turns you into the dreaded appliance operator. You can’t mold your setup the way you want it. You can’t support any configuration the software author hasn’t previously thought of.

Your favorite software, SK

It may be morbid, but it has to be said: many, many programs become a “silent key” with their creators. No one had the source, so there was no way to continue fixing bugs and adding features after the death of the original author. This is also a liability for emergency communications and I’m shocked that (for example) Vara has made inroads there. As far as I can tell, it’s all dependent on one guy.

Even if the author has the forethought to make a plan for releasing the code after they burn out or shuffle off this mortal coil, no one else is experienced in developing it and there’s no incentive to make it easy for others to build. Likewise, there’s no reason for the author to produce developer-focused documentation. Having the code itself is only a small part of the equation.

You might think this issue is mitigated if the software is offered by a company. Not so. Companies are bought out or fail all the time. A great many companies offered packet radio software in the 90s and 2000s. Almost none of them are around today.

An aside about hardware

Although the move to software modems has been a good thing overall for amateur radio, old-style hardware modems (“TNCs”) did have one advantage: cross platform support was “free” since they tended to operate over serial connections that worked on all operating systems. That was just about their only advantage though, as they were bulky and expensive. They were also full of proprietary firmware. As an example, although SCS Pactor modems are every bit as proprietary as Vara, they are simple to use with any operating system and architecture.

Success Stories

There is hope however. Some of the most popular packages in amateur radio today are open source:

WSJT-X
Fldigi
Pat
ARDOP
Direwolf

You can take a look at the source of any of these programs and learn something. If something doesn’t work the way you want it, you change change it! If you don’t know how to develop software, it’s quite likely someone else has had the same idea already. They can redistribute their own version or contribute back to the original.

The ARDC is doing great work in this area, funding important pursuits like Mercury, a high-performance software modem.

Closing Thoughts

There’s nothing illegal about using proprietary software on the air in the USA, provided protocols are “documented”. It should be noted that what qualifies as “documented” has never been precisely defined by either the FCC or case law.

This blog post is not an invitation to zealotry: sometimes practicality must trump idealism, and this is why I own a (secondhand) Pactor modem and am a paid Vara subscriber. I make it a point, however, to use Open Source Software whenever possible.

This is an invitation to support Open Source and Free Software by using it over alternatives, contributing when you can, and choosing open source when you write new software.

RSS Redux

Thu, 06 Jan 2022 11:07:41 -0500

RSS Redux

It’s been a while since my last post - lots going on in life. I’m back and looking to rehash a previous topic: RSS via email. In a previous post, I talked about rss2email on SDF. I eventually moved that setup over to another pubnix called radiofreqs.space, but that was taken offline at some point in 2021. This left me with no good way to get RSS feed updates delivered to my inbox. It was a good time to look into an alternative anyway as I was already moving my personal email over to Migadu, but that’s another post!

FeedMail

My next move was to scour HackerNews threads for solutions as I can’t have been the only person too lazy to set up a personal server for sending email from one of my domains. I located a comment mentioning FeedMail, which remains my primary solution to this day. Just like rss2email, it will send you a separate email for each new post from every feed you’re subscribed to.

The setup is very straightforward and the site itself has a pleasing, minimal look. It supports having separate emails for administration and feeds, even supporting different emails on a per-feed basis. The “credit” system is very easy to understand and hopefully will make FeedMail sustainable in the long term. It supports OPML import, so I was quickly off to the races. I had a minor issue with one of my feeds not following the Atom standard properly and I found FeedMail’s maintainer to be incredibly responsive.

However, once I was forced to step back and re-evaluate my RSS feed consumption, I realized that there were several feeds I followed that don’t fit nicely into the one-email-per-post model. These feeds shared a couple characteristics:

They were high traffic, commonly 5-10 new posts during an average weekday
I was interested in, at most, 1 or 2 of those new posts on a given day. The others were deleted after a glance at the title. My first thought was maybe I shouldn’t be following these feeds at all but I didn’t want to give up those interesting articles buried in the noise.

Given the payment system that FeedMail uses, these feeds were burning a lot of credits sending me emails I would immediately delete. Back to HN to find a complementary solution!

RSS by email

Another helpful HN comment led me to RSS by email. This service is even more minimalist than FeedMail: you interact with it solely through the use of email. Special messages to the service are used to manage feeds, and you get a lightly-formatted digest of all new articles daily at a UTC hour of your choosing. This is the perfect solution for the issue of feeds with low signal to noise ratio. Now, I can just scan past any articles I don’t care about each day.

As far as I can tell, RSS by email is being run as a hobby by its creator, so I worry about its long-term viability. There is no donation link that I can find.

That being said, my heuristic going forward is that low-traffic or otherwise high priority feeds are managed by FeedMail, while firehose-type feeds are handled by RSS by email.

Thanks!

My goal for 2022 is to post more than one blog post every two years…

Images from the ISS

Wed, 15 Jan 2020 19:51:43 -0500

Images from the ISS

Last year I became a licensed amateur radio operator, first getting my Technician license and ultimately getting my General class license a month or so later. While I was a Technician class operator with limited privileges on the HF bands, I sought out things to do with my full rights on VHF and UHF frequencies.

Turns out, there are quite a few satellites in Low Earth Orbit that are usable by ham radio operators to communicate with one another over long distances. They’re basically signal repeaters in the sky. Anyway, this post is not about those satellites, it’s about the International Space Station (ISS). The ISS has similar capabilities to the amateur satellites I mentioned - you can even talk to the crew sometimes! One other thing the ISS can do is send images receivable by anyone with a handheld radio or RTL-SDR.

Slow-Scan Television

The ISS crew sends these images on a VHF frequency (145.800 MHz) using a mode called Slow-Scan Television (SSTV). It’s essentially images over radio and it sounds like this. Typically the images are sent by the ISS crew to commemorate specific events and the broadcast times are announced in advance.

Receiving the Signal

As I mentioned, there are many ways of receiving and decoding the SSTV signal from the ISS, but you’ll need 4 basic things:

A receiver that can pull in FM signals on 145.800 MHz. This could be a handheld radio that has coverage on that frequency or something general like an RTL-SDR
An antenna: I’ve found that the signals sent by the ISS are pretty strong and can be received with even a marginal antenna like the dipole that comes with the RTL-SDR
Pass-prediction software: The ISS is not geosynchronous like the satellites that stream television or radio. It moves through the sky. Plug “ISS pass prediction” into your favorite search engine or just use the ISS Detector app like I do.
Decoding software: Something to turn the sounds into an image. Again, searching “SSTV decoding software” will turn up answers, but I use the Robot36 app on Android. The ISS typically uses the “PD120” format.

My Method

I have an antenna specifically designed for working Low-Earth Orbit satellites called an Arrow Antenna, but I’ve been told that more basic antenna work fine. I connected that antenna to my handheld transceiver (Yaesu FT-60R) and opened the Robot36 app on my phone. I then made sure the phone was close to the radio and turned the volume up. Since I was using a highly directional antenna, I used the ISS Detector app to help me point it correctly at the sky. The Robot36 app began decoding once it heard the start of the image signal, ultimately producing this

which I saved off from my phone and cropped it a bit. Not bad for an image from space!

Reviving a Raspberry Pi with NetBSD

Tue, 23 Apr 2019 10:35:48 -0400

Reviving a Raspberry Pi with NetBSD

I started getting into BSDs (specifically NetBSD) as a result of my membership in SDF. I have a good deal of Linux experience but hadn’t really had a good reason to try out one of the BSD operating systems. I remembered I had a first-gen Raspberry Pi (Model B) languishing in my desk drawer and decided to put NetBSD’s slogan (“Of course it runs NetBSD!”) to the test.

The Model B features a 32-bit armv6 chip and 256MB of RAM (!). I poked around on the NetBSD homepage and found this page which told me “earmv6hf” was the NetBSD port I wanted for an rpi 1.

This article on opensource.com does a fantastic job of showing the initial steps to get it running, and I won’t attempt to reproduce those here. Instead, I want to fill in some gaps.

WiFi

So you’ve followed the opensource.com article above and you’re sitting at a root prompt with your Pi hooked up to a TV and a keyboard plugged into one of the two USB ports. Now, you want to be able to use the Pi as a “server” and stash it somewhere. Unfortunately, Pis this old don’t have onboard WiFi, so you’ll need an adapter. This is one area where NetBSD is a bit rough around the edges: the WiFi adapter support isn’t as strong as some Linux distributions. Even if your Pi has onboard WiFi, it is unlikely that NetBSD supports it.

I used this adapter and I know that it works out of the box with NetBSD 8.0. If that Amazon link ever dies, the adapter is the Edimax EW-7811Un. Be aware that this adapter does not have dual-band support, so you’ll have to stick with 802.11n. This means you can’t use the 5GHz band of your router.

Once you have the adapter, shut down the pi at the root prompt with shutdown -p now. You’ll eventually see a message saying “press any key to restart”. Once you see this, unplug the Pi. Now, plug in the Edimax adapter to the empty USB port and reconnect the Pi. When you’re back at your root prompt, follow the instructions here to set up wpa_supplicant, which is the NetBSD daemon that handles WPA/WPA2. I was able to copy/paste the “simple” example from the wpa_supplicant page:

ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=wheel
network={
        ssid="my favourite network"
        psk="hunter2"
}

I replaced the ssid and psk with my network’s info. The tutorial tells you to edit /etc/rc.conf and add the following lines:

wpa_supplicant=YES
wpa_supplicant_flags="-i iwn0 -c /etc/wpa_supplicant.conf"

iwn0 is a network interface - find yours by typing ifconfig and looking for interface name whose address starts with 74:da:38 (provided you’re using my recommended adapter). Mine is urtwn0, so I replaced iwn0 in the above command with urtwn0. Now you should be able to find the Pi on your network and SSH into it - no need for the TV anymore!

bash issues

After enabling bash as my shell per the opensource.com guide, I found that some things were not on my $PATH by default. I loaded a custom .bashrc file to get my normal aliases, and found that “pkg_add” was no longer on the PATH. Also, it seems that no .bash_profile is created by default in your user’s home directory. To fix both these issues, add the below lines to the newly-created ~/.bash_profile:

PATH=$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin:/usr/pkg/bin
PATH=${PATH}:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin

source .bashrc

pkgin

The opensource.com guide talks about pkgsrc, which allows for the installation of both binary and source packages for NetBSD. If you’re coming from Linux though, you’re probably expecting something that works like yum or apt. pkgsrc does not work this way and crucially, it doesn’t have a great story for updates. Pkgin is what you’re looking for here. Run pkg_add -v pkgin as root and then follow the guides on the pkgin homepage from there. Pkgin uses pkgsrc under the hood. Remember that your old Pi 1 is very underpowered, so expect package installations to take a while.

After you’ve installed pkgin, there will be a message telling you to run pkgin update. Before you do this, check the file in /usr/pkg/etc/pkgin/repositories.conf. The url in there was wrong by default on my installation. For an rpi 1, it should be ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/earmv6hf/8.0/All. Once that’s correct, run pkgin update.

Assorted gotchas

Never unplug the Pi without running shutdown -p now first and waiting a bit if you’re doing that over SSH. The older models seem especially sensitive to corruption of the SD card. If this happens to you, you’ll have to hook your Pi up to a TV again and single-user mode will be active. From there you can use fsck -tufs -y /dev/<your sd card> You can typically find your sdcard name by running fdisk. In many scenarios this will repair your disk. If it fails for some reason, well, GOTO the top of this guide and reimage.
Many of the tools you’re used to having out of the box in Linux distributions will not be installed. NetBSD is highly configurable and does not attempt to decide what you should use. Most things will be easily available on pkgin though.

Go forth!

Now you can install the packages you’re used to, like vim, git, etc. Happy hacking!

Thanks!

I hope you’ve enjoyed this post. Hit me up on Mastodon with any questions or comments!

The Best RSS Reader Is Your Inbox

Wed, 03 Apr 2019 20:28:42 -0400

The Best RSS Reader Is Your Inbox

I was very late to the RSS game, especially for a programmer. It wasn’t until a little over a year ago that I got serious about making RSS/Atom a part of my daily content diet. Since that time, I’ve used a few readers, namely Inoreader and an instance of Tiny Tiny RSS lovingly hosted by SDF for the benefit of MetaARPA members like myself. I enjoyed both those products, but it always felt a bit like my feeds were “locked away” (even though OPML export is easy in both tools).

It finally dawned on me over the last week or so that all RSS readers (that I’ve used anyway) really just reinvent email. There’s folders the navigate things and “stars” to save articles you want to refer to later. There’s the concept of read and unread articles. You can load images or just view text. Sometimes the full content is available, other times there’s just a link. Most importantly, I find that RSS readers implement “rules” that do not compare to what is available from various email clients. I thought to myself: “I bet I could write a tool that scraped RSS feeds and sent emails very easily!”. Turns out, I’m far from the first to have this idea (much like many of my ideas).

rss2email

Enter rss2email. It appears this tool has been around for quite a while and I’m very late to the game (again). I imported my aforementioned OPML file with r2e opmlimport and I was off to the races. Well…sort of. You have to configure a way to send email of course! Since I was running this on the SDF Cluster, I was able to use their SMTP server by adjusting the config file (~/.config/rss2email.cfg). By default, it wants to use sendmail and to be honest I was more comfortable with straight SMTP. The key setting here is email-protocol. There are other SMTP settings in there that were pretty clear in my opinion.

Time to read some feeds! Be sure to invoke r2e run the first time with the --no-send flag or be prepared for a deluge of email as it sends you everything from all your feeds! After that, r2e run will suffice. Helpfully, r2e has -V flags that control the verbosity of its output. Once you have a manual run that worked, I’d recommend setting up a cron job to run it on a schedule.

Once I got all that working, I perused the other options in the config file, including some for HTML mail. I don’t typically like HTML mail, but the formatting used by default when you enable HTML is very light. There is a “digest” mode that sends one email for each feed for a given run, but it attaches the articles as emails to the digest email, which was off-putting to me. Maybe you like that. In any case, hopefully this helps someone else even later to the game than me!

Other Options

A quick web search indicates that there are services out there which will do this for you, though many seem to be targeted at businesses wanting to email their customers when they have a new post. One of those products might work better if you aren’t able to set up rss2email for whatever reason!

Thanks!

I hope you’ve enjoyed this post. Hit me up on Mastodon with any questions or comments!

Server-Sent Events

Sun, 17 Mar 2019 17:47:41 -0400

Server-Sent Events

Often when the topic of web application notifications comes up, us web developers are quick to reach for WebSockets. There exists a wealth of information on them in the blogosphere and most importantly: on StackOverflow. However, WebSockets are too big a hammer for many of the cases in which they are most commonly used. Instead, I’d like you introduce you to Server Sent Events.

Why use SSE?

Unlike WebSockets, Server-Sent Events are for one-way communication (server pushes to client). When you’re implementing live notifications, this is all you typically need. If you really need full-duplex communication between client and server (maybe you’re making a chat app), well, then stick to WebSockets.

I find the main benefit of SSE to be its simplicity - it’s just HTTP(S)! When using WebSockets, the connection starts as HTTP, but an “Upgrade” request is sent and the connection is no longer plain HTTP. With SSE, you can continue to write HTTP handlers the way you always have in the language and framework of your choice. You just set some headers, and then write data to the client as you normally would (well, in a simple text format I’ll detail below…). This means you don’t have to change anything about your web proxy or any other part of your stack. Feel free to continue to use the same authentication you use for all your other HTTP handlers. Sysadmins love it!

Furthermore, the JavaScript interface for working with Server-Sent Events handles automatic reconnects! The reconnect time is even configurable. With all that said, let’s move onto the SSE protocol.

The SSE Protocol

The SSE protocol is text-based, and has mercifully few concepts to learn. If all you want to do is send a line of text to the client, this is the format:

data: something happened!\n\n

The second newline means “send the message”. The first newline is there in case you want to send messages that span more than one line. This is useful for sending JSON:

data: {\n
data: "foo": "bar"\n
data: }\n\n

Now, it may be the case that you want to have different kinds of messages in the same “pipe”. Maybe a table is being updated by others and rows can be added or removed. You want to notify the user of both kinds of changes. Well, SSE has you covered there too with the “event” tag:

event: add\n
data: added one\n\n

event: delete\n
data: deleted one\n\n

Finally, you can give events ids with the “id” tag:

id: 1\n
data: i'm one!\n\n

It’s past time to give the people what they want: code samples! All server-side code in this post will be in Go, but it should translate to the language of your choice easily.

The Backend

The implementation of a full SSE server is out of the scope of this blog post (which is already too long), if you’d like something more in-depth for Go, I heartily recommend this post from ThoughtBot.

Instead, we’ll cover the basics of writing an HTTP handler to do Server-Sent Events in Go. As is customary for lazy bloggers like myself, I’ve elided error handling. Furthermore, the code assumes you have a some sort of broker which hands out channels (little typesafe queues that Go provides) for clients to read from. This broker also allows ID generation and registering/deregistering clients. Don’t get bogged down here, just think “this broker returns things that we can read messages from”. More information on doing this is in the ThoughtBot blog post referenced earlier. I haven’t included that here because it will be heavily dependent on your use case.

package main

func main() {
	// set up the fictional broker
	broker := &Broker{}
	http.HandleFunc(eventHandler(broker))

	http.ListenAndServe(":3131", nil)
}

func eventHandler(broker *Broker) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		id := broker.GenerateID()
		msgChan := broker.RegisterNewClient(id)

		w.Header().Set("Content-Type", "text/event-stream")
		w.Header().Set("Cache-Control", "no-cache")
		w.Header().Set("Connection", "keep-alive")

		// in real code, check to make sure this cast works!
		flusher, _ := w.(http.Flusher)

		for {
			select {
			case msg := <-msgChan:
				fmt.Fprintf(w, "data: %s\n\n", msg)
				flusher.Flush()
			case <-r.Context().Done():
				broker.DeregisterClient(id)
				return
			}
		}
	}
}

Ok, there are some things to unpack here. I don’t want to talk too much about the Go-specific parts, because this is an SSE tutorial. Here are the important bits:

The setting of the headers, specifically the Content-Type header. This must be set to text/event-stream. The other headers are for ensuring that the browser doesn’t cache messages and telling it that this connection will be long-lived.
Writing data out to the client with Fprintf. We’re using the aforementioned format here with the data: tag.
The “infinite” for loop, which writes data to the client until the user navigates away from the page.

I claim that this is conceptually straightfoward when compared to WebSockets and allows nearly endless tailoring to fit the needs of your application. To summarize: set the right headers, and write data out in the SSE format until the user leaves or otherwise says they don’t want updates anymore.

The Frontend

The EventSource interface is all you need!

var client = new EventSource("https://mysite.local:3131")
client.onmessage = (e) => { console.log(e.data) }

onerror and onopen handlers are also available for handling errors and performing tasks when the connection is opened.

You might do any number of things in the onmessage handler, perhaps rendering a new row in a table, or adding the line of data you just received to an array and having Vue/React/etc rerender your component!

If you broke up your events by type like I mentioned earlier, you can set up your handlers like this:

client.addEventListener('add', (e) => { console.log(e.data) }, false)
client.addEventListener('delete', (e) => { console.log(e.data) }, false)

Thanks!

I hope you’ve enjoyed this post. Hit me up on Mastodon, especially if I did something wrong!