Posts on Joel Beckmeyer's Blog

Consistency

joel@beckmeyer.us (Joel Beckmeyer) — Sun, 04 Apr 2021 00:00:00 -0500

I’ve seen a lot of talk about this stuff:

“Check out my FOSS project (hosted on Github)”
“Wayland is a great innovation and boon to the community! Also, there are very few tools/alternatives available yet for your favorite X11 tool!”
“We love open source! Also, we develop the most popular proprietary operating system!”
“Do as I say, not as I do.”

We love to poke fun at and expose this kind of stuff, which is all fine and dandy. I think it’s an interesting (and important) part of our humanity that this kind of thing bugs us so much. Think about that last point, which at least in my experience, is something I loved to fault authorities for.

Hypocrisy is fun and also infuriating to uncover in others, but how often do we do a “consistency check” on ourselves? Is what we are saying evidenced by the rest of our actions?

That’s a hard look sometimes. I know it is for me, since I’m very quick to judge others, but don’t often think about how I fail at my own principles.

Example: As a FOSS advocate, it’s nearly natural to assume that everything will be better and easier with more people using FOSS. When evidence seems to point to the contrary (e.g. fighting with Matrix/Element to get it working for my family and friends), I don’t own up to the fact that it isn’t easier, and that is an actual problem.

If we truly want to build a welcoming and wholesome community, let’s be careful to do a consistency check to make sure nothing smells foul.

Better?

joel@beckmeyer.us (Joel Beckmeyer) — Sat, 03 Apr 2021 22:15:44 -0400

There are many that say
(and I tend to agree)
that free software is the best there could be.

But please don’t mistake
using software that’s free
as a right to superiority.

There are many that go
from day to day living
and don’t give a thought to what they are using.

Are they worse for this?
Are you better for caring?
Sometimes the truth can be quite baring.

That not every human
in present circumstance
is able or willing to take a chance.

‘Cause that’s what it is,
taking a chance and going
into the unknown with fear, and knowing

that what you might find,
may not truly be better.

But instead simply different;
and still made by a stranger.

Moving Back To OpenSSL

joel@beckmeyer.us (Joel Beckmeyer) — Mon, 22 Mar 2021 11:00:00 -0400

Void Linux recently announced that they were going to move back to OpenSSL after originally switching to LibreSSL in 2014. It seems that there are a lot of things at play here.

It seems that the main focus of the recent announcement is on the maintainability and other difficulties of not using the one true SSL/TLS library. To me, this pragmatically makes sense. However, every time something like this happens I get this lingering feeling of worry…

Microsoft moving their default browser from their own implementation to Chromium, and other browsers following suit.

Linux distributions moving en masse to systemd.

Distributed email being slowly crushed and killed by Google with GMail.

And many other examples that aren’t immediately coming to mind.

I think it’s great that OpenSSL as a project has made a comeback from the Heartbleed fiasco, and that it is apparently more actively developed nowadays, but the fact that we are even at the point of moving back to OpenSSL due to difficulties with building software is worrying. To me, it looks like a symptom of software becoming too entrenched and dependent on a single piece of software.

This kind of accusation coming from anyone is going to be hypocritical, since we all depend on Linux, X11, Wayland, systemd, or some common piece of software that we take for granted and don’t lose sleep over. However, I think what’s categorically different about this one is that an alternative was adopted, worked on, but eventually “failed” (at least for Void, but also possibly for Linux as well).

I don’t know what the fix for this specific issue would be. I’m not nearly familiar enough with SSL/TLS or how you would develop software to be agnostic of dependencies like this. But I think in order to honor principles like the Unix philosophy, the KISS principle, and countless others, we need to figure out a way to be more modular for dependency issues like this.

The Generation Ship Problem

joel@beckmeyer.us (Joel Beckmeyer) — Fri, 19 Mar 2021 15:00:00 -0400

After talking about the hardware and software problems of digital permanence, I’m struck by a classical Sci-Fi motif with a conundrum: the Generation Ship; a ship outfitted with all of the technology, infrastructure, and storage to support lightyear-scale human travel.

But what about that technology on the ship? If we build one of these ships, we need to accomplish one of several things in regards to information storage:

1. Innovate to the point where the lifetime of the storage devices is able to support lightyear scale travel.

That’s a tall order, given where we are right now with physical storage devices. As I mentioned in one of my previous posts, the average lifetime of physical storage devices is less than 100 years, no matter if it is a hard drive, solid-state drive, etc.

2. Provide the facility to create new storage devices to replace the failing old ones.

Again, in my mind a tall order, since it would require facilities on the ship to create storage devices. The problem of having materials is at least solvable by just sending the ship with all of the materials it needs in advance.

3. Provide the facility to revitalize storage devices.

One of the main reasons I’m even thinking about this is because I’m an individual with limited resources. Accordingly, I think about things in terms of broken/working, on/off, etc. With enough resources, there is a much larger chance of being able to repair, re-purpose, and otherwise revitalize storage devices, increasing their lifetime. E.g., if the only failure in the hard drive is the control circuit, that is an “easy enough” repair.

I like to toy with the idea of a generation ship a lot in my head, but I think it’s really fun to think about the technical possibilities and needs of a ship like this.

Volatile Formats

joel@beckmeyer.us (Joel Beckmeyer) — Thu, 18 Mar 2021 14:24:00 -0400

Note: This is a continuation of the thoughts I started thinking about in my Volatile Mediums blog post.

The next level up from physical mediums for data storage is the way that the data is stored. In the digital age, we have a plethora of formats for storing information. For me, one of the most interesting areas of information storage is the analog-digital space.

The fundamental problem of storing audio, video, and other replications of the physical world is that there is so much information that we can collect with sensors (think microphones, video cameras, etc.). It would be great if we could go get the best camera and microphone out there, record whatever people record these days, and have that exact physical experience “played back” for us on a screen and speaker/headphones.

Unfortunately, there are several problems with this. Among those is the actual design of the sensor. It takes a lot of careful thought, engineering, and the like to create a truly good microphone or camera. And after all of that, this sensor will cost something. Hopefully, that cost will correspond to the actual technical ability of that sensor! In any case, not everyone can have the best camera or microphone due to any number of constraints, not just those listed above.

The second problem is the sampling issue. The sensor will create some sort of output that can then be measured, or sampled, by an ADC (analog-to-digital converter). The very word “sample” belies what this nearly magical box is doing: it is only looking at certain portions or timestamps of the analog signal. Granted, the time between samples can be very small (e.g. 44.1 kHz is a fairly common sample rate for audio), but there is still some loss of signal. Once the ADC creates these samples, it converts them into a digital format (something that can be stored on a CD, hard drive, thumb drive, etc.).

The third problem is the encoding issue. The ADC creates all of these samples, but we need to start thinking about storage limitations. Storing the raw output of a sensor can take a lot of space: an average album length (40 minutes) could easily take 400MB of space! Now, again, the physical storage space is moving in the upward direction to combat this, but storing isn’t the only problem. One prime issue is internet bandwidth.

The solution to this is compression, like a ZIP file. It makes big files smaller by doing some fancy math tricks that can be reversed by a computer to reconstruct the original file. However, for audio/video files, another level of compression exists which actually gets rid of some of the information in the original file to save more space. This is called “lossy” compression, as opposed to “lossless” compression.

Great! We’ve found a way to save more space. The problem with lossy compression is that we have to decide which information to throw away. Usually, this is frequencies that the average human ear/eye can’t perceive. But, let’s just say that some compression is a bit too “greedy” when it comes to saving space and starts to cut into the band of frequencies that can be perceived. Also note that the design of these compression algorithms is an artform and takes lots of careful consideration.

The final problem I want to mention is the codec problem. There are many different codecs available today, and for each and every one of them to be useful, you need to have a way to decode each and every one of them. Unfortunately, this is sometimes very difficult.

It could be a licensing issue, where you don’t have the correct software installed or purchased to actually decode that file on your computer.

Or it could be a physical constraints issue, where your computer isn’t powerful enough to decode the file at a fast enough rate for you to view it without stuttering, buffering, etc.

Third, it could be a personal preference. Some people have much more sensitive eyes/ears and need to have formats that are more transparent, meaning that the lossy file is perceptually identical to the source it was encoded from.

With all of these issues at play, I think there are several key points to make:

1. Codecs need to be freely available for widespread use with no strings attached.

Can’t stress this one enough: we need to make sure we are doing everything possible to not let our information die when a corporation or individual makes a decision that impacts the “who, what, where, when, and how” of their codec usage.

2. Lossless compression is good, but it is not the only thing we need.

We need to remember that not everyone has the ability to use lossless codecs, whether that be because of internet bandwidth limitations, storage limitation, or the like. Instead, we need to continue to innovate in the lossy compression space to narrow the perceptual gap between lossy and lossless more and more.

3. A codec should never become obsolete.

This one may sound weird, but the fact is, if we’re talking about long-term storage of information, we can’t let codecs die, since there may come a day where we need a codec to decode great-grandpa’s album that never made it big.

OpenWRT + Unbound + adblock

joel@beckmeyer.us (Joel Beckmeyer) — Fri, 05 Feb 2021 19:03:15 -0500

I decided to do some work on my Linksys WRT32X running OpenWRT to make it a little more useful.

Unbound is a DNS resolver which I like because it’s recursive, meaning it directly queries the root servers instead of relying on existing DNS servers run by Google, Cloudflare, your ISP, or the like. I already have it running on several of my servers and computers, but I figured it would be great if everything on my network can use Unbound and be, well, unbound from all of those intermediary DNS servers.

Luckily, OpenWRT already has Unbound packaged, and also has a useful LuCI app that goes with it (LuCI is the graphical web interface that comes with OpenWRT). All I had to do was install luci-app-unbound, which pulls in all of the necessary dependencies to run unbound.

After that finished installing, I refreshed LuCI/OpenWRT and went to “Services” on the top, and there it is!

At this point, you’ll have to get your hands dirty. You can either dig through some LuCI menus or SSH in and make some edits. For reference, I’m using “Parallel dnsmasq” section from the README for unbound in the OpenWRT packages (which has a lot of other useful information as well!). Essentially, I made the edits to /etc/config/unbound and /etc/config/dhcp after SSH’ing in. However, you can make the same edits through LuCI.

For the /etc/config/unbound edits, you can make the edits to the file in LuCI directly at “Services -> Recursive DNS -> Files -> Edit: UCI”:

For the /etc/config/dhcp edits, you can make the edits by finding the same fields under “Network -> DHCP and DNS”:

However, the field names are different from the lines in the config, so they would need to be researched to determine which fields in LuCI map to which lines in /etc/config/dhcp.

At this point (or maybe after restarting unbound and dnsmasq, which is a lot easier using SSH and /etc/init.d ... restart as well), OpenWRT should now be using unbound for resolving all DNS lookups, while dnsmasq is only used for DHCP-DNS.

Bonus: you can also enable a nice status dashboard in LuCI under “Services -> Recursive DNS -> Status”, but this requires installing several more software packages: unbound-control and unbound-control-setup. You will also need to change a line in /etc/config/unbound:

...
option unbound_control '0'
...

becomes

...
option unbound_control '1'
...

A word of warning: there is another section on “Unbound and odhcpd” which tries to cut out dnsmasq completely. However, when I tried to set this up, I got myself into a lot of trouble (had to reset OpenWRT, re-install any extra software packages, and restore configuration from backup). It is also possible that if you mess up the configuration for the “Parallel dnsmasq” method, you could end up in a similar error state and have to start over. Please be careful when doing this and don’t change anything you’re not supposed to.

Now, moving on to adblock, which should be much simpler to setup. First, install luci-app-adblock and refresh. Navigate to “Services -> Adblock”:

Check the settings at the bottom. The only thing you need to get going is to go to the “Blocklist Sources” tab and choose your blocklists.

The adblock readme has some more info on what each list is. After that, make sure “Enabled” is checked under the “General Settings” tab:

and click the “Refresh” button above:

Then you’re good to go; adblock should work out of the box with unbound; cheers!

ADDENDUM: Another word of warning: once you’ve setup adblock, it will download the blocklists, merge them into a single file at /var/lib/unbound/adb_list.overall, and try to restart unbound. I recommend not trying to view/interact with adblock or unbound during this restart, which can take anywhere from 30 seconds - 2 minutes. Just leave them alone in LuCI for a little bit…

Hello doas

joel@beckmeyer.us (Joel Beckmeyer) — Sat, 30 Jan 2021 15:15:55 -0500

Today, I switched my workstation from sudo to doas. I’m running Void Linux, and the process was fairly easy.

First, I needed to figure out how to remove sudo (yes, I realize I could have installed doas first, then removed sudo, but I decided to do it the hard way.) As it turns out, the advanced usage section of the XBPS manual details how to use the ignorepkg entry in xbps.d with nothing other than this exact use case! I created the file /etc/xbps.d/20-ignorepkg-sudo.conf with contents

ignorepkg=sudo

and then ran sudo xbps-remove sudo (an ironic command).

After that, because I was stupid and removed sudo before I had set up doas, I had to use plain-old su to change to the root user and run xi opendoas. I also configured doas in /etc/doas.conf with the following:

# see doas.conf(5) for configuration details
permit nopass keepenv :admin

I ran groupadd admin, usermod -aG admin joel, and then logged out so that my user account would see the new group perms.

And just like that, I can now run doas xbps-install ... and all of my other commands, just substituting doas for sudo.

The one thing I immediately missed was sudoedit. Before I accidentally tried to use sudo for the first time, I had already accidentally tried to run sudoedit at least 5 times. I had to fix this. I saw a discussion on Reddit where one user suggested writing a script to replace the sudoedit functionality. I quickly starting hacking together something like that. I started with:

#!/bin/sh
mkdir -p /tmp/doasedit
doas cp $1 /tmp/doasedit/tmp_file
$EDITOR /tmp/doasedit/tmp_file

And quickly ran into my first road-block. The script is going to have to change the permissions of that file before the user can edit it. But if the script changes the permissions, how can I restore it to the original location with the right permissions? cp /tmp/doasedit/tmp_file $1 won’t work. I thought about just using cat to overwrite the file contents in-place (cat /tmp/doasedit/tmp_file > $1). That could create some issues if a program has the file open. Instead, a better option is to create two copies of the file–one for editing, and one for preserving file attributes:

#!/bin/sh
mkdir -p /tmp/doasedit
doas cp $1 /tmp/doasedit/edit
doas chown -R $USER:$USER /tmp/doasedit/edit
doas cp $1 /tmp/doasedit/file
$EDITOR /tmp/doasedit/edit
cat /tmp/doasedit/edit | doas tee /tmp/doasedit/file 1>/dev/null
doas mv -f /tmp/doasedit/file $1
rm -rf /tmp/doasedit

Of course, the issue with this is that it only works with absolute paths. I want to make it work for relative paths as well. I’m going to take advantage of realpath, which is part of the coreutils package from Void. As a bonus, this will also take care of the edge case where the given file is a symlink (IIRC, sudoedit didn’t follow symlinks, so I may be diverging here):

#!/bin/sh
mkdir -p /tmp/doasedit
srcfile="$(realpath $1)"

doas cp $srcfile /tmp/doasedit/edit
doas chown -R $USER:$USER /tmp/doasedit/edit
doas cp $srcfile /tmp/doasedit/file

$EDITOR /tmp/doasedit/edit

cat /tmp/doasedit/edit | doas tee /tmp/doasedit/file 1>/dev/null
doas mv -f /tmp/doasedit/file $srcfile

rm -rf /tmp/doasedit

At this point, it works…okay-ish. It can only be used in one instance currently since I hard-coded /tmp/doasedit/file and /tmp/doasedit/edit, but that’s easily fixed:

#!/bin/sh

destfile_pfx="$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)"

while [ -d "/tmp/doasedit/$destfile_pfx" ]; do
	destfile_pfx="$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)"
done

mkdir -p /tmp/doasedit/$destfile_pfx
srcfile="$(realpath $1)"

doas cp $srcfile /tmp/doasedit/$destfile_pfx/edit
doas chown -R $USER:$USER /tmp/doasedit/$destfile_pfx/edit
doas cp $srcfile /tmp/doasedit/$destfile_pfx/file

$EDITOR /tmp/doasedit/$destfile_pfx/edit

cat /tmp/doasedit/$destfile_pfx/edit | doas tee /tmp/doasedit/$destfile_pfx/file 1>/dev/null
doas mv -f /tmp/doasedit/$destfile_pfx/file $srcfile

rm -rf /tmp/doasedit/$destfile_pfx

At this point, the only thing missing is the check to see if the file was actually edited:

...
cat /tmp/doasedit/$destfile_pfx/edit | doas tee /tmp/doasedit/$destfile_pfx/file 1>/dev/null

if cmp -s "/tmp/doasedit/$destfile_pfx/file" "$srcfile"; then
	echo "Skipping write; no changes."
else
	doas mv -f /tmp/doasedit/$destfile_pfx/file $srcfile
fi
...

I put this in a repo on GitHub if anyone is interested. I know that a major weakness of this script is the number of times it calls doas, which could break flows where password is required every time doas is run.

Volatile Mediums

joel@beckmeyer.us (Joel Beckmeyer) — Fri, 29 Jan 2021 23:36:00 -0500

I’ve recently been thinking a lot about storage mediums [1] – especially in the long-term.

Technology has made a lot of progress. Digital storage mediums started out only being able to store 224KB on a tape drive for an average lifetime of up to 30 years. Now, we can store terrabytes of data on hard drives and solid-state drives. However, no one ever really answered the question about long-term storage.

(Note: the following is based off an assumption that the storage medium is only being used to make backups or archive data. The device itself could be unplugged and stored when no backup is in progress.)

Even though theoretically hard drives could store data for 20+ years, random bit flips, drive failure, etc. all make hard drives too volatile of an option. As always, of course redundancy takes away some of these issues.

SSDs are in an even worse position: they cost significantly more than hard drives per TB right now, and last I heard, there were still issues with bit fade when unpowered.

CD/DVD is sounding a lot better, but there are some serious issues here too. Variable quality directly impacts the storage lifetime. Physically storing the discs is a lot more risky since the disc itself doesn’t have as much built-in protection as a hard drive or SSD has. You’ll need a much larger quantity to store the terrabytes of data that you can easily dump on one hard drive. And finally, life expectancy is still fairly low – while manufacturers of recordable discs (the ‘R’ in CD-R, DVD-R, etc.) claim life expectancies of 100-200 (!) years under optimal conditions, others are slightly more conservative, giving an estimate of 30 years. Oh, and remember how I mentioned this is for recordable discs? That means they’re single write. The random access (RW - CD-RW, DVD-RW, etc.) discs have even lower life expectancies.

All in all, humanity has not gotten very far with the digital storage medium. All of these life expectancies have an inconsequential variance when we zoom out to the century view of history.

[1] And no, I’m not talking about the kind you pay to see your dead great-great-aunt to figure out if you’re actually related to George Washington.

This is intended to be the beginning of a learning series/personal study on the issues surrounding information preservation, digital permanence, and their related issues.