A month ago, Amazon released Amazon Linux 2015.03, their “continuous release” Linux distribution. Widespread sadness ensued, and I tweeted:

While factually correct, that’s a bad tweet which I quickly regretted.

It’s bad for being counterfactual, and displaying no trace of empathy.

• “If you didn’t want surprise updates” – who does want surprise updates?
• “You should have locked repo_releasever” – I realise that now, thanks a lot asshole.
• “See release notes for details” – …aaaand you’re making me Google it.

It’s clearly directed at the person whose day sucked because of this issue, and it’s kicking them while they’re down.

Worse still, it wasn’t really without empathy; I had a bad day following the 2014.03 update a year ago, so I know full well what it feels like. I guess I’m still kicking myself for my lack of foresight, and made a frustrated tweet aimed in the wrong direction. Sorry, people of Twitter.

The Constructive Part

Amazon says:

“Beginning with the 2011.09 release of the Amazon Linux AMI, the repository structure is configured to deliver a continuous flow of updates that allow you to roll from one version of the Amazon Linux AMI to the next.
…
In other words, Amazon Linux AMIs are treated as snapshots in time, with a repository and update structure that gives you the latest packages that we have built and pushed into the repository.”

This means, by default, that an instance built using the Amazon Linux 2014.09 AMI will now upgrade packages to those in 2015.03. Some of these upgrades will be harmless, while others will break your stuff. Since computers lack a “don’t break my stuff” option, you either live with that risk or disable these updates.

The way to disable these updates is by pinning a Yum variable called repo_releasever. If you want to do this on a running instance, the file to edit is /etc/yum.conf. You can set this when starting new instances by providing this userdata:

#cloud-config
repo_releasever: 2015.03

See “How do I lock my AMI to a specific version?” in the Amazon Linux FAQs.

The Critical Part

Amazon employs loads of smart people, and I’m sure there was a robust debate when considering the appropriate default to this setting. But since I can’t find a public justification for this decision, I will be blunt: they screwed this one up. It’s not hard to think of ways this default will end up ruining your day, and I struggle to find even tenuous arguments in support of it.

I don’t recall the specifics of my issues with 2014.03 – the default ruby changed from 1.8 to 2.0, and that definitely screwed up the mcollective packages I was using. I also recall that the mysql gem I compiled for use with Chef broke, perhaps because the glibc version changed.

2015.03 took greater care when changing the default python version; running systems updated from 2014.09 would install 2.7 alongside 2.6, but would not change the default. This caused problems for anyone invoking the default python and expecting to interact with the yum repository (as all the system tooling was updated to use 2.7).

The idea of “immutable infrastructure” is something of a fiction at the best of times, but I don’t imagine anyone bakes an AMI expecting it will boot up with a certain set of packages on one day, and a different set of packages on another. Similarly, I don’t imagine anyone expects a CM run on their system to upgrade glibc or language runtimes unless they’ve explicitly configured it to do so.

tl;dr

• The Amazon Linux AMIs ship with a default upgrade policy that thoroughly violates the principle of least surprise.
• This can be easily corrected by setting appropriate userdata (for new instances) or updating yum.conf (for running instances).
• I’ll be linking to this post in August to avoid looking like an asshole again in September.