Embedded Distribution

Distribution agnostic

Our current distribution is based on Debian Jessie. We have an atomic updater based on deltas of filesystems between versions, which allow us to change any part of the system in a safe and validated manner. We have built custom tooling for generating filesystem images, generating deltas from images and for upgrading embedded systems atomically, that does not depend on the source of the operating system packages.

The distribution has evolved through time, it has changed from OpenWRT to Angstrom and then to Debian. All have been seamlessly updated without user intervention.

The software stack is based on Gnome with Epiphany running in kiosk mode for our displays. On the lower levels we prefer Python because of its expressiveness and maintainability. Sadly, we have not been able to migrate to Wayland yet, but tests are progressing.

Our hardware runs without any closed-source binary firmwares or kernel drivers.

The updater

Our updater is based on a double-firmware setup, with an active and a to-be active image. The filesystems are read-only on the block level, and we distribute delta upgrades between versions. Each firmware is then validated before toggling the boot loader. Both the firmware and the upgrade are securely validated. Variable data is kept on a separate device, and can be safely discarded in case of inconsistency during boot.

The future

Our solution uses a completely stateless root filesystem with a separate partition for variable data, similar to how Project Atomic and OSTree (From the Eminent Colin Waters) works. We are currently investigating the possibilities of rebasing our distribution on top of CentOS for ARM or Fedora 22 with ostree-rpm as the base, but keeping our delta image upgrades. The difference to the end user would be unnoticeable, except for the larger than average download for that upgrade.

Philosophy

We don't really want to be in the business of maintaining a distribution, but rather ended up here because of a void in the embedded space. This is why we try not to modify any upstream packages. Sometimes modifications are necessary, and we have a policy of always attempting to upstream our changes. Custom forks are generally painful, and a business should be concerned about forks of free software.

What we want from an upstream

When we evaluate and work with distributions, we look at several things: open source community, the activity of mailing lists and policy about upstreaming software changes. We also care about licensing, security and maintenance. If a project lacks a security mindset, and does not publish CVE's or other security related updates, it's generally not a project that is safe to build upon. This goes both for individual packages, and a distribution as a whole.

Being a good citizen

Being part of a community is important. Sometimes, business needs require that you go off on a tangent and build something no one else needs. Then you might also find yourself sidelined when the community decides on a different direction, and ploughs ahead at 120km/s. Lone developers can be more productive in the short run, but in the end it's always a winning scenario to work with a community. Even if you don't always get what you want, you come out ahead.

In short, even though we built our own tooling for generating atomic firmware images, we like to have company along on a project like this, and are looking to align ourselves with community supported methods. It is generally a better way of doing development.