Building container images with Fedora, Buildah and GitLab-ci

We recently got a full pipeline for building container images in GitLab-CI using buildah. And since a co-worker asked me just yesterday for how to get it started, here I am, writing!

For this demo we'll build a small container image based on Fedora.

What you get

Make-based build system for containers of any kind (Fedora, Ubuntu, Alpine, Rust, Go)
Continuous integration with CI automatically building containers
Custom Fedora-based root filesystems with only the necessary ingredients, for when your project has larger dependencies
Completely un-privileged container builds (SELinux enforcing, privileged=false, and no capabilities added)
Up-to-date software (and dependencies) in your builds

Container prerequisites

To build a container using buildah inside a docker container with gitlab-ci, one needs to use the vfs storage-driver for buildah and podman.

This could be done by the build.mk include file, but that causes conflicts on systems that have previously used another driver.

You need to specify the driver inside the building container, using /etc/containers/storage.conf.

Below is an example from our container container:

[storage]
# Default Storage Driver
driver = "vfs"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

GitLab-CI runner configuration

Our gitlab-runner configuration look like this:

 [[runners]]
  name = "$RUNNER_NAME"
  url = "https://gitlab.com/ci"
  token = "$RUNNER_TOKEN"
  executor = "docker"
  [runners.docker]
    image = "busybox"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

This is the default configuration for the docker executor in gitlab-ci runner, no customization needed.

Basic project layout

My example container project, uses a small hello.sh shellscript and bundles it into a container imagewith only our required dependencies.

The build process does the following:

Builds our binary
Runs our testsuite
Builds and publishes our container image

The resulting .gitlab-ci.yml lives here, and below we'll cover our steps.

Testing the project

This is the simple hello world of GitLab-CI:

test_hello:
  stage: test
  tags:
    - x86_64
  image: ubuntu:latest
  script:
    - bash -x hello.sh

We name it test_hello (arbitary label), and run it in the test phase of the build. The test is run on our x86_64 runners (not our slower arm runners).

The test is run in the ubuntu:latest container image, and the test script is simply to launch our hello.sh.

Building the project

The mocked up build phase is pretty basic, but has some important parts:

build_binary:
  stage: build
  tags:
    - x86_64
  image: registry.gitlab.com/modioab/base-image/fedora-28/container:master
  script:
    - make project/hello
  artifacts:
      paths:
          - project/hello
      expire_in: 1 week

The build_binary is a simple label, can be anything. stage: build means it's run as part of the build phase.

Our image in this case is a container with all build tools in it, our build image. It's got make, buildah and other things that are handy for building container images.

The script part here is slightly different from the test above, as I'm calling make to generate project/hello from our source files. Replace this with whatever build steps you've got in your own projects. (And with whatever build container you may wish to use.)

The artifacts section declare that we're storing the output, generated project/hello binary for the next phase, and expire_in is just to not clutter the history too much.

The publish phase

The final section of .gitlab-ci.yml is the publish section to build and publish the container image.

container:
  stage: publish
  tags:
      - buildah
      - x86_64
  image: registry.gitlab.com/modioab/base-image/fedora-28/container:master
  before_script:
      - make -f build.mk login
      - podman info
  dependencies:
      - build_binary
  script:
      - make -C project build-publish

The tags section has buildah in it, since we tag our container-capable builders with that. (Configured as per the config above)

The image section still uses our fedora-28/container:master image, with make, buildah, docker and other things inside it.

The before_script uses the login feature of build.mk. This uses the GitLab CI credentials to login to the gitlab container registry using the gitlab-ci variables.

Also note that if your build container doesn't have an /etc/container/storage.conf that points out the storage driver to be vfs, you should add --storage-driver=vfs to podman info.

The dependencies means that the artifact (project/hello) from the build step gets unpacked there.

And the script in turn just calls the build-publish rule inside the project subdirectory.

The Dockerfile

Even though we use buildah we still use Dockerfile rather than a build script, and here is the Dockerfile in its entirerty:

FROM scratch
MAINTAINER "D.S. Ljungmark" <spider@modio.se>

env LANG C.UTF-8
env LC_CTYPE C.utf8

ARG URL=unknown
ARG COMMIT=unknown
ARG BRANCH=unknown
ARG HOST=unknown
ARG DATE=unknown

LABEL "se.modio.ci.url"=$URL "se.modio.ci.branch"=$BRANCH "se.modio.ci.commit"=$COMMIT "se.modio.ci.host"=$HOST "se.modio.ci.date"=$DATE


ADD rootfs.tar /
COPY hello /usr/local/bin/hello
CMD "/usr/local/bin/hello"

A breakdown of the Dockerfile:

FROM scratch means that we start with an empty image
MAINTAINER is metadata for the container image
env LANG and LC_CTYPE sets the container to use utf-8
ARG lines define default incoming arguments
LABEL sets the metadata for this container image. We have standardized on embedding: URL, Commit ID, Branch, Build Host and Build Date
ADD line adds the rootfs.tar to /
COPY line to install our binary hello program to /usr/local/bin
CMD line to set our default command to hello

Makefile magic

The actual container building is done inside the Makefile in the project directory. In total, it looks like this:

IMAGE_REPO = registry.gitlab.com/spindel/example-container-small/example

FEDORA_ROOT_ARCHIVE = rootfs.tar
IMAGE_FILES += $(FEDORA_ROOT_ARCHIVE)
FEDORA_ROOT_PACKAGES = bash
FEDORA_ROOT_PACKAGES += openssh-clients

include ../build.mk

IMAGE_REPO points at where the container image will be published. Builds will always be tagged with the branch they are built from. In this case, the final build will be tagged as registry.gitlab.com/spindel/example-container-small/example:master and can be pulled using the same label.
FEDORA_ROOT_ARCHIVE points at a filename (rootfs.tar) which will be built to contain a fresh root filesystem, that can be added to the container.
IMAGE_FILES is a list of files that need to be built before the container/build phase should happen.
FEDORA_ROOT_PACKAGES is a list of packages to be added to the root filesystem. In this demo, we want /usr/bin/bash and /usr/bin/ssh.

build.mk

The main magic is done by build.mk, a freely available include file for building container images.

It uses git archive to get a proper snapshot of the git tree, and if you add the various make targets to build files, all that's needed to build a container image is to call make build.

For more details, configuration, etc. see the build.mk documentation

Deploying

Due to the online nature of builds, redoing a build will generate a fresh container image with updates from upstream (fedora) while using the same container tag. This can be both a positive or a negative. Evaluate your own needs before you automatically rebuild tags.

Builds are always tagged with the branch name, and if you wish to use kubernetes rolling upgrades, you will want to use tags for the version numbers and deploy using those.

If you wish to deploy a branch without issues, name it “latest” and kubernetes and docker will automatically refresh it when restarting a container.

Conclusion

With this setup, we generate custom container images, either containing just what's needed to run a single program, like postgresql, or more complex setups with both applications and frameworks.

Projects that share traits build a shared base image with it's dependencies in them, and use layers on top of that, while those that are stand-alone get only exactly the dependencies they need.

These containers will have fewer layers than if you build on upstream, and may have a larger shared base between them. They are also continuously updated with upstream packages. This also guarantees that we check signatures of all files installed in our containers, something that is hard or impossible with some upstream container images from the docker hub.

We use a continous build pipeline to ensure that we get all upstream updates, while keeping resulting containers relatively small.