We recently got a full pipeline for building container images in GitLab-CI using buildah. And since a co-worker asked me just yesterday for how to get it started, here I am, writing!
For this demo we'll build a small container image based on Fedora.
What you get
- Make-based build system for containers of any kind (Fedora, Ubuntu, Alpine, Rust, Go)
- Continuous integration with CI automatically building containers
- Custom Fedora-based root filesystems with only the necessary ingredients, for when your project has larger dependencies
- Completely un-privileged container builds (SELinux enforcing, privileged=false, and no capabilities added)
- Up-to-date software (and dependencies) in your builds
To build a container using
buildah inside a
docker container with
gitlab-ci, one needs to use the
vfs storage-driver for
This could be done by the
build.mk include file, but that causes conflicts on
systems that have previously used another driver.
You need to specify the driver inside the building container, using
Below is an example from our
[storage]# Default Storage Driver driver = "vfs" # Temporary storage location runroot = "/var/run/containers/storage" # Primary Read/Write location of container storage graphroot = "/var/lib/containers/storage"
GitLab-CI runner configuration
Our gitlab-runner configuration look like this:
[[runners]]name = "$RUNNER_NAME" url = "https://gitlab.com/ci" token = "$RUNNER_TOKEN" executor = "docker" [runners.docker] image = "busybox" privileged = false disable_cache = false volumes = ["/cache"] shm_size = 0 [runners.cache]
This is the default configuration for the
docker executor in gitlab-ci
runner, no customization needed.
Basic project layout
My example container project,
uses a small
hello.sh shellscript and bundles it into a container imagewith
only our required dependencies.
The build process does the following:
- Builds our binary
- Runs our testsuite
- Builds and publishes our container image
The resulting .gitlab-ci.yml lives here, and below we'll cover our steps.
Testing the project
This is the simple
hello world of GitLab-CI:
test_hello:stage: test tags: - x86_64 image: ubuntu:latest script: - bash -x hello.sh
We name it
test_hello (arbitary label), and run it in the
test phase of the
build. The test is run on our
x86_64 runners (not our slower
The test is run in the
ubuntu:latest container image, and the test script is
simply to launch our
Building the project
The mocked up
build phase is pretty basic, but has some important parts:
build_binary:stage: build tags: - x86_64 image: registry.gitlab.com/modioab/base-image/fedora-28/container:master script: - make project/hello artifacts: paths: - project/hello expire_in: 1 week
build_binary is a simple label, can be anything.
stage: build means
it's run as part of the build phase.
image in this case is a container with all build tools in it, our build
image. It's got
buildah and other things that are handy for building
script part here is slightly different from the
test above, as I'm
make to generate
project/hello from our source files. Replace this
with whatever build steps you've got in your own projects. (And with whatever
build container you may wish to use.)
artifacts section declare that we're storing the output, generated
project/hello binary for the next phase, and
expire_in is just to not
clutter the history too much.
The publish phase
The final section of
.gitlab-ci.yml is the publish section to build and
publish the container image.
container:stage: publish tags: - buildah - x86_64 image: registry.gitlab.com/modioab/base-image/fedora-28/container:master before_script: - make -f build.mk login - podman info dependencies: - build_binary script: - make -C project build-publish
tags section has
buildah in it, since we tag our container-capable
builders with that. (Configured as per the config above)
image section still uses our
fedora-28/container:master image, with
docker and other things inside it.
before_script uses the
login feature of
build.mk. This uses the
GitLab CI credentials to login to the gitlab container registry using
the gitlab-ci variables.
Also note that if your build container doesn't have an
/etc/container/storage.conf that points out the storage driver to be
you should add
dependencies means that the artifact (
project/hello) from the
step gets unpacked there.
And the script in turn just calls the
build-publish rule inside the
Even though we use
buildah we still use
Dockerfile rather than a build
script, and here is the
Dockerfile in its entirerty:
FROM scratchMAINTAINER "D.S. Ljungmark" <email@example.com> env LANG C.UTF-8 env LC_CTYPE C.utf8 ARG URL=unknown ARG COMMIT=unknown ARG BRANCH=unknown ARG HOST=unknown ARG DATE=unknown LABEL "se.modio.ci.url"=$URL "se.modio.ci.branch"=$BRANCH "se.modio.ci.commit"=$COMMIT "se.modio.ci.host"=$HOST "se.modio.ci.date"=$DATE ADD rootfs.tar / COPY hello /usr/local/bin/hello CMD "/usr/local/bin/hello"
A breakdown of the Dockerfile:
FROM scratchmeans that we start with an empty image
MAINTAINERis metadata for the container image
LC_CTYPEsets the container to use
ARGlines define default incoming arguments
LABELsets the metadata for this container image. We have standardized on embedding: URL, Commit ID, Branch, Build Host and Build Date
ADDline adds the
COPYline to install our binary
helloprogram to /usr/local/bin
CMDline to set our default command to
The actual container building is done inside the
project directory. In total, it looks like this:
IMAGE_REPO = registry.gitlab.com/spindel/example-container-small/exampleFEDORA_ROOT_ARCHIVE = rootfs.tar IMAGE_FILES += $(FEDORA_ROOT_ARCHIVE) FEDORA_ROOT_PACKAGES = bash FEDORA_ROOT_PACKAGES += openssh-clients include ../build.mk
IMAGE_REPOpoints at where the container image will be published. Builds will always be tagged with the branch they are built from. In this case, the final build will be tagged as
registry.gitlab.com/spindel/example-container-small/example:masterand can be pulled using the same label.
FEDORA_ROOT_ARCHIVEpoints at a filename (
rootfs.tar) which will be built to contain a fresh root filesystem, that can be added to the container.
IMAGE_FILESis a list of files that need to be built before the container/build phase should happen.
FEDORA_ROOT_PACKAGESis a list of packages to be added to the root filesystem. In this demo, we want
The main magic is done by build.mk, a freely available include file for building container images.
git archive to get a
proper snapshot of the git tree, and if you
add the various
make targets to build files, all that's needed to build a
container image is to call
For more details, configuration, etc. see the build.mk documentation
Due to the online nature of builds, redoing a build will generate a fresh container image with updates from upstream (fedora) while using the same container tag. This can be both a positive or a negative. Evaluate your own needs before you automatically rebuild tags.
Builds are always tagged with the branch name, and if you wish to use kubernetes rolling upgrades, you will want to use tags for the version numbers and deploy using those.
If you wish to deploy a branch without issues, name it “latest” and kubernetes and docker will automatically refresh it when restarting a container.
With this setup, we generate custom container images, either containing just
what's needed to run a single program, like
postgresql, or more complex
setups with both applications and frameworks.
Projects that share traits build a shared
base image with it's dependencies
in them, and use layers on top of that, while those that are stand-alone get
only exactly the dependencies they need.
These containers will have fewer layers than if you build on upstream, and may have a larger shared base between them. They are also continuously updated with upstream packages. This also guarantees that we check signatures of all files installed in our containers, something that is hard or impossible with some upstream container images from the docker hub.
We use a continous build pipeline to ensure that we get all upstream updates, while keeping resulting containers relatively small.