Local debugging for GitLab CI Runners

Our production containers are built FROM scratch with a minimal fedora rootfs, and our applications layered on top. Normally the GitLab CI cluster builds everything, using docker build containers. Especially during early development, or debugging the build process itself, it can be hindering to wait for the push-build-retry cycle. It would be beneficial to run the same build process locally on a developers workstation, directly from their repo clone.

Enter the deprecated gitlab-runner exec docker, while waiting for its future replacement.

TL;DR

The files needed to reproduce the results, and build further, can be found in gitlab.com/notCalle/example-local-runner.

Preparation

First, make sure you have Docker installed and running on your workstation.

To install the gitlab-runner, you can find general installation guidance in GitLab documentation.

For macOS, the gitlab-runner package is also available in Homebrew.

Running with the thing

To explore the requirements for running a local gitlab-runner, we'll use a trivial project with .gitlab-ci.yml containing two stages, of one job each:

stages:
  - build
  - test

build:
  stage: build
  image: alpine
  script:
    - touch the.thing
  artifacts:
    paths:
      - the.thing

test:
  stage: test
  image: alpine
  dependencies:
    - build
  script:
    - ls -l the.thing

Because gitlab-runner output contains the $ shell prompt before commands it runs, we'll use the % prompt for commands entered on the command line. We'll also use … to denote that output was cut from examples.

The first limitation of the exec runner is that it does not handle pipeline stages, so manual work is required to resolve job dependency order, and execution.

First we run the build job:

% gitlab-runner exec docker build
…
$ touch the.thing
Job succeeded

After the build job has completed successfully we can continue with the test job:

% gitlab-runner exec docker test
…
$ ls -l the.thing
ls: the.thing: No such file or directory
ERROR: Job failed: exit code 1
FATAL: exit code 1

Oh no, it turns out that passing of artifacts is not implemented. Let's try using the cache instead, making sure that both jobs have the same parameters. We'll use some YAML trickery to ensure this; the build job has a complete cache definition with an anchor, and the test job has a reference to the anchor.

stages:
  - build
  - test

build:
  stage: build
  image: alpine
  script:
    - touch the.thing
  cache: &build_artifact_cache
    key: build
    paths:
      - the.thing

test:
  stage: test
  image: alpine
  cache: *build_artifact_cache
  script:
    - ls -l the.thing

Let's give that a try, by first running the job in the build stage:

% gitlab-runner exec docker build
…
Successfully extracted cache
$ touch the.thing
Creating cache build...
the.thing: found 1 matching files
Created cache
Job succeeded

So far, so good. Let's continue with the job in the test stage:

% gitlab-runner exec docker test
…
Successfully extracted cache
ls: the.thing: No such file or directory
$ ls -l the.thing
ERROR: Job failed: exit code 1
FATAL: exit code 1

It turns out that the cache, by default, is ephemeral, so our second run gets an empty cache. Let's try mounting a docker volume for the cache. Create a directory to hold the cache volume:

% mkdir cache

Mount the cache volume inside the CI container for the build job:

% gitlab-runner exec docker --docker-volumes `pwd`/cache:/cache build
…
Successfully extracted cache
$ touch the.thing
Creating cache build...
the.thing: found 1 matching files
Created cache
Job succeeded

Also mount the same cache volume when running the test job:

% gitlab-runner exec docker --docker-volumes `pwd`/cache:/cache test
…
Successfully extracted cache
$ ls -l the.thing
-rw-r--r--    1 root     root             0 Jul 24 09:50 the.thing
Creating cache build...
the.thing: found 1 matching files
Created cache
Job succeeded

Success!

The cache is actually stored as a zip file, so the artifacts can easily be extracted for further local testing, such as loading a docker-layer.tar artifact into the local docker registry to run tests inside the container.

% tree cache
cache
└── project-0
    └── build
        └── cache.zip

2 directories, 1 file
% unzip -l cache/project-0/build/cache.zip
Archive:  cache/project-0/build/cache.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  07-24-2018 11:50   the.thing
---------                     -------
        0                     1 file

Make it tidy

Repeatedly typing and remembering long command lines is boring, so we'll improve our life a bit further by adding a Makefile to the project:

.PHONY: clean

.DEFAULT: cache
    gitlab-runner exec docker --docker-volumes `pwd`/cache:/cache $@

cache:
    mkdir $@

clean:
    rm -rf cache

With a .DEFAULT make trick, every job in the gitlab-ci file becomes a valid make target, so we can simply type them in order as arguments to make, prepended with a clean, to ensure no old cache artifacts are present.

% make clean build test
rm -rf cache
gitlab-runner exec docker --docker-volumes `pwd`/cache:/cache build
…
$ touch the.thing
Creating cache build...
the.thing: found 1 matching files
Created cache
Job succeeded
gitlab-runner exec docker --docker-volumes `pwd`/cache:/cache test
…
$ ls -l the.thing
-rw-r--r--    1 root     root             0 Jul 24 10:29 the.thing
Creating cache build...
the.thing: found 1 matching files
Created cache
Job succeeded

Summary

As we have seen here, the local gitlab-runner is perfectly viable for debugging CI build jobs, but job dependencies require careful consideration, as the local runner only runs a single job at a time. For more complex builds you will probably want to create separate jobs for local and pipeline builds, and pipeline artifact dependency related failures can't be debugged locally.