NUbots uses Buildkite for its continuous integration pipeline. The Buildkite server listens for code changes on GitHub and issues build jobs to our build agents (the iMacs in the lab), as they become available.
This CI process works in combination with the Git Feature Branch workflow we use for development:
Developers branch off
main, creating a new branch with a name of the form
<lastname>/<purpose-of-branch>. For example:git checkout -b biddulph/ball-detector
Once development on the branch is complete, or reaches a point where it can be reviewed, the developer makes a pull request (PR) on GitHub. This triggers a webhook to Buildkite to build and test the PR.
Buildkite receives the webhook and creates separate build jobs for different parts of the PR (build, check code format, test, etc) and sends different jobs to different agents based on their availability.
Agents receive a job, clone the code, pull the existing
mainimage for DockerHub, and build any new Docker changes. They then mount the code and run the job in a container based on the image, reporting the results back to Buildkite.
Buildkite receives the results and updates the build status in the Buildkite UI as well as on the PR page on GitHub.
When the build passes, the PR is reviewed by another member of the team. If approved, the PR can be merged into
mainthen triggers another CI build. This time, in addition to building the code, a separate job is created to rebuild the main image and push it to DockerHub. This job is handled by agents that have the DockerHub login credentials, indicated by the
dockerhub=truetag on the agent.
The build pipeline is configured in the
.buildkite/pipeline.yml file in the NUbots repo. This file specifies a number of "build steps", which are separate jobs such as "Build for NUC" and "Validate C++ and Protobuf formatting".
To add, remove, or change build jobs, we edit the pipeline file. The Buildkite pipeline documentation has everything you need to know to make changes.
Buildkite has live-updating logs of each build, which you can use to monitor the build progress or troubleshoot errors when a build fails.
If for example your PR build fails due to formatting errors, you can check the logs to see which files are incorrectly formatted. To fix the build, you would reformat the offending files and push a new commit, which will trigger a rebuild.
To view the logs for a build, scroll to the build checks at the bottom of the PR page and click the Details link on the
When Buildkite receives a webhook to build and test a PR, the Buildkite agents pull down the code and start building a Docker image based off the Dockerfile in that branch.
In the process of building our Docker image, several pacman packages are downloaded from public mirrors (if that specific layer itself is not cached). Because these packages are downloaded numerous times, across multiple PRs, caching them locally can speed up build times and save network bandwidth.
It is also possible that the signature on a package that is located on a public mirror changes. This change of signature can cause issues for us because we download date-locked packages and still expect an older signature. Caching such packages locally may alleviate these issues.
In order for pacman to use this package cache, the cache must be added to
/etc/pacman.d/mirrorlist. Because our build happens inside a buildx container, we cannot use
localhost as the address for the cache. Instead, we alias the IP address of the host machine as
host.docker.internal. Therefore, the following is added to the mirrorlist here in the Dockerfile:
Server = http://host.docker.internal:7878/$repo/os/$arch
The following instructions show how to setup and configure a new Buildkite agent with Docker and Docker Compose. These steps have been collected into a single script you can download here. Before running the script, ensure you have set the Buildkite agent token and DockerHub password correctly in the env variables at the top of the script. Also, make sure to run the script with
# Remove old Dockersudo apt-get remove docker docker-engine docker.io containerd runc
# Install packages to allow apt to use a repository over HTTPSsudo apt-get updatesudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
# Add Docker's official GPG keycurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# Add Docker's stable repositorysudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"# Install latest Docker CEsudo apt-get updatesudo apt-get install docker-ce docker-ce-cli containerd.io
# Install Python 3 and pipsudo apt-get updatesudo apt-get install python3.6 python3-pip
# Install Docker Composesudo -H pip3 install docker-compose
Steps from https://buildkite.com/docs/agent/v3/ubuntu.
# Add the signed Buildkite apt repositoryecho "deb https://apt.buildkite.com/buildkite-agent stable main" | sudo tee /etc/apt/sources.list.d/buildkite-agent.listsudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 32A37959C2FA5C3C99EFBC32A79206696452D198
# Install the Buildkite agentsudo apt-get updatesudo apt-get install -y buildkite-agent
# Add the agent token to the configuration file.# Replace INSERT-YOUR-AGENT-TOKEN-HERE with the token from Buildkite.sudo sed -i "s/xxx/INSERT-YOUR-AGENT-TOKEN-HERE/g" /etc/buildkite-agent/buildkite-agent.cfg
# Start the agent and configure it to run on system startupsudo systemctl enable buildkite-agent && sudo systemctl start buildkite-agent
# Create a 'docker' user group and add local users to allow for use without sudosudo groupadd docker # Create user groupsudo usermod -aG docker nubots # Add nubots to the groupsudo usermod -aG docker buildkite-agent # Add buildkite-agent to the groupnewgrp docker # Activate group changes
# Configure Docker to run on system startupsudo systemctl enable docker
For agents to push built
main images to DockerHub, they need the service account login credentials. The username of this account is
nubotsdocker, and is specified in the Buildkite pipeline file. The password needs to be made available to the agent when it runs, via the
DOCKER_LOGIN_PASSWORD env variable.
Credentials are provided to the agent via env variables using agent hooks. Specifically using the environment hook. The default path for this hook is
After the credentials are added, the agent needs to be marked as having the credentials by setting the
dockerhub=true tag in the agent configuration file. The default path for this file is
For more information about agent configuration and hooks, see the following pages in the Buildkite documentation:
Create a directory to store the cache configmkdir ~/pacman-cache && cd ~/pacman-cache
Download the install scriptwget https://raw.githubusercontent.com/NUbots/NUbots/main/doc/PacmanCache/install.sh
Make the install script executablechmod +x ./install.sh
Run the install script./install.sh