Setting up a CICD pipeline for embedded Linux device development

In this series of posts, we propose a simple and full open source CI/CD pipeline for your Linux device development on GitLab that takes advantage of the power of Pantahub and Pantavisor.

Table of Contents (4 part series)
Part I: keep devices up to date
Part II: generate flashable images
Part III: template internals
Part IV: pipeline customization

In the previous post, we added the needed configuration to generate and store flashable images with the stable version of your project to the pipeline.

It the third part, we are going to take a look at the internals of our CI in order to understand how it works.


If you have read part I and part II of this series, you may have wonder why did we set up these templates. The answer is we started this CI process for our initial devices. This pipeline keeps a number of initial devices to help user get started with Pantavisor development up to date in a daily basis. That way, we can integrate to master at the different apps and bsp that form those devices and automatically detect at the end of the day if any of the landed merge requests breaks the baseline boot up for the different devices. It has also helped us to track and revert other failing commits even with a modest set of tests. Furthermore, we automatically publish the initial images built by the CI for anyone who wants to flash their devices and start their projects from there.

At the beginning, everything was in .gitlab-ci.yml inside of our project, but as other projects came up, we saw the opportunity to reuse the code and thus we move the common script to a template that can be included into any GitLab project. The motivation to start a new CI project that use this template is always the same: keep a group of related devices up to date, which means keep their bsp and apps (containers) up to date.

In the end, the CI we have proposed in these blog post is just a relatively simple script that is run by GitLab CI in a GitLab runner. The strength of it comes from the use of Pantahub and Pantavisor. The GitLab runner will do the actual update and build, but the results are published to Pantahub, and remotely updated to a number of devices with Pantavisor.

It is time now to explain the template internals so you can discover first hand (if you haven’t already) the simplicity of it.

Explaining our CI template

Our template is hosted here. We are going to go through its architecture, state machine and tools in this section.


This diagram shows the architecture of the CI and how the scripts that are run in a GitLab runner sits in relation to the device and the cloud.

At the left, we have a number of devices that we want up to date. At the center, the .gitlab-ci.yml with a list of devices (pairs of latest and stable devices, generated with the mkdevice-ci script) in the repo and GitLab CI variables. At the right, Pantahub receives the updates and is in charge of storing and updating each Pantavisor device state in turn.

The logic of the CI is therefore contained inside the code repository (included from the templates) and can be overridden following the rules of GitLab CI. This logic is then executed in a GitLab runner, which can be either the GitLab shared ones or one of your own.

The storage of the binaries is managed by Pantahub and run in a Pantavisor-enabled device, while the storage of the images is done in AWS, although this last part optional.


There are five stages and they can be divided in two groups that are run sequentially: scheduled stages and pushed stages.

Scheduled stages

The scheduled jobs are executed either by the GitLab scheduler or by the user via GitLab user interface. They are two: post and push.

  • Post: checks for bsp and apps updates in the list of latest devices. If positive, it updates them by posting to Pantahub, which in turn updates the Pantavisor devices.
  • Push: gathers all update information and advances the state of the device in the repository. This is done by committing to the repository the device revision that was updated in the last stage into the recipes folder. This is important because each commit in the repo represents a state of the devices.

Pushed jobs

The pushed jobs are executed when either a commit or a tag is pushed to the repository. They are three jobs: build, validate and deploy.

  • Build: Only triggered when pushing a new tag. It clones the device from Pantahub using the reference in the recipes folder and posts it to the stable device. Using that reference, it builds the flashable image and uploads it to AWS.
  • Validate: Checks the device status is DONE and fails otherwise. It tests either latest device or stable device depending on how was the job triggered, so it will check the most recently updated one.
  • Deploy: Only triggered when pushing a new tag. Uploads the tag metadata collected from the previous stages to the stable.json file at AWS.


The main script we used in part I to set up the device-ci project was the mkdevice-ci script. As of today, it supports two commands: init and install.

With the init command, you generate the default update-ci.yml configuration file. This file contains the list of device pairs.

The install command takes the information from update-ci.yml and automatically generates a hidden file named .gitlab-ci.yml. This one is important because it’s what GitLab CI takes to start the jobs. This is the file that you have to edit if you want to perform an override on the template, as we are going to see in the next post.

What’s next?

The original idea was to show how to override the pipeline to make is customizable, but we are going to do that in a further part because this post was getting considerably big.

In the not planned next part of the series, we are going to use the knowledge gained in this post to show how to customize our template to fit your needs.