Setting up a simple device CI/CD pipeline for your project: part I

In this series of posts, we propose a simple and full open source CI/CD pipeline for your Linux device development on GitLab that takes advantage of the power of Pantahub and Pantavisor.

Table of Contents (4 part series)
Part I: keep devices up to date
Part II: generate flashable images
Part III: template internals
Part IV: pipeline customization

In the first part, we will start with the basic setup to keep a couple of devices automatically up to date with the latest and stable versions of your project’s code base.

Background

It happened again! Something stopped working and you have to probably spend long hours debugging. Was it the config file? Was it any of the latest fixes on the code landed by your team? An incompatibility with some dependency update? When did it stop working? Did it even work at some point? You could easily revert to the previous working security copy of your system. That could make it. After that, you just have to redo all the changes made by the team since the last security copy and everything will be fine. Wait… what were those changes?

This process could have been less painful if you had used a proper CI/CD pipeline in your device development workflow. One pipeline that would have added all the advantages of entirely working with distributed version control: traceability, change history in every file, automatic backup, branching and so on. It would have added the rest of continuous integration and development perks: shorter development life cycles, automatic building, testing and deployment, early detection of bugs, etc.

What will you need?

This tutorial assumes that you have gone through the prerequisites section in our documentation and you have installed pvr and registered in Pantahub.

As we are taking our gpio management tutorial as a starting point for this one. We are going to need the same material, though this is optional. In our case, we are going to double it so we can have a periodically updated device (latest version) and an on-demand updated device (stable version):

  • 2 Raspberry Pi 3 B+
  • 2 Power cable for RPi3
  • 2 Internet-facing Ethernet cables
  • Your computer
  • Two or more LEDs
  • over 50 ohm resistor for each LED
  • One female to male breadboard jumper wires per LED plus an additional one for GND
  • Breadboard

As we have already said, it is not mandatory to set up the gpio hardware for this tutorial, but it is indeed funnier 🙂

In case you don’t have the opportunity to get your hands on the hardware, or you simply don’t want to, you will just have to substitute the claiming process of each device with these commands:

pvr login
pvr dev create gpio-latest-device
pvr dev create gpio-stable-device

This will create a couple of virtual devices in your Pantahub account. The rest of the tutorial will be the same for you.

Lastly, if you are brave and you don’t have a Pi but you have any other device, you can always try them. Our CI templates are agnostic to platform or architecture.

A simple example device

Our starting point will be a Raspberry Pi device flashed with our Pantavisor stable initial image. To see how to do the whole flashing and claiming process, go to our get started guide to see a step-by-step tutorial.

We are going to fork the code of our gpio app to take control of its source code. Wait for its pipeline to finish and install with pvr the resulting app. We will refer to this pipeline as the container-ci, to differentiate it from the device-ci, which is the one we are going to set up in this tutorial.

As soon as container-ci finishes, we are going to add it to our cloned device with pvr.

Then, we commit and post the changes to Pantahub. You can see if the device got the new update in your device panel.

To complete the preliminary setup, you can check the history and list of installed apps in your Pantahub device.

These are the apps that device-ci is going to keep up to date, getting the latest from that GitLab registry tag when the scheduled job is executed. They are just the default apps in the initial device you just flashed plus the gpio add you just installed.

Additionally, device-ci will keep the bsp (kernel, drivers and Pantavisor binary) up to date with the latest stable version from our repositories.

Now, let’s double it!

We are going to claim the other Raspberry Pi and post the contents of the previously set up device, so we can keep two (initially) identical devices: in the future, one will have the latest stuff from our development, the other one will keep the stable version and will be only updated when tagged.

This is the final pyhsical set up, ready to roll.

Set up the GitLab project

Now, we are going to setup the proper device-ci. For this, we will need a new GitLab project.

There is a number of variables we need to set. For this tutorial we are going to only use PHUSERPHPASS and PERSONAL_ACCESS_TOKEN. Check in the GitLab documentation how to create the last one (you will need api, readuser, readrepository and read_registry permissions). Don’t forget to mask all these variables!

The Pantahub credentials are needed so the jobs can clone and post on your private Pantahub devices.

The GitLab token is going to be used by the jobs to automatically push changes to the repository we just created when any of the installed apps or bsp need an upgrade. This repo will automatically create a folder named recipes that will contain a reference to the Pantahub device and revision. That way, tagging any of the states of the project will be possible to create stable versions on tested commits.

Import our template

Then, we clone the project and add the device-ci templates project as a submodule.

To configure the gitlab-ci.yml, we have added a tool called mkdevice-ci to aid you with the boring stuff. First, we need to init the repository to generate the configuration file.

You have to manually edit that config file to add the information about your your devices. The device list expects a list of pairs of device names. It allows to keep a list of devices with different apps or even devices with different architectures.

In a real life setup, we could want this pair of devices to be different, so we can keep a running latest device and another running stable device per pair, but we are going to use the same device for simplicity sake. Just remember the first device in the pair will be updated on GitLab schedule, while the second one will be updated on manual tag pushing.

Now, use the install option of mkdevice-ci to generate the .gitlab-ci.yml file required by GitLab to run its CI. To complete the device-ci setup, just add, commit and push the changes to your repository.

Test it

In order to test the pipeline, we can push a change to the master branch in our previosly forked gpio app project. For example, we are going to push a commit that makes the LED blinking instead of staying fixed when configured from Pantahub metadata.

We can do that by substituting gpio.sh with the following code:

set_gpio_value() {
    if [ ! -e $GPIO_PATH/gpio$1 ]; then
        log "Exporting GPIO $1 with OUT direction"
        echo $1 > $GPIO_PATH/export
        echo "out" > $GPIO_PATH/gpio$1/direction
    fi  
    if [ $2 = 1 ]; then
        if [ `cat $GPIO_PATH/gpio$1/value` = 1 ]; then
            echo 0 > $GPIO_PATH/gpio$1/value
            log "Switching GPIO $1 value"
        else
            log "Switching GPIO $1 value"
            echo 1 > $GPIO_PATH/gpio$1/value
        fi
    else
        log "Setting GPIO $1 to 0"
        echo 0 > $GPIO_PATH/gpio$1/value
    fi  
}

Wait! we haven’t yet configured the scheduled job. You can do that with the help of GitLab UI. This scheduled job will be in charge of go through your list of configured devices and check for upgrades on the bsp or apps. If any update is found, it will compile a list of changes and automatically post those changes to your devices.

The scheduler can be set as you wish. In this case we are going to update our latest devices daily. In part three of these series of tutorials, we are going to see how to override our templates and run the update job after a change in the app repository is pushed.

You can manually trigger the scheduled job with GitLab UI.

The pipeline is now running. The post job will check for updates in your device, while the push job will compile a list of changes and push changes to the device recipes, a list of files that contains the name and revision of the devices at each point.

The pipeline has found some changes! You can check those changes in the message of the commit that was just pushed.

After every automatic or manual commit push, a validation job will check if the device status is OK after the new revision is pushed to Pantahub.

In our little example, we can visually see stable at the left, with the old gpio app version. At the right we have the latest version with the blinking LEDs.

What’s next?

In the next part of these series, we are going to take a look at how to generate flashable images with the status of your device, so you can get factory images to install your project in your devices