What's all this Jenkins stuff anyhow? (Part 1)

One of the topics that generates most interest during discussions with customers currently is that of Continuous Integration (CI for short). With it, a single server can offload the development team from much of the time consuming building and testing of everything from individual embedded software modules through to complete projects with testing on a hardware-in-the-loop system (HIL). We're not going into details in this blog post, rather we will give the high-level overview of what a CI does and what you need to consider.

There are several different CI platforms available at the moment, but the ones most often mentioned to us to date have been Jenkins, Hudson (the precursor to Jenkins) and Bamboo. We have been working with and testing Jenkins together with testIDEA and our Python scripting tool isystem.connect, so it will be our focus here. However, the principles, and sometimes even the steps, discussed here apply equally to other CI platforms too.

So, what is a CI platform? Essentially it is a server which is configured to download your source code, build/compile your projects source code and then deploy the resulting binary. In the world of microcontrollers, deployment means simply generating the binary code but it can also include downloading that image into a microcontroller and then testing the code in situ.

Figure 1 - Simplified process flow for embedded developers using Jenkins[1]

 

For this to work, however, there are a couple of things that you will need:

  1. A repository - a CI platform makes use of the fact that the code that belongs to the project is stored in a version control system and that the development team checks in their changes to that repository on a regular basis. Such a repository can be based upon git, subversion or CVS. The repository can be cloud-based or hosted on your own servers.
  2. A server - this can be a cloud-based server (such as Amazon Web Services, who provide a free trial) or a real machine in your office dedicated to CI tasks. If you wish to actually use your CI to test your code on a real microcontroller, you'll need to go down the route of a machine in your office since you will need to physically connect your debugger and microcontroller board to the PC.
  3. Some software - Jenkins can be run on almost any OS, so you'll need Windows or a Linux variant. If you wish to include testing on the target microcontroller, you should probably stick with Windows for now, since testIDEA currently requires Windows. On the same machine you'll also need to install your cross-compiler and it makes sense to use 'make' for building your application code, which is supported for Windows through MinGW.

Once these items are in place, you can install your CI, compilation tools and anything else required for turning your embedded software into a binary file. The basic flow that will be followed thereafter is shown in Figure 1 above, namely:

  1. The development team writes their code and checks it in to your chosen repository. Ideally, the team will upload their changes into the repository on a daily basis.
  2. The CI (in this case Jenkins) is configured to check out a copy of the repository, with all the latest changes, on a regular basis. Typically this is set to be outside of office hours after all changes have been submitted. Equally, the process of checking-in source code changes can also be used to trigger the process.
  3. The CI then uses the rules you have defined to build your application's binary image from the source code it checked out of the repository.
  4. If the build is successful, congratulations - you have success. If not, sorry - you have a failure in your code base.

At the basic level, you now have a tool that checks you can build your embedded software projects every evening and, if any issues snuck into any of the source files, you'll find out the next morning. However, just because you've successfully built the project, doesn't mean the resulting binary will actually work! More on this later…

 

Figure 2 - Jenkin's browser-based interface

So, what does a CI give you to work with? Well, Jenkins offers a browser-based interface for the display of build status output and configuration, looking similar to that in Figure 2. On the left-hand side, there are links to the user interface that allow us to create new projects or configure Jenkins. This is followed by a region that shows what is currently in the build queue and what projects are currently being built. The rest of the interface is given over to the "dashboard" that displays the projects that are configured on the system and their status.

Figure 3 - Individual project status

As can be seen in Figure 3, we have three projects configured on the system. Jenkins marks projects that built successfully (e.g. Jenkins Dashboard View) with a blue dot in the first column. A red dot indicates that the build failed. Grey is used to indicate disabled, pending or aborted builds. Finally, yellow indicates an "unstable" build. This happens, for example, when the project can be built but the test step resulted in some failures.

Another nice touch are the "weather" icons. These show the general trend of the project. A "sunny outlook" indicates all is well with the project, whilst "rain clouds" indicate that some or all of the last few builds had some issues. The following columns follow up with concrete data on the date/time of the last successful build, the last failure and how long each build takes. This last column can be very helpful in understanding how much load your CI platform is under and whether it makes sense to look at expanding it to include more severs to handle it.

So, for this blog post, that's all we have time for unfortunately. If you want to get started yourself, feel free to grab a copy of John Smart's book "Jenkins". Otherwise, don't despair, as Part 2 will be available shortly and will show how projects are configured and what is needed to include testing into the process.

As always, keep building safer embedded systems and Happy BugHunting!

[1] - Bowes, Jesse. "Continuous System Simulation." (2006): n. pag. Bowesjesse.pdf. University of Colorado. Web. 1 Nov. 2016.