Building test into your development process

In our earlier post we looked at how the intrepid team at "Hudson Technologies" tackled a transient software bug in an oven control system. Taken from Lisa Simone's book "If I only changed the software, why is the phone on fire?", Josie and Li Mei are confronted in chapter 4 with an oven that is burning moulded parts instead of warming them. The cause of the problem is found to be in the software that calculates the new PWM value needed to control the heater. The calculated value depends on the current oven temperature and a reference temperature measurement.

The cause of the bug was due to an unsigned variable being used to store a potentially negative signed value, and the previous blog post showed that, had the developer had an IDE like winIDEA, they would have seen that some of the code they had written could never be reached by the logic they had written. In this post, we are going to take a look at this application once more and see how the developer could have written tests for his code as he was developing it that would have highlighted the failure before the product was released to the customer.

Design principle of oven control system based upon Arduino DUE; the Arduino M0 Pro was actual used whilst preparing this blog entry

As hardware for this example we are going to use the Arduino M0 Pro, with its on-board CMSIS-DAP debugger, coupled with winIDEA Open. With this combination of tools you will be able to repeat everything you see here. Additionally we have uploaded the project to bitbucket so that you can download it too.

We simulate the entire application using the Arduino platform as follows:

  • The LED on pin 13 is flashed to indicate the code is executing.
  • Analogue input A0 is the input from the oven temperature sensor.
  • Analogue input A1 is the input from the reference temperature sensor.
  • The PWM output to control the heater is connected to pin 2.

For the purposes of this blog, we are not really interested in the actual analogue input values nor the PWM control; they are simply implemented to show the principle. Our main focus is getting the code in place to be able to check that the code calculating the PWM value for the oven works properly.

In the original code, some of the required variables were defined as global variables. In order to make the code easier to test, and make the PWM calculation code reusable, we rewrote it as a separate function using values passed to it. As such, the resulting code looks like this:

unsigned char calculate_new_oven_ON_time(unsigned char actual_temp, unsigned char ref_temp)
{
  signed char delta_temp;
  unsigned char newPwmValue;

  delta_temp = actual_temp - ref_temp;

  newPwmValue = NOMINAL_TEMP_PW + (GAIN * delta_temp);

  if (newPwmValue > 100)
  newPwmValue = 100;

  if (newPwmValue < 0)
  newPwmValue = 0;

  return newPwmValue;
}

The function now returns the required PWM value based upon the values measured at the reference and oven temperature sensors. Since Joise and Li Mei had the source code to hand, they were also able to build a table of expected oven PWM return values (see page 99 of the book for more information). In order to test this function works as expected, we will need to test a cross-section of these values, so we have reconstructed part of the table here:

Oven Temp
ADC
Reference Temp
ADC
Corresponding
Temperature °C
Calculated
PWM Value
111 70 -16 100
87 70 24 62
71 70 51 19
60 70 69 0
52 70 82 0
46 70 92 0

Now, we could laboriously hard program these values into the code and see what results come out. Or, within the debugger, we could set a breakpoint at the start of the calculate_new_oven_ON_time() function and modify the input values in SRAM, checking the outcome at the end of the function. That does all seem like rather a lot of hard work. It would be better if we could define some tests that could be automatically executed to prove the functionality this function after every time it is changed.

Well, it just so happens that winIDEA includes such a testing tool, named testIDEA, that is available directly from the menu bar. Since we already have a table of potential inputs and expected outputs, we can simply plug-in these values and store these tests so that they can be executed every time we want to make sure we haven't broken our code.

testIDEA is integrated into winIDEA and available from the menu bar.

Even our free winIDEA Open IDE includes a standard version of testIDEA that can execute basic tests on the Arduino M0 Pro over the CMSIS-DAP debug interface. If you have downloaded the project from Bitbucket you'll be able to open the "Oven Controller.iyaml" file within testIDEA and execute the tests for yourself. You'll see that the tests for the first three inputs used in the above table pass with flying colours, but the last three tests fail (the entries expecting a PWM result of 0).

Broken code test results

Simply swap the commented out line of code in the source code (lines 73/74), rebuild, download and then run the testIDEA tests again. And, if all has gone to plan, you'll be able to see that all the tests have passed. There are also pre-built versions of the binaries available too. Simply deselect one and select the other in winIDEA from the menu under Debug->Files For Download…

After the code has been fixed, all tests pass with flying colours!

So, even with a low-cost development board like a Cortex-M based Arduino and its free debugging interface, there is no need for poor quality, untested code. Give it a test run and let us know how you get on!

And, in the mean time, happy BugHunting!