Coverage Concepts

Analyzer ››
Parent Previous Next

1Introduction

Code coverage is a measurement used in software testing. It describes the degree to which the source code of a program has been tested.

There are two types of code coverage:

Statement coverage – Marks statements or instructions as executed or not executed.

Decision coverage – Mark conditions of conditional statements or instructions as true, false or both (true and false).

Code coverage helps you find areas of a program not exercised by a set of test cases, create additional test cases to increase coverage and identify redundant test cases that do not increase coverage. Code coverage is an indirect quantitative measure of test quality.

Code coverage is helpful to find code that is never executed. This “dead code” represents undesired overhead when assigning memory resources and is also a potential hazard since this code was not executed during coverage test.

Code coverage measurement is a requirement for certification according to standard DO-178B/C SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION. Other standards regulating life critical applications point directly or indirectly to code coverage to improve quality, two among these are ISO13485:2003 (Medical) and SC45A (Nuclear). If developing a device that will be used in aerospace, defense, medical, nuclear, railway signaling, mining, oil, safety, etc. chances are that measuring code coverage is required.

2Basics

iSYSTEM tools measures code coverage by recording all addresses being executed. Results are displayed by correlating object code to source code.

In iSYSTEM nomenclature code coverage is called execution coverage in contrast to data or access coverage.

iSYSTEM tools can be used to comply with different standards. It has already been used successfully by many customers in the aerospace market (DO-178B, for certification to level C, B and A), by medical device manufacturers and widely in the automotive industry.

It is possible to access all code coverage features of iSYSTEM tools using several programming languages: C++, C#, Python, Perl, TCL, Visual Basic, VBScript, and others. For more information refer to the iSYSTEM.Connect documentation and the iSYSTEM winIDEA automation interface documentation.

iSYSTEM tools have two different operation modes for acquiring code coverage information: off-line and real-time.

Off-line execution coverage

The program execution is recorded and then analyzed. This off-line analysis allows great flexibility in the amount of information that can be extracted. In addition to statement coverage, information about the way branches were followed is also extracted to obtain object level decision coverage.

Off-line execution coverage is limited to the amount of data that can be recorded by the tool used.  In order to extend this limit, higher end tools have more memory and the top of the range tools even have the possibility to start uploading data to the host PC and analyzing it on the fly while still sampling, extending the time that an off-line coverage session can run. In ideal conditions, when the system is well balanced, off-line coverage can run indefinitely.

The amount of time a program can run while coverage is measured depends on the target CPU speed and the trace buffer size. Upload time to the PC depends on the trace buffer size and the tool used. In case of iTRACE GT or Active GT development platform with 1GB or more trace buffer, it is recommended to use iC3000 GT unit which offers a high-speed trace upload to the PC. iC5000 units have a fixed amount of memory and stream to the host PC automatically. Off-line execution coverage trace buffer size is configurable in the ‘Hardware/Analyzer Setup’ dialog (1, 10 or 100%).

Off-line execution coverage analysis produces the following information:

Statement Coverage: each processor instruction is marked as executed or not executed. By correlating this information upwards to the source code, executed and not executed lines of source code are identified.

Decision Coverage: conditional branch instructions are additionally analyzed to identify if the branch was taken or not each time the instruction was executed. This produces object code level decision coverage (instruction condition evaluates to true, false or both (true and false). By cumulatively merging the information for each branch four different states are possible:

the branch was never executed,

branch condition evaluated to true at least once,

branch condition evaluated to false at least once,

branch condition evaluated to both true and false at least once.


Off-line execution coverage has its advantages and disadvantages:

It can be implemented on all development systems featuring program trace. It features also object code level decision coverage, not available on real-time execution coverage.

The execution time of the off-line execution coverage is limited by the trace buffer size and this can be too restrictive if low end tools are used. By using better tools this limit rises and even disappears with upload-while-sampling when the system is well balanced.

Off-line execution coverage is the preferred method for unit-tests. The time limitation imposed by the hardware is seldom reached and more information is available.

Real-time Execution Coverage

Real-time execution coverage is based on hardware logic on the tool (debugger/emulator), which in real-time monitors all executed program addresses and updates internal records (Tag RAM).

Even when the trace buffer is filled to capacity the coverage information can still be gathered. A major advantage of the real-time execution coverage is that it can run indefinitely, which does not apply to off-line coverage. On the downside, because no history information is available while operating in this mode, branch coverage is not available.

Real-time execution coverage is available for all in-circuit emulation systems with bus trace (including V850ES/Fx3 ActivePRO POD, which has a different trace port type) and on-chip debug emulation systems with trace port, where iSYSTEM's proprietary RTR (Real-time reconstruction) technology is implemented (MPC55xx except for MPC551x, MPC56x, ARM ETM).

3Coverage in practice

iSYSTEM tools provide accurate information on what memory addresses where fetched and executed by the microprocessor. iSYSTEM tools also provide unequivocal information on the object level, showing the coverage information for the assembler program that actually executed.

Source code coverage information is derived from these two. Correlation has many ways of complicating the resulting source code coverage information. This isn't surprising as source code is not what is actually running on the processor. It is important to keep this in mind and not fall into simplistic interpretations: starting from the source code, there is a chain of transformations that produce the object code that will run in the actual target.

For C programs usually these steps are: preprocessing, compilation, optimization, assembly, linking and conversion. Details vary between systems (compilers, assemblers, linkers, converters). These steps are not bijective: information is added and removed in every step. For example, linking generates end addresses for variables and throws away symbolic names.

iSYSTEM tools are very powerful in that they enable the user to fully understand the actual code coverage situation. There are no blind spots where untested code could be hiding. To harness this power, the following points should be understood to interpret source code coverage information measured by iSYSTEM tools:

1) The object code may not correspond to the object code as seen from the perspective of a source level programmer. Take this example:


1 int i;
2 for (i = 0; i < 10; ++i)
3 {
4   somefunction(i);
5 }


line 2 is typically (no matter the processor architecture used) split in two or three blocks: one to initialize the loop, another to check the condition and increment the loop variable, and a tail to cleanup everything after it exits. These two or three blocks are also typically located in non-contiguous memory locations. Further, you have one or more conditions that apply to different blocks. In the end, iSYSTEM tools will show the information for all three blocks merged for the same line. That is why source code lines could be only partially covered (some of it executed, some not).

This is how it's shown using iSYSTEM tools:



Notice how on the assembler window the for statement is split into two nonconsecutive blocks.

2) Complex conditions. For example:

1 if (a == 3 || b ==4)
2 {
3   somefunction(c);
4 }


line 1 of the example can be only partially tested, even if it is reached from the program. These situations are shown by default by iSYSTEM tools with the greatest possible amount of the detail. If this condition is reached during execution and the evaluation short-circuit logic applies, when later inspecting the source code coverage information, the line will show as not executed. If you drill down on the assembler information for this line you will see that only part of the code corresponding to this source line was executed.

How this is shown when using iSYSTEM tools:




Note how the source code shows the "if" statement as executed (the arrow root is green) but neither path taken. This is to signal a mixed case that has to be analyzed on the assembler level. On the disassembly window you see that the first branch condition was true, so the second one was never evaluated. Note the coverage legend on each assembler line.

3) Another aspect are optimizations. Compilers may merge several source code lines, deem some unnecessary, reorder the execution flow. iSYSTEM tools take great care to show accurately what is actually going on. On the source code view of the coverage information you can tell at once if a line of code generates any object code at all. For lines that do you get coverage information in a fine grained way (that can be configured). A line of code can be executed, not executed, partially executed, contains branches, that can be taken, not taken, or mixed in the case you have several branches.


Here you see how each source code line that generates code has next to it a lowered square that is, depending on the coverage information available, red (not executed), green (executed) or has an arrow, indicating that there is at least one conditional instruction on the object code corresponding to that line of source code.

4) Libraries. As iSYSTEM tools gather coverage information directly from the hardware the analysis of library code does not differ from analysis of the user code. If source code is not available the information is given on the object and assembler levels only. Full information is still available. What is more, this doesn't complicate your overall view of the coverage information, as you can configure coverage to ignore library code or better, have the library code be part of your coverage report and use the hierarchical GUI and reports to dig into the parts that you are interested in.




Here you see an example of a library function with a complex flow. You see that not all instructions were executed.



Here you see a hierarchical view of the coverage information for library functions. There is no available source code for these functions, that is why the "lines" information columns are empty. Nevertheless you have full code coverage information for every bit of the object file and assembler correlation. You can view detailed information for each function and each assembler instruction, or you can collapse parts of the tree to have an overview.

5) Compiler generated code. The compiler sometimes needs to introduce code to bridge the gap between what is possible in source code and on the processor. This is mostly the case for operations that don't translate directly into machine code. For example a conversion of a floating point variable to an integer is usually performed by code that is inserted inline or a library function call is introduced by the compiler at that point. As this is code that wasn't explicitly written by the programmer, similar benefits apply to using our tools here. For example:



See how the assignment statement "c=d;" uses code that isn't obviously part of the source. What is even more interesting is that this code includes conditional execution and even function calls. In this particular case you see that, a part of the code wasn't executed (including a function call). This could lead to catastrophic failure if the function that was never called is never tested and has bugs that can cause problems.

6) Compiler bugs. Seldom, but still, compiler can have bugs. These bugs are rare on the code that is generated, as compilers are typically tested in this respect. Most of the time bugs are in the debug information generated by the compiler and used by other tools. It is important to remark that these happen seldom, but shouldn't be ruled out.