Test results

Concepts ››
Parent Previous Next

Test results

After test execution status of each test is marked with green or red overlay icons. Status icons are set also on each section of test case, which has results set. Note that not all sections define constraints on test results. Only the following pages define such constraints:

We can see not only that the results of the expression was false, but also the actual value returned from the function.
Note 1: Hex or Dec mode for values printed can be set either in:

Note 2: For floating point values the operator '==' must be used with caution. To make comparison for equality of floating point values more convenient, winIDEA uses vague comparison, where values are considered equal, if their relative difference is less than epsilon. The default value of epsilon is 1.e-5, but can be customized with winIDEA option /IDE/Debug.Symbols.Format.VagueFloatPrecision. Enter this option URL and new epsilon value in testIDEA section Options. Alternatively this value is accessible in winIDEA Debug | Debug Options dialog, tab Symbol.

Test results in the Status view

After the test execution the summary of test results is shown in the Status view. The first line shows some statistic, while other lines contain errors, if they occurred.

Example of the Status View without errors:

Example of the Status View with errors:

Tip: To see the test case, which has failed, double click the error line in the Status view. The corresponding test case will be selected in the Test tree.

Test reports

(Professional version only)

To save test results for documentation purposes, red marks in GUI are not very convenient. For this reason we can save test results to file in several different formats. iSystem testIDEA can save test results in XML, YAML, CSV, XLS, or XLSX file formats.

XML format is convenient for usage in other tools, including viewing in web browsers, because it is widely supported.

YAML format is not so well known, but it is much more readable and as such convenient for quick overview in text editor.

CSV (Comma Separated Values) format is also understood by many other tools, for example it can be imported to rquirements management tools, such as DOORS.

XLS and XLSX are Excel (old and new) formats. We can show XML file in web browser with the help of XML Stylesheet Template. This section describes generating of reports, while XSLTs are described in section iSystem Test Report Stylesheets

Test report can be saved with the main menu option Test | Save Test Report .... We get the following dialog:

As usual, description of input fields is given in their tool-tips.

Note: Internet Explorer does not support document created with option Embed XSLT/CSS into XML report checked. See IE 10, XML and embed XSLT for more details.

An example of the generated test report is shown below. First we can see a table with test configuration parameters, and a table with test statistic. Tables with results of each executed test case are following.

Because one test can fail with more than one error, the sum of number of errors per category can be greater than the number of failed tests in the statistic table.

Coverage results

Coverage report contains the following results: