[UTest] Several Bugs in unittest-parallel Script
❗ 🚨 🚨 🚨 Test Crashes not reported AT ALL -- in Plain Text Report 🚨 🚨 🚨
In the following log, we can see two tests crashing and dumping their core. Yet, the script neither reports these tests as failed nor does the script exit with a non-zero exit status.
NOTE: Our CI runs the JUnit reporter. (See next section.)
/home/immanuel/Documents/Work/PhD/mutable/mutable/include/mutable/util/Pool.hpp:131: Condition 'nh.mapped() == 0' failed. deleting would create a dangling reference to pooled object.
timeout: the monitored command dumped core
/home/immanuel/Documents/Work/PhD/mutable/mutable/include/mutable/util/Pool.hpp:131: Condition 'nh.mapped() == 0' failed. deleting would create a dangling reference to pooled object.
timeout: the monitored command dumped core
===============================================================================
All tests passed (27390 assertions in 439 test cases)
Execution time: 7.086653470993042s
❗ Test Crashes reported as Failures, not Errors -- in JUnit Report
The same scenario as above run with the JUnit reporter of the script has the script report the test crashes as failures. That is incorrect. Failures mean that checks performed by a test failed. If a test exits abnormally (that is neither 0
for SUCCESS nor 1
for FAILURE), it should be considered as ERROR.
<testsuites><testsuite errors="0" failures="2" tests="27282" time="7.5353264808654785"><testcase classname="unittest.global" name="reader_writer_mutex/simulation" time="0.000" status="run" />
❗ Test Timeouts reported as Failures, not Errors -- in JUnit Report
With a timeout of 1 second, several tests will time out. This should have them be reported as test errors, not failures. The PlainText report shows them as test timeouts.
<testsuites><testsuite errors="0" failures="16" tests="25928" time="6.3560826778411865"><testcase classname="unittest.global" name="ColumnStore/ctor" time="0.000" status="run" />
Edited by Haffner Immanuel