[ << Regression tests ] | [Top][Contents][Index][ ? ] | [ Programming work >> ] | ||
[ < Compiling regression tests ] | [ Up : Regression tests ] | [ Memory and coverage tests > ] |
8.4 Identifying code regressions
Before modified code is committed to master, a regression test comparision must be completed to ensure that the changes have not caused problems with previously working code. The comparison is made automatically upon compiling the regression test suite twice.
Before making changes, a baseline should be established by running:
make test-baseline
After making the changes, the code should be checked by running:
make check
After ‘make check’ is complete, a regression test comparison will be available at ‘out/test-results/index.html’. For each regression test that differs between the baseline and the changed code, a regression test entry will displayed. Ideally, the only changes would be the changes that you were working on. If regressions are introduced, they must be fixed before committing the code.
Note: The special regression test ‘test-output-distance.ly’ will always show up as a regression. This test changes each time it is run, and serves to verify that the regression tests have, in fact, run.
Once ‘make test-baseline’ and ‘make check’ have been run, the files that differ between ‘test-baseline’ and ‘check’ can be repeatedly examined by doing:
make test-redo
This updates the regression list at ‘out/test-results/index.html’. It does not redo ‘test-output-distance.ly’.
When all regressions have been resolved, the output list will be empty.
Once all regressions have been resolved, a final check should be completed by running:
make test-clean make check
This cleans the results of the previous ‘make check’, then does the automatic regression comparison again.
[ << Regression tests ] | [Top][Contents][Index][ ? ] | [ Programming work >> ] | ||
[ < Compiling regression tests ] | [ Up : Regression tests ] | [ Memory and coverage tests > ] |