8.4 Identifying code regressions

Before modified code is committed to master, a regression test comparision must be completed to ensure that the changes have not caused problems with previously working code. The comparison is made automatically upon compiling the regression test suite twice.

Before making changes, a baseline should be established by running:

make test-baseline

After making the changes, the code should be checked by running:

make check

After ‘make check’ is complete, a regression test comparison will be available at ‘out/test-results/index.html’. For each regression test that differs between the baseline and the changed code, a regression test entry will displayed. Ideally, the only changes would be the changes that you were working on. If regressions are introduced, they must be fixed before committing the code.

Note: The special regression test ‘test-output-distance.ly’ will always show up as a regression. This test changes each time it is run, and serves to verify that the regression tests have, in fact, run.

Once ‘make test-baseline’ and ‘make check’ have been run, the files that differ between ‘test-baseline’ and ‘check’ can be repeatedly examined by doing:

make test-redo

This updates the regression list at ‘out/test-results/index.html’. It does not redo ‘test-output-distance.ly’.

When all regressions have been resolved, the output list will be empty.

Once all regressions have been resolved, a final check should be completed by running:

make test-clean
make check

This cleans the results of the previous ‘make check’, then does the automatic regression comparison again.


LilyPond — Contributor’s Guide