darktable integration tests

This repository hosts the darktable integration tests. Each test renders a raw image from images/ through a fixed XMP sidecar with darktable-cli and compares the result against a committed reference image, both on the CPU and the OpenCL path.

Test setup

run                : main driver, runs the tests

deltae             : CIE 2000 delta-E report between two images (C helper,
                     built from deltae.c by ./run)
count-diff-pixels  : number of differing pixels between two images (C helper,
                     built from count-diff-pixels.c by ./run)
check-performance  : reports timing regressions from logs/perfs.log
check-failures     : interactive review of the failures of a log
check-cpu-gpu      : refreshes the cpugpu.maxpix thresholds from a log

src/               : source code for helper scripts
images/            : test images
logs/              : run logs, timings (perfs.log) and their analysis
nnnn-name/         : one test

A test directory contains:

file
<name>.xmp	the sidecar to apply, names the image via DerivedFrom
<name>.dtdata	optional, .dtdata sidecar placed next to the image for the run
expected.png	the reference output, created on the first run
cpugpu.maxpix	max tolerated CPU/GPU pixel difference
CONFIG	optional, one extra darktable-cli conf option per line
README	optional, first line is a label shown next to the test
test.sh	optional, replaces the default driver, returns 0 if OK

A test passes when the max delta-E against expected.png stays below 2.3 (and the average below 2.3/3).

Requirements

• darktable-cli, found either in the PATH, through the DARKTABLE_CLI environment variable, or in /opt/darktable
• compare (ImageMagick), for the diff images
• zopflipng, only needed to create the expected.png of a new test
• loupe, only needed by check-failures
• a C compiler and libpng (pkg-config module libpng), to build the C deltae and count-diff-pixels. Both are built multithreaded when the compiler supports OpenMP, which makes them noticeably faster on large images; a compiler without OpenMP builds them single-threaded

Running the tests

./run                     # all tests
./run 0001-exposure       # a single test
./run --op=exposure       # all tests using that module

Options:

--disable-opencl           do not run the OpenCL path
--disable-timing           do not output timing
--no-deltae                light check, does not need the delta-E module
--fast-fail                stop on the first failing test
--op=<n> | --operation=<n> run the tests with matching operation n
--gdb | --gdb-cl           run the CPU / OpenCL path under gdb

Every run writes logs/test-<date>.log, appends the timings to logs/perfs.log and finally reports timing regressions. To review the failures of a run:

./check-failures logs/test-<date>.log

Adding a test with the default driver

1. Create the directory <nnnn>-<meaningful name>.
2. Develop one of the test images in darktable, then copy its XMP into that directory as <meaningful name>.xmp.
3. Optionally add a README, a CONFIG and a <meaningful name>.dtdata (see the table above).
4. Run ./run <dir> a first time: as there is no expected.png yet, the output is optimized with zopflipng and saved as the reference. Check that it really is the expected output.
5. Run ./run <dir> again, everything must now be 0:

Test 0001-exposure
      Image mire1.cr2
      CPU & GPU version differ by 25699 pixels
      ----------------------------------
      Max dE                   : 0.00000
      Avg dE                   : 0.00000
      Std dE                   : 0.00000
      ----------------------------------
      ...
      Pixels above tolerance   : 0.00 %
  OK

6. Commit the .xmp and the expected.png.

Adding a test with a specific driver

Create <nnnn>-<meaningful name>/test.sh instead. It may do whatever the test needs and must return 0 when the test passes, 1 otherwise.