Where the hours go in a slow regression suite

A ledger for one regression cycle

Take a hypothetical vendor of lab scheduling software, Calderwick Lab Systems. Its testers work eight hour days, so Monday at 9:00 to Friday at noon is 35 working hours. For release 5.2 the QA lead wrote down what the team's one QA stack was doing in each of them.

The suite ran for 12 of the 35 hours, across three runs. For 10 more, the stack was being built, repaired or restored. Of the 36 failures, 27 traced to the stack: rows the previous cycle had left in the database, and the setting the hand build missed. Those 27 took four and a half of the six triage hours.

Halving the suite's run time would return 6 hours. The stack's own 10 hours and the 4.5 hours of triage it caused come to 14.5, and no change to the tests touches them.

Keep the same ledger for one cycle before changing anything. A shared spreadsheet works, with one row each time the stack changes hands and the clock time beside it. Chat messages that say the stack is ready fill in most of the rows afterward.

Setup time, reported apart from the test body

Inside the suite, a test runner can split each test's time into phases. pytest prints the slowest ones when asked.

pytest --durations=20 --durations-min=1.0

The help text for --durations reads "Show N slowest setup/test durations", and each line of the report is tagged setup, call or teardown, so one test can appear three times. These are the top lines for Calderwick's suite, with every value invented.

41.20s setup    tests/booking/test_instrument_queue.py::test_reassign
39.85s setup    tests/booking/test_run_export.py::test_export_batch
 6.10s call     tests/booking/test_run_export.py::test_export_batch
 1.95s teardown tests/booking/test_instrument_queue.py::test_reassign

When setup rows lead the list, the time belongs to fixtures. At Calderwick, 260 of the suite's 1,900 tests request a fixture that reloads the bookings dataset, at about 40 seconds a load. That is 10,400 seconds, or 2 hours 53 minutes of the 4 hour run, spent putting the same rows back.

Two changes shorten a reload without touching what the tests assert. A fixture scoped to the module loads the dataset once for every test in that file, which works when those tests only read. For tests that write, PostgreSQL can copy a prepared database in place of replaying a seed script, with CREATE DATABASE bookings_run TEMPLATE bookings_baseline. The PostgreSQL manual gives the limit: no other session can be connected to the source database while it is copied, and the feature is not yet intended as a general purpose way to copy a database. Keep the baseline closed to connections and copy from it.

Give each worker its own database

Parallel workers are the usual next step. With -n auto, pytest-xdist starts as many processes as the machine has physical CPU cores. Playwright runs test files in parallel by default, in worker processes that cannot communicate with each other, and it runs the tests inside one file in order in the same worker.

The runner separates its own processes and leaves the application's database as it found it. Four workers pointed at one QA database share every row, so a test that cancels booking 1041 in one worker fails a test that reads booking 1041 in another. Those failures change from run to run with the order the workers happen to take, and they land in the triage row of the ledger.

Each worker therefore gets a database of its own, copied from the baseline before the run. Where a group of tests depends on one expensive fixture, the --dist loadscope mode keeps a module's tests together in one process, and the xdist documentation suggests it for expensive module level fixtures.

Work out the ceiling before building any of it. Four workers at a perfect split take each four hour run to one hour, and Calderwick's three runs from 12 hours to 3. That takes the cycle from 35 working hours to 26. Twenty three of those belong to rows of the ledger that workers never touch.

Rerun what failed, from a known state

Both of Calderwick's reruns were full runs. The first run had changed the data, so nobody trusted a rerun of the failed tests alone on a stack in an unknown state.

pytest can rerun the failures by themselves. The help text for --last-failed reads "Rerun only the tests that failed at the last run". Calderwick's 1,900 tests average 7.6 seconds each across the four hours, so 36 failed tests come to about four and a half minutes. That result means something only when the stack is back at its starting state first.

Keep one full run at the end, on the build that ships, because a fix can break a test that passed on Tuesday. With a reset before each targeted rerun, the two rerun rows shrink from 8 hours to one full run and a few minutes.

Hours that belong to the environment

On Petrichor, we rebuild a QA stack from your saved environment configuration before each test cycle. Our provisioning tool works in a fixed order: servers with the right operating systems, the installed software, the environment settings, your application, then the base users and dataset. Nobody changes the configuration without your QA lead's approval, so the stack your testers open on Monday is the one that was approved, and a failure traces back to the release.

Two of the setups most teams start with answer other rows of the ledger. A one step reset back to baseline replaces the hand restore before each fix build and each targeted rerun. Parallel stacks let two releases test at once, and on a second stack a manual pass no longer waits for the automated run to give the data back.

How long a rebuild takes depends on what your configuration installs. Time the first one and write it into the ledger beside the hand build it replaced. Resetting a QA environment between test runs puts four reset methods side by side.

When the test code is the slow part

A ledger can come out the other way. Where the pipeline already rebuilds the stack and the suite runs 14 hours, most of the cycle sits in the suite rows, with call at the top of the durations report. That time is in the tests themselves. A browser test signs in through the login page before every case, or a fixed wait sleeps ten seconds where the page was ready in one.

That work belongs to the people who wrote the suite. Your team writes and runs the tests, and we build and run the environments they run on. A faster stack leaves a 14 hour suite at 14 hours.

Talk to a consultant and bring the ledger from your last cycle, with the hours your testers spent waiting on the stack marked.