What enterprise buyers ask to see about your regression testing

Where a public framework asks for test records

Buyers' questionnaires are seldom public. NIST's Secure Software Development Framework is, so it serves here as the worked example. Version 1.1, published in February 2022 as Special Publication 800-218, has a task numbered PW.8.2: scope the testing, design the tests, perform the testing, and document the results, including recording and triaging all discovered issues in the development team's workflow or issue tracking system.

Two of that task's examples bear on regression testing. One says to add tests for previously reported vulnerabilities to the project's test suite so that errors are not reintroduced. The other says test plans should take into account the infrastructure and technology stacks the software will be used with in production. A third task, PO.5.1, asks the vendor to separate and protect each environment involved in software development, so expect a row on whether test and production share servers or data.

PW.8 is written about security testing. A question about functional regression asks for evidence of the same shape: what was tested, on which build, in which environment, with which results, and what happened to each failure.

The framework reaches vendors through procurement. CISA released its Secure Software Development Attestation Form on March 11, 2024, based on practices from the framework, for producers of software used by the federal government. Commercial buyers write their own versions, and some will hold a signature until the vendor shows how releases are regression tested.

Five items in the record for one release

Here is the record for one release at a hypothetical vendor, Brackenfell Software. Every value is invented.

Release:        6.4.0
Build:          commit 3f9c2ab, artifact checksum in build.sha256
Environment:    QA configuration version 14
Dataset:        base dataset version 9, loaded by the seed script at that commit
Suite:          regression suite at commit 3f9c2ab
Results:        results.xml, saved beside this file
Open failures:  two, each linked to a ticket and accepted by the QA lead
Approved:       QA lead, with the date

A buyer's engineer can read that in a minute and ask for any file it names.

Two details decide whether the results file holds up under that reading. Skipped tests are listed by name with the reason for each, because a suite that passes with a block of tests switched off has answered a smaller question. A failure that passed on a rerun is recorded as both, with the ticket that explains the difference.

When the QA server was changed by hand between runs

Four of the five items are files that a pipeline or a tracker already produces. The environment item is a statement about servers. On a QA server that has stayed up for months, patched and adjusted in place, nobody can say after the fact which package versions were installed on the day of the run. The record ends up saying "our QA server", which names a machine that has changed since.

A stack rebuilt from a written configuration before each cycle gives that item a short answer, the configuration's version number. On Petrichor, we rebuild a QA stack from your saved environment configuration before each test cycle. Nobody changes the configuration without your QA lead's approval, so the stack under test is the stack that was approved.

QA teams at one of our customers build test stacks across operating systems and databases, destroy each stack after its run and keep the baseline results as history. Their QA lead approved three saved configurations and knows what each one builds. Test environment provisioning from approved configurations walks through one configuration line by line.

Keep the record apart from the stack it describes

A stack that gets rebuilt or torn down takes its local files with it. Copy the five items to storage that outlives the stack when the run ends, before the next rebuild starts.

The framework has two lines on keeping evidence. Task PS.3.1 asks for the necessary files and supporting data to be archived and retained for each software release. An example under task PO.4.2 says to allow only authorized personnel to reach the gathered information, and to prevent any alteration or deletion of it. A repository with protected history fits both lines, and so does storage that refuses overwrites.

How long to keep each record depends on the contract. A workable floor is the last day any customer still runs that release, because the question can arrive about whichever version the buyer has installed. Reproduce a customer bug on their exact version covers the builds to archive for those older releases.

Wording for the questionnaire row

With the five items saved for every release, the row takes two sentences, and each clause points at a file.

Give that answer only when every clause is true today, because the next request is for the record itself. If one clause is still an intention, leave it out and say when it starts.

What the record leaves unanswered

The record shows that a named suite ran on a named build in a known environment. Whether the suite covers the feature a buyer cares about is a separate question, and a buyer may ask for the list of test cases next. Security testing is separate too. The examples under PW.8.2 include fuzz testing and penetration testing, and a functional regression record answers for neither. A pass in QA also carries whatever gap exists between QA and production, and What a passing test proves in dev, QA and staging starts with the database edition.

Your team writes and runs the tests and signs the release. We build and run the environments they run on, and we help you put environment, release and testing practices in place, then verify they are followed. When your customer sends a security questionnaire, we fill out the hosting section with you and explain the controls behind each answer. The testing rows stay with your team, and the record above is what fills them.

Talk to a consultant and bring the testing rows from the last questionnaire a customer sent you.