Compatibility test matrices across operating systems and databases

Strike what Microsoft's and Oracle's platform tables leave out

Oxbend Works, a hypothetical vendor of maintenance-management software that customers install on their own servers, publishes this supported platform matrix: Windows Server 2019, 2022 and 2025 and RHEL 8 and 9, with SQL Server 2019, 2022 and 2025 or MySQL 8.0 and 8.4. Five operating systems by five databases makes 25 cells. When that grid lives only in the install guide, testing covers whichever combination a free machine happens to have.

Check every cell against the database vendors' tables before any tool gets involved. Microsoft's SQL Server on Linux release notes list RHEL 9 and 10 for SQL Server 2025, and RHEL 7.7 to 7.9 or 8 for SQL Server 2019, which strikes SQL Server 2025 on RHEL 8 and SQL Server 2019 on RHEL 9. On Windows, all three SQL Server versions are supported on all three Windows Server versions from RTM, and an operating system supported for a major version stays supported for its later cumulative updates unless noted otherwise, so a CU does not reopen the question. Oracle's supported platforms page lists MySQL 8.4 on Windows Server 2019 and 2022 and on RHEL 8, 9 and 10. Windows Server 2025 has no entry for any MySQL version, which strikes both MySQL cells on that row.

Four cells come out, leaving 21.

MySQL 8.0 moved to Sustaining Support on April 21, 2026, and the supported platforms page now lists only 8.4 and 9.7. Sustaining Support carries no new fixes, so the four 8.0 cells stay in the grid only because Oxbend's customers still run them, and their engine build stops changing.

These tables change. SQL Server 2025 added RHEL 10 and dropped SLES, so the grid gets rechecked whenever a database vendor ships a major version. A customer who insists on a cell outside the vendor's list raises a support-policy question for the product team, and testing that cell does not make it supported.

A 21-row floor under pairwise testing

Oxbend's customers differ in two more ways the grid does not show. Some install fresh, and some upgrade the database engine in place from the previous major version. Twelve of the 21 cells have that upgrade path: SQL Server 2022 from 2019 on four operating systems, SQL Server 2025 from 2022 on four, and MySQL 8.4 from 8.0 on four. SQL Server 2019 and MySQL 8.0 are the oldest versions in the grid, and SQL Server 2022 on RHEL 9 has no in-place path because 2019 never ran there. Server locale is the other dimension, here en-US, de-DE and fr-FR, because SQL Server setup on Windows picks the instance collation from the OS locale.

Every combination comes to 33 operating system, database and install cells times three locales, or 99. Pairwise selection covers every pair of values at least once, and in this grid that sets a hard floor. Each of the 21 supported operating system and database pairs needs a row of its own, so no pairwise set can have fewer than 21 rows. A 21-row set exists, with install type and locale folded into those same rows. Three-way coverage needs at least 63 rows (21 pairs times three locales), and a 63-row set exists too. Pairwise selection trims the smaller dimensions and leaves the grid itself at 21.

PICT, Microsoft's open-source pairwise tool, takes the model as plain text:

OS: WS2019, WS2022, WS2025, RHEL8, RHEL9
Database: SQL2019, SQL2022, SQL2025, MySQL80, MySQL84
Install: fresh, upgrade
Locale: en-US, de-DE, fr-FR

IF [OS] = "RHEL9" THEN [Database] <> "SQL2019";
IF [OS] = "RHEL8" THEN [Database] <> "SQL2025";
IF [OS] = "WS2025" THEN [Database] NOT LIKE "MySQL*";
IF [Database] IN {"SQL2019", "MySQL80"} THEN [Install] = "fresh";
IF [OS] = "RHEL9" AND [Database] = "SQL2022" THEN [Install] = "fresh";

pict matrix.txt generates the pairwise set, since order 2 is the default, and pict matrix.txt /o:3 generates three-way combinations. PICT's packing is heuristic, and row counts vary 5 to 10 percent between random seeds, so the output can land a little above 21. Without /r, PICT returns the same rows every run, which keeps release candidates comparable. Rotating the seed with /r:N gives each release candidate a different set, so over several candidates the sweep reaches triples the first set missed.

NIST's combinatorial testing group finds that most failures trace to one or two parameters, with progressively fewer needing three or more, and warns that "it is not enough to test all pairs of values." Oxbend has a three-condition failure sitting in this grid. A tempdb collation conflict needs SQL Server, on Windows, installed under a locale whose default collation differs from the collation of the product's database. The three locales map to three different collations, so whatever collation the product database pins, at least two locales give the instance a different one. Here the Windows Server 2025 row forces that case into every pairwise set, because every cell on that row is SQL Server and pairwise has to put Windows Server 2025 with each locale. A failure that needs a specific version, an upgrade install and a locale together, such as SQL Server 2022 upgraded from 2019 under fr-FR, gets no such guarantee. Pairwise covers each pair of those values and holds all three together only by chance. Three-way coverage guarantees it, at 63 rows.

Twenty-one rows makes a release-candidate sweep, and 63 rows triples the build count to catch failures that need three conditions. Which one a team runs is a budget call.

Which cells get full regression on every build?

The cells that hold most customer installs get it, plus the newest operating system and database version the product supports, even when almost nobody runs that pairing yet. Oxbend counts installs from license check-ins, 400 in all (every figure here is hypothetical). Seven cells meet the rule: Windows Server 2022 with SQL Server 2022 (96 installs), Windows Server 2019 with SQL Server 2019 (71), Windows Server 2019 with SQL Server 2022 (48), RHEL 8 with MySQL 8.0 (39), Windows Server 2022 with SQL Server 2019 (30), RHEL 9 with MySQL 8.4 (22), and Windows Server 2025 with SQL Server 2025 (2, promoted as the newest pairing).

At 40 merges a week, full regression on all 21 cells means 840 full runs. The tiered plan runs 280 full regressions on tier 1, 440 smoke runs and 55 nightly full runs on tier 2 across five weeknights, and three full runs per release candidate on tier 3. That comes to 335 full runs a week against 840, plus three per release candidate. Each release candidate also gets the 21-row pairwise sweep, which is where install type and locale vary across the compatibility test matrix.

RHEL 8 with MySQL 8.0 sits in tier 1 with 39 installs while Oracle has already moved 8.0 to Sustaining Support. The install base sets the test tier, and the lifecycle date sets when Oxbend announces the end of 8.0 support.

Tiers built from last quarter's installs trail the customer base. A customer who moves to Windows Server 2025 while keeping SQL Server 2019 lands in a tier-3 cell that last ran on the previous release candidate. Re-rank before each release, and promote any cell that produces a support ticket. A smoke suite also misses what only full regression exercises, and tier 2's nightly full run bounds that gap to one day of merges.

Most matrix failures come from install-time settings and in-place upgrades.

In MySQL, lower_case_table_names defaults to 0 on Unix, 1 on Windows and 2 on macOS, and it can be set only when the server is initialized. Changing it afterward is prohibited. A query that names Orders against a table created as orders passes on a Windows cell and fails on a RHEL cell. For InnoDB tables that move between platforms, the recommended setting is lower_case_table_names=1 on every server. A matrix has to test the setting Oxbend's install guide tells customers to use, so the Linux cells get it at initialization, before any data exists.

The SQL Server equivalent is collation. Windows setup takes it from the OS locale. On Linux, mssql-conf set-collation changes it with the service stopped, and any user database already on the instance has to be backed up and detached first and restored afterward. Set before the first user database exists, it needs none of that.

Each documented break below depends on how the engine was set up or how it reached its current version.

A grid of fresh en-US installs misses two of these outright: the full-text break appears only after an upgrade, and the collation break only under a non-matching locale. Collation conflicts hit hardest when the product ships its database as a backup to restore, because the backup carries the collation of the machine that made it. Declaring temp table character columns with COLLATE DATABASE_DEFAULT is the fix in the product code, and the locale cells prove it stays fixed.

An upgrade cell builds the previous major version, loads the dataset, upgrades the engine in place, and then runs whatever the product's upgrade guide tells customers to run, and nothing more. A missing full-text rebuild step then surfaces as Msg 30010 in a test cell, before any customer files a ticket. This is the engine parity that Development, QA and staging asks for, narrowed to the settings that break.

Every upgrade cell is a second build of the same operating system and database pair, so the grid grows from 21 cells to 33 before locales. Keep upgrade cells to the previous major version. Customers two majors behind need a support conversation more than a test cell.

Matrix runs: a new stack per combination, torn down afterward

A matrix run gives each cell a stack of its own, built from one approved configuration, loaded with the same dataset, and destroyed after the suite finishes. Each result then depends on the product build and that cell's settings, with nothing carried over from the previous run.

Petrichor engineers build each stack from the configuration the customer's QA lead approved, in a fixed order:

1. Servers with the operating system the cell names
2. The software, including the database engine at its pinned build
3. The environment configuration, where lower_case_table_names, the SQL Server collation and the MySQL authentication plugin get set
4. The customer's application
5. The base users and the dataset

Then the suite runs, and its results go into the baseline history.

On Petrichor, one software company's QA teams test this way across operating systems and databases, destroy each stack after its run, and keep the baseline results as history. QA environments already turn over faster than dev environments, because each validation run needs a fresh copy with a specific dataset, and a matrix multiplies that turnover by the number of cells. Access can be gated by team, so a Windows test team sees only the Windows combinations.

The dataset needs the most care when a product supports two engines. A vendor supporting SQL Server and MySQL keeps one canonical dataset and generates a load artifact for each engine. After loading, the stack gets checked: row counts per table and the schema version have to match the dataset's manifest. A MySQL artifact whose table names carry mixed case fails on the RHEL cells for the reason in the previous section. Reset QA for the next test covers the starting-state record.

None of the tooling is unusual. Everybody uses Salt, Ansible and Terraform, and so does Petrichor. The platform runs on an API with command line wrappers, which is how Petrichor engineers operate it.

Petrichor keeps an engineer watching the automation, because automation fails silently. A teardown can fail and leave a stack running past its run, so after each run, nothing tagged with that run's ID should still exist. The quieter failure is a cell that reports green because the runner never connected and zero tests executed, which the test-count comparison against the baseline catches.

A fresh build per cell costs build work on every run. Long-lived environments start faster and drift, and a matrix built on drifting environments reports the drift as product failures.

Baselines per engine version and cumulative update

A baseline compares cleanly only when everything except the product build matches. Its key holds the product build, OS build, database engine version and cumulative update, install type, locale, and dataset version, and each run compares against the last result with the same key except the product build. Record the engine build with every result, using SELECT @@VERSION on SQL Server and SELECT VERSION() on MySQL.

One cell's failure reads differently depending on which other cells fail with it.

Timing needs its own boundary. MySQL 8.4's new defaults for I/O capacity, the Linux flush method, the adaptive hash index and change buffering mean a timing baseline taken on 8.0 says little about 8.4. Start a new timing series when the engine major version changes, and keep comparing pass and fail across that boundary.

A fine-grained key means each new cumulative update starts that cell with a thin history. Pass and fail still compare across the update, and only the timing series restarts. Some columns stop churning on their own: CU 32 was the final cumulative update for SQL Server 2019, so those cells now change engine build only when a security GDR ships.

Where hosted CI runners are enough

Plenty of products never need environments outside their CI system's own matrix. On GitHub Actions, a job matrix generates at most 256 jobs per workflow run. Service containers need a Linux runner, which on GitHub-hosted runners means Ubuntu. The hosted images cover Ubuntu 22.04, 24.04 and 26.04, Windows Server 2022 and 2025, and macOS, and RHEL and Windows Server 2019 are absent from that list.

Applied to the grid, the 8 cells on Windows Server 2022 and 2025 can run on hosted runners, with the database installed inside each job because service containers do not run on Windows. The other 13 cells, on Windows Server 2019 and both RHEL versions, need self-hosted runners or VMs, and that is where test environments for multiple operating systems turn into an infrastructure job. All 99 exhaustive cells fit under 256, though a Team plan runs at most 60 standard jobs at once, so the grid runs in waves. Splitting each cell into three test shards makes 297, over the cap for one workflow run. SQL Server container images are supported only on x86-64 Linux hosts, and emulation such as Rosetta 2, Prism or QEMU is unsupported, so ARM runners and Apple silicon laptops cannot host a supported SQL Server cell.

Three situations call for staying with what the team has:

• Every cell runs as a container on an Ubuntu runner, and the grid fits under the cap. The CI system's matrix, with its include and exclude keys, is the right tool.
• The product runs only as the vendor's own SaaS. With no customer-installed grid to cover, the matrix is production plus the next version.
• A team on AWS or Azure already has an engineer building an image per cell and tearing the cells down. Keep that setup, and keep any application that depends on managed services it cannot replace where it is.

The cells hosted runners cannot hold are RHEL, Windows Server 2019, and any cell that needs a licensed SQL Server Standard install. Petrichor runs Linux and Windows, and our engineers build those stacks as development and QA environments, starting with dev and QA while production stays where it is, often on AWS.

Talk to a consultant and bring the supported platform list from your install guide. Ask which combinations a matrix run would cover.