Reproduce a customer bug on their exact version

Patch levels and settings to collect first

Microsoft's reference page for SERVERPROPERTY documents a behavior change inside SQL Server 2019's cumulative updates. SERVERPROPERTY('ServerName') returns the Windows server name in uppercase at RTM, as saved from CU 1 to CU 8, in uppercase again from CU 9 to CU 12, and as saved from CU 13 on. A server named server01 with an instance called INST1 reports SERVER01\INST1 on CU 12 and server01\INST1 on CU 13, and on the uppercase builds the property disagrees with @@SERVERNAME. An application that compares those two strings case-sensitively fails at a customer on CU 12 and passes on a developer's laptop on CU 13. The ticket says 10.4, and so does the developer's build.

To reproduce a customer issue, start from the customer's patch levels and settings, collected in the first reply.

Ship the collection as one script inside the product's support tooling, so the customer runs one command and pastes the output into the ticket. Replicated's support bundle does the same job for Kubernetes applications. The ServerName column in the paste names the instance, which catches the customer DBA who ran the query against the wrong named instance.

Eight items slow the first reply. Asking later costs more, because the second round trip comes after a failed reproduction. A per-customer deployment record, described in Assign an owner to each release, answers the first row before anyone asks.

Which releases stay built between cases?

The releases carrying most of the last 90 days of tickets stay built, and every other release is built per case from its template. Take a hypothetical vendor of a field inspection application with 178 customers on eight releases. Only the platform names and dates are factual, and every count is invented.

The three busiest releases carry 128 of 173 tickets, 74 percent. That concentration keeps the lab affordable, because ready capacity goes where the tickets land and the other releases wait as templates. Release 12.0 has fewer tickets than 10.2 and stays built anyway. A release that shipped this quarter reaches customers faster than it collects tickets, so it stays built for its first quarter whatever its count.

Releases 9.6 and 9.1 carry the oldest platforms. SQL Server 2016 left extended support on July 14, 2026. Windows Server 2016 follows on January 12, 2027, and extended support for SQL Server 2017 runs through October 12, 2027. Microsoft sells Extended Security Updates for SQL Server 2016 into July 2029, so a customer paying for them keeps receiving security updates, and each one moves the build the 9.6 template has to match. A release leaves the lab on the vendor's own end-of-support date. Until then, the platform date sets how it is handled: built only per case, kept off shared networks, and destroyed when the case closes.

Templates pinned to the majority CU

Each case starts from a clone of the template for its release. Long-lived snapshots make poor clean copies. Snapshots are no substitute for backups in VMware's guidance, which supports up to 32 in a chain, recommends 2 to 3 for performance, and advises keeping no single snapshot longer than 72 hours. A "clean 9.6" snapshot kept for a year is exactly the long-lived snapshot that guidance warns against.

Windows templates get generalized with %WINDIR%\system32\sysprep\sysprep.exe /generalize /shutdown /oobe, adding /mode:vm when the image redeploys on the same hypervisor. Generalizing removes the SID and other machine-specific data.

A clone needs one more step before it can reproduce anything. It gets a new computer name, and the SQL Server instance inside it keeps the template's name in sys.servers. Until someone runs sp_dropserver '<old_name>' and sp_addserver '<new_name>', local and restarts the instance, @@SERVERNAME reports the template's name while SERVERPROPERTY('ServerName') reports the clone's. That is the same mismatch a customer on CU 12 sees, produced by the lab instead, so on an unrenamed clone the ServerName bug reproduces whatever CU the customer runs. Put the rename in the clone's first-boot script. SQL Server's own image preparation (setup.exe /ACTION=PrepareImage in the template, /ACTION=CompleteImage on the clone) avoids it by configuring the instance after the clone is named.

Pin each template to what customers run, never to the latest build. The 10.4 template carries the SQL Server CU and Windows build that most 10.4 customers reported, and the installers for those exact builds sit in the lab's own store. Before reproducing anything, rerun the collection script on the clone and diff it against the customer's paste. A clone that fails the diff gets fixed or rebuilt before anyone reproduces on it.

The majority CU leaves customers on other CUs without an exact match. Build a CU variant only when the paste points at CU-sensitive behavior, because every variant is one more template to repin.

Checkpoints belong inside a single case. A Hyper-V standard checkpoint captures memory state, so an engineer can return to the moment before the failing step and try again (Set-VM -Name <vm> -CheckpointType Standard, Checkpoint-VM, Restore-VMCheckpoint). Production checkpoints use VSS, skip memory, and bring the VM back powered off. Standard checkpoints can cause consistency problems for systems that replicate between nodes, such as Active Directory, so the lab's domain controller never gets rolled back this way.

Date-dependent bugs need the clone's clock moved. Hyper-V's time synchronization integration service sets the guest clock back to the host's, so turn it off first (Disable-VMIntegrationService -VMName <vm> -Name "Time Synchronization") and keep the clone off the lab domain, whose controller corrects the clock too. Any lab key or evaluation license with an expiry reacts to the new date as well.

Our provisioning tool builds each development and QA stack in the same order: servers with the right operating systems, then the software, the environment configuration, the customer's application, and the base users and dataset. We use the same automation tools as everyone else, Salt, Ansible and Terraform, and configure each customer's panel around that team's variables. In a support template, the last step is where the vendor's dataset for the release loads. We also install and support one software vendor's product for its customers who must self-host.

Licensing old platforms for test use

Every layer in a template needs a license that allows test use where the lab runs, and the two evaluation editions carry clocks that can stop a stack in the middle of a case.

• The vendor's own product. Issue lab keys from the vendor's licensing system, marked non-production, one per template, each with an expiry. Loading the customer's license file puts their entitlements and identity in the lab, and if the product binds keys to the host name, the key fails on a clone anyway.
• Windows Server evaluation media. The Windows Server 2025 evaluation expires after 180 days and must be activated over the internet within the first 10 days to avoid automatic shutdown. A lab network with no internet access hits the 10-day limit first. Evaluation media suits a stack that lives for one case and fails a template kept for a year.
• SQL Server 2025. The 2025 license terms let Developer edition run "on any device, including third party shared devices" to design, develop, test and demonstrate your programs, with no production use. Enterprise Developer and Standard Developer editions exist in 2025, so a template for Standard customers can run the Standard engine. The Evaluation edition is licensed only to demonstrate and internally evaluate SQL Server itself, on your own devices, and stops running 180 days after installation with no further notice.
• Older SQL Server versions. The 2025 terms cover 2025. A template running SQL Server 2017 or 2019 installs under that version's own terms, so read the hosting clause in each before the template goes onto shared servers.
• Visual Studio subscriptions. Subscriber software is licensed per user for development and test, so every engineer who uses it in the lab needs a subscription. Since October 1, 2019, it may not run on dedicated hosted cloud services from Microsoft's Listed Providers, a list that names Alibaba, Amazon, Google and Microsoft. Azure stays allowed during the subscription term. Check where the lab's servers sit before installing subscriber software.
• Red Hat Enterprise Linux. The no-cost Developer Subscription for Individuals covers up to 16 physical or virtual nodes and is meant for individuals, one per account. Companies buy the Developer Subscription for Teams through Red Hat sales. Older RHEL releases sit on the Customer Portal.

License-included instances settle the licensing question at a price. On AWS, SQL Server Standard license-included adds $0.48 an hour to an m7i.xlarge ($0.8656 against $0.3856 for Windows alone), which comes to $2,803.20 a month across eight servers left running and $115.20 when it bills only for the case hours priced below. Free developer editions avoid that charge and require a terms check for every version in every location. Visual Studio's terms also carry a data rule: acceptance testing must not use live production data, and any copy used must be discarded when testing is complete.

When the customer sends a .bak file

The vendor's own dataset is the default, and a customer backup comes in only when the bug depends on the customer's data. In order of preference:

• the vendor's dataset for that release, loaded as the last build step
• a script that recreates the few rows involved
• sanitized extract produced by the customer
• the full backup, under the contract's data terms

A customer backup is untrusted input and regulated data at once. Microsoft's backup and restore documentation says a malicious .bak file can take over the SQL Server instance and escalate to the host or VM before any validating script runs. Its guidance: treat restores as a high-risk operation, use isolated instances, and "never restore backups from unknown or external sources." A customer's backup is an external source, so a lab that restores one is going against that guidance. The restore runs on a throwaway instance on its own network segment, and the stack is destroyed afterward.

Read the header on that same isolated instance, since RESTORE HEADERONLY parses the file too. Its SoftwareVersionMajor, SoftwareVersionMinor and SoftwareVersionBuild columns name the build that took the backup, and its Collation and CompatibilityLevel columns should match the customer's paste. A mismatch means the backup came from another server, or from before the customer's last patch.

SQL Server backups cannot be restored on an earlier version. A restore onto a newer version upgrades the database, which keeps its compatibility level if the newer instance supports it. Compatibility level holds some query processing behavior at the older level, and discontinued features get no protection from it, so a newer engine running an older-looking database can make the bug disappear. Restore onto the template for the customer's release, then run DBCC CHECKDB.

The instance collation is a template setting taken from the customer's paste. A database restored onto an instance with a different collation can fail on temp-table comparisons, and that failure belongs to the lab.

Delete the customer copy when the case closes, record the deletion in the ticket, and limit access to the people on the case. Our customers whose own staff are developers already file tickets for us to do database work, take backups and restore data. We gate environments by team, so a stack holding a restored copy can be visible only to the support team working the case.

The vendor dataset misses the customer's data shape: the bug that needs millions of rows, or one orphaned record from a migration years ago. Sometimes the full backup is the only way in, and that path costs the most in separation work and paperwork.

Evidence kept after the stack is destroyed

The ticket keeps enough to rebuild the case without the stack:

• fingerprint pasted by the customer
• template name and template version
• dataset ID, or the customer backup reference with its deletion date
• reproduction script and its logs
• outcome and the next owner

A bug that appears on the customer's version and is gone on the newest release becomes an upgrade or backport question. When the exact version will not reproduce it, suspicion moves to the customer's own network, identity provider or security software. A bug that also appears on the newest release goes to development, and the handoff is the template version plus the dataset ID. Development rebuilds from those instead of inheriting a running VM. If development needs the live stack until a fix ships, the ticket carries an expiry date for it.

A stack that held only the vendor's dataset can be reset instead of rebuilt, using the methods in Reset QA for the next test. Each matrix run on our platform creates test environments across operating systems and databases, destroys them after the run, and keeps the baseline results as history. A support case fits the same pattern, with the results kept and the stack gone.

Stacks kept "until the fix ships" that nobody destroys are the common failure. They hold customer data past its deletion date and run database engines that no longer receive patches.

Sixteen servers, priced running and stopped

Take the eight releases from the release table, each with an application server and a SQL server on Windows, m7i.xlarge (4 vCPU, 16 GiB), 200 GB of gp3 storage per server, and a hypothetical 40 cases a month at 6 hours each. Prices are AWS's published on-demand rates for US East (N. Virginia), checked September 27, 2026.

A stopped EC2 instance stops billing for compute, its EBS volumes keep billing, and every start bills at least one minute.

The decision that matters is $4,503.81 against $441.09, about ten times. Keeping four releases as snapshots instead of stopped volumes saves about $104 a month and costs time on every case that touches them. A volume created from a snapshot pulls its blocks from S3 on first read, so the first SQL Server start and the first DBCC CHECKDB on a rebuilt stack run slowly unless the volume is initialized first or Fast Snapshot Restore is on, at $0.75 an hour per snapshot per Availability Zone. The kept-built list in the release table sets how long support waits for its first reproduction; the bill barely changes with it.

Two costs sit outside the table. Engineers spend time repinning templates as customers take new CUs, and a few hours of that a month outweigh the $256 storage line. A customer backup left in object storage past its case keeps billing and keeps the data around.

Vendors who should skip the lab

A support lab suits vendors whose customers run many versions on infrastructure the vendor does not control. Three kinds of vendor should do something smaller, and one kind of bug needs the customer's own install instead.

• SaaS vendors with one version in production. Staging answers the question. Development, QA and staging covers that setup.
• Kubernetes applications shipped through Replicated. The customer's support bundle comes first, and sbctl serves it as a local API so an engineer can run kubectl against it. A matching cluster from Compatibility Matrix comes next. Those clusters are ephemeral, billed by the minute plus a startup charge, cannot be resized or rebooted, and in the cloud cannot configure CNI, CSI, CRI or ingress. The flow fits bugs in manifests and cluster behavior. A bug in the customer's external database or storage still needs a VM stack.
• Vendors already on AWS or Azure with an owner for the account. A scripted stop schedule and snapshots give roughly the $441.09 bill with tools the team already runs, and that may be the whole lab.
• Bugs rooted in the customer's network, identity provider or security software. No lab holds the customer's proxy or directory, so the route is a session on the customer's own install. Where we host a third-party application for a customer, we patch and maintain it and coordinate access when the software vendor needs to get in.

Bring your supported-version list, with customer counts per release, to a call with one of our cloud engineers.