Migration rehearsal: cutover timings and the rollback deadline
Rehearsing a migration means running the production cutover runbook against a restored copy of the data, with outbound mail and payments blocked. The team comes away with measured timings for each step, a connection list built from recorded traffic, and a rollback deadline agreed before cutover night.
Connections the architecture diagram leaves out
Architecture diagrams show the connections someone remembered to draw. Cutovers tend to break on the others, such as a payment processor that allowlisted the Elastic IP on your NAT gateway, or a month-end report job reading from a file share nobody listed.
Build the list from recorded traffic instead. Turn on VPC Flow Logs for the application's network interfaces long enough to cover a month end, then query them in Athena for every distinct remote address and port. On each server, ss -tnp on Linux and Get-NetTCPConnection on Windows show live connections with the owning process. SQL Server keeps its own record. Grouping sys.dm_exec_sessions by host_name and program_name shows the clients connected when you run it, so sample it every few minutes through the month end, or record logins with an Extended Events session. The client supplies that host name, so treat it as a lead to confirm.
A traffic capture misses scheduled work until the job runs, so read the schedulers directly. That means crontab -l for each service account, /etc/cron.d and systemctl list-timers on Linux, Get-ScheduledTask on Windows, and the SQL Server Agent jobs stored in msdb. Each job goes on the list with its schedule, what it reads and what it writes.
The cheapest test of the list is moving QA first. Petrichor starts customers on QA, or sometimes dev, and production waits. QA goes through the same allowlists, credentials and restore procedure while only testers depend on it, and if the team doesn't like how it runs, it goes back where it came from. When a team asks us to take production on day one, we slow the project down until we know the application.
Four AWS dependencies stop working once the application runs somewhere else:
- EC2 instance profiles hand the SDK's default credential chain its AWS keys through the instance metadata service at 169.254.169.254, which hands out AWS credentials only on EC2. Anywhere else the SDK finds no credentials, and calls to S3, SQS or SES fail, often inside a background worker.
- ACM exports a public certificate only if it was requested as exportable, a paid option, and no certificate created before June 17, 2025 qualifies. Every other public certificate from ACM stays in AWS, so the destination needs its own certificate and a renewal path. An exported one still expires after 198 days, and deploying each renewal is your job.
- A partner that allowlisted your NAT gateway's Elastic IP rejects traffic from the new address. Send each partner the new address before the first rehearsal, and test from it.
- DynamoDB tables, Lambda functions and Step Functions workflows have no direct equivalent outside AWS, and each one turns its component into a rewrite.
Time each step on restored data
Each rehearsal runs the production runbook, command for command, on the destination hardware against a copy restored from a recent production backup. A sample database tells you nothing about the night a 600 GB database has to cross the internet.
When Petrichor runs a migration, our engineers design the target environment, present it to your team, and test the cutover with your developers repeatedly before the production date. Your developers run the application checks in each pass, since they know which screens and jobs matter. Petrichor keeps a person on top of its automation, because a script can report success while the application is down.
Give the runbook a column for measured time and fill it in on every pass. Estimates written before the first rehearsal usually come in low.
| Runbook column | What goes in it |
|---|---|
| Step | One command, script or screen action, in execution order |
| Owner | One named person, plus a named backup |
| Done when | The check that proves the step worked, such as a row count or an HTTP 200 from a named URL |
| Measured time | Wall-clock start and finish from each rehearsal pass |
| Abort rule | The result that stops the cutover at this step |
Take a hypothetical freight broker, Larkspur Freight, whose dispatch application runs on a 600 GB SQL Server database, and assume backup compression brings the full backup to 200 GB. At a sustained 500 Mbps, those 1,600 gigabits take 3,200 seconds to cross, which is 53 minutes before the restore starts. A cutover that copies the full backup on the night carries those 53 minutes plus the restore, the checks and the DNS change. Seed the full backup days ahead and the night only has to move the last differential or log backups. An 8 GB file crosses the same link in about two minutes.
DNS goes in the timings too. A hostname with a 3,600-second TTL needs lowering to 60 at least an hour before cutover, since resolvers keep the old answer that long, and a day ahead is safer. Route 53's procedure for moving a live domain waits two days after lowering an NS record TTL of 172,800 seconds, for the same reason. Some clients keep their own cache. A JVM running with a security manager holds lookups forever by default, and connection pools keep sockets open to the old address until they recycle. Put those restarts in the runbook. After the switch, read the old instance's error log for failed logins against Dispatch: error 18456, state 38 ("Failed to open the explicitly specified database"), or state 40 when Dispatch is the login's default database. Each entry names the client address, and each address is a client the inventory missed.
Every pass also copies data out of AWS. AWS's price list for us-east-1 charges $0.09 per GB for the first 10 TB a month beyond a 100 GB free allowance, so four passes of the 200 GB backup in one month bill 800 GB, about $72, once normal traffic has used the free 100 GB. File stores cost more. One copy of 4 TB of uploaded documents is 4,096 GB, about $369. If all of your data is leaving AWS, or all of it is leaving one AWS service, AWS Support can credit that egress. Approval has to come before the transfer starts and gives you 90 days to finish, so file once the rehearsal dates are fixed.
Seeding the database days ahead
For SQL Server on EC2 or on your own Windows servers, restore the full backup on the destination WITH NORECOVERY days ahead, then keep applying log backups as they are taken. On the night, stop the application, take a final log backup with BACKUP LOG Dispatch TO DISK = N'D:\cutover\Dispatch_final.trn' WITH NORECOVERY;, restore it, and bring the destination up WITH RECOVERY. NORECOVERY on that last backup leaves the source database in the restoring state, so nothing can write to the old copy by mistake, and the log chain stays intact for a return trip.
RDS for SQL Server narrows the options. AWS's documentation says you can't take native log backups on RDS, so the native path out is a full backup through msdb.dbo.rds_backup_database to S3, restored WITH NORECOVERY, followed by a differential at cutover. That differential is valid only if no snapshot ran after the full backup, and with automated backups on, RDS takes one in the backup window every day. RDS also cancels any native backup that overlaps that window. The full backup therefore has to run after the last backup window before cutover, and the rehearsal shows whether the backup, the copy and the restore fit in the hours left. Because the source stays online the whole time, the runbook has to stop every writer before the final differential. Native backups also need the SQLSERVER_BACKUP_RESTORE option in the instance's option group and an IAM role for the S3 bucket. Set both up weeks ahead so the first rehearsal isn't spent on permissions.
For PostgreSQL on RDS, logical replication keeps the destination current up to cutover. Turning it on means setting the static parameter rds.logical_replication to 1 and rebooting the instance. That reboot is a production change in its own right, so schedule it well before the first rehearsal. Four documented gaps in logical replication show up at cutover:
- Sequences stay at their start values on the subscriber. Before opening writes, set each one past the current maximum, for example
SELECT setval('orders_id_seq', (SELECT max(id) FROM orders));, or the first insert fails on a duplicate key. - Schema changes don't replicate. Freeze schema migrations for the replication window, or apply additive changes to the subscriber first.
- Large objects don't replicate either, so any table that uses them needs another copy method.
- An unread replication slot keeps WAL on the source and can fill its storage. Drop the rehearsal subscription after every pass and confirm the slot is gone from
pg_replication_slots.
What counts as a pass?
Write the pass criteria before the first run, with numbers, and hold every later pass to them. For Larkspur, a pass means all of these hold:
- Row counts per table match between source and destination at the cutover point, and
DBCC CHECKDBon the restored database reports no errors. - Each named business transaction completes through the normal user interface: create a load, assign a driver, close it out, and find it on the invoice run.
- Every scheduled job from the inventory runs once on the destination and writes what it wrote on the source.
- Measured downtime, from stopping writes to reopening them, fits the window the business approved, with the rehearsed rollback time added on top.
- A full rollback has run end to end at least once.
Restoring a user database brings its tables and its database users. Instance-level objects stay on the old server, since logins live in master and Agent jobs, Database Mail profiles and backup history live in msdb. Restore onto a new instance with freshly created logins and every SQL authentication user comes up orphaned, because its SID no longer matches a login. The application reaches the server and still can't open its database. Create the logins with their original SIDs (CREATE LOGIN app_user WITH PASSWORD = '...', SID = 0x...;) or remap afterward with ALTER USER app_user WITH LOGIN = app_user;, and add a runbook step that scripts the Agent jobs onto the new instance.
A restored copy of production holds live customer email addresses, webhook URLs and payment tokens, and its scheduled jobs will use them. Before the first job runs, point SMTP at a capture server such as Mailpit, swap payment and messaging credentials for each provider's sandbox keys, and give the rehearsal network outbound rules that allow only the destinations on the inventory. Then run the jobs and read what they tried to send.
After the first customer write
Until a customer writes to the destination, rolling back is a routing change. On SQL Server you run yourself, RESTORE DATABASE Dispatch WITH RECOVERY; brings the source out of the restoring state with nothing lost, and DNS goes back to the old address.
Once writes land, the old database is missing rows, and a rollback has to carry them back. SQL Server can do that with log shipping in reverse. The source already sits in the restoring state at the cutover point, so log backups from the destination restore onto it without a new full backup. That path closes if the destination runs a newer SQL Server version, because no SQL Server backup restores to an earlier version than the one that created it. Upgrading from SQL Server 2016 to 2022 in the same cutover gives up the reverse path. A source on RDS never enters the restoring state, so it has no reverse log path at all. In both cases the plan has to say what replaces it.
Keeping writes closed while the checks run usually costs less than carrying writes back. Bring the destination up with the application in maintenance mode, let the application owners work through the pass criteria, and open writes only after they sign off. While writes stay closed the source is still current, and a rollback loses nothing. The rehearsal tells you how long that window has to last.
Put the rollback deadline in the runbook as a clock time. In Larkspur's cutover, if dispatchers need a working system by 02:30 and the rehearsed rollback takes 40 minutes, the decision is due by 01:50, and setting it at 01:30 leaves room for a rollback that runs long. One named person makes that call, with a named backup. Any writes accepted before a rollback need an owner as well. Either the reverse log path carries them or someone reconciles them from the application's records, and the runbook says which.
Workloads better left where they are
Some inventories end the project. If the connection list is dominated by DynamoDB, Lambda and Step Functions, the migration turns into a rewrite of the application's core, and that workload is better off on AWS. Our engineers will tell you that on the call.
Small, quiet systems rarely justify the full rehearsal. An internal tool with a 20 GB database and no integrations, whose owners have approved a weekend outage, can move on a checklist and one test restore on the destination. The full treatment earns its cost when the approved downtime is shorter than a cold copy and restore, or when other systems write into the application.
Bring the application you would move first, with its connection list, to a call with a Petrichor engineer.