Release management when a hosting provider runs production

Who can stop the release?

At least three groups touch a release of software that a hosting provider operates: the vendor's engineers, who know what changed in the code; the host's engineers, who run the servers, the database and the backups; and the customer, who approves the window and whose users will be first to notice if invoicing breaks. Before the window opens, the three need to agree on who can stop the release, which artifact ships, how far back it can roll, and which version each customer ends up running.

Take a hypothetical release in which every group finished its own steps. The vendor's pipeline reported success and the host's monitoring showed every service up, so the first sign of trouble was a user's report the next morning about an export job that never ran. Each group had checked its own part, and nobody had the job of deciding whether the release as a whole worked or should stop.

Name one release owner before the window opens, plus a second person with the same authority in case the first cannot be reached. For an application release this is usually the vendor's engineering lead, because the most likely reason to stop is a behavior only the vendor can judge. The owner makes the go, stop and rollback calls, and the people running each step report their results to that person.

If the pipeline runs on GitHub Actions, environments can enforce the approval. GitHub environments allow up to six required reviewers, a setting that stops whoever started a deployment from approving it, and a wait timer. On the Free, Pro and Team plans, required reviewers and wait timers work only in public repositories, so a vendor with private code on those plans has to record the approval somewhere else.

System access belongs in the same plan. One customer runs an application from another software vendor that we patch and maintain, and that vendor calls us when it needs access. For a release, record who grants the access, which account it uses, when it expires and who confirms it was removed.

Operating system patches are the other overlap. Microsoft publishes its monthly Windows security update on the second Tuesday of each month. If the host installs it on Thursday and the vendor ships a release on Friday, a failure found on Monday has two suspects. Agree a gap of a few days between the two, or run both in one window with a check after each.

Pin the artifact by its digest

An instruction to "deploy 4.2" can mean two different binaries. The Kubernetes documentation says plainly that a tag can be moved to point at a different image, and a build server that rebuilds 4.2 after a dependency update produces a new binary under the same name. Name the artifact by its content hash instead.

For a hypothetical service called billing-api at version 4.2.0, read the digest from the registry with docker buildx imagetools inspect registry.example.com/billing-api:4.2.0 and put registry.example.com/billing-api@sha256:9f2c41d0... in the Deployment manifest. After the rollout, kubectl get pods -l app=billing-api -o jsonpath='{..imageID}' shows what the nodes pulled. For a Windows installer, Get-FileHash .\BillingSetup-4.2.0.msi -Algorithm SHA256 gives the hash to record beside the version number.

Self-hosted installs run into the same ambiguity. Petrichor engineers join client calls for a software vendor whose customers must host its product themselves, and most of the customers running on AWS or Azure ask the same thing: which image are we supposed to use? A release record that names the image, its digest and the machine size it was tested on answers that question once for every customer.

Configuration gets the same treatment. Record the keys that change and a checksum of the file the host will deploy (sha256sum appsettings.Production.json), and refer to secrets by vault path or key name. A password pasted into a change ticket stays readable for as long as the ticket system keeps history.

Renaming a column across three versions

Some schema changes can be undone only by a restore, which is the slowest recovery path and the only one that loses data. Whether the previous version runs against the new schema decides which recovery options exist, so the vendor states it for every migration in the release record.

Splitting a breaking change across releases keeps rollback open. Renaming customer_name to billing_name in the billing service takes three. Version 4.2 adds and backfills the new column and writes to both, 4.3 switches reads to billing_name, and 4.4 stops using customer_name, with a migration that drops the NOT NULL constraint on that column so inserts that omit it succeed. The old column is dropped as a separate step once 4.4 has run cleanly on every customer stack.

Until that drop, each version can roll back to the one before it. After it, getting back to 4.3 takes a restore, because 4.3 still writes to the dropped column. The moment before the drop is the last safe rollback point for the whole rename, and the release record should name it.

A release that drops a column in the same version that stops reading it has one way back, and that is a restore. Petrichor engineers tell customers when a plan like that is a bad idea, and explain why.

What PostgreSQL and SQL Server lock during a schema change

On PostgreSQL, adding a column with a constant default only updates the catalog, while a volatile default such as clock_timestamp() rewrites the table and every index on it. CREATE INDEX CONCURRENTLY keeps writes flowing but cannot run inside a transaction block, so a migration tool that wraps each file in a transaction fails on it. A concurrent build that fails partway leaves an invalid index in place, which the planner ignores and every write still has to maintain.

To make a column on a large table NOT NULL without a long lock, add CHECK (billing_name IS NOT NULL) NOT VALID, then run VALIDATE CONSTRAINT, which takes a SHARE UPDATE EXCLUSIVE lock and lets writes continue. On PostgreSQL 12 and later, SET NOT NULL skips the full table scan once that constraint is validated.

In the rename, that constraint waits for the 4.3 migration. Added in 4.2, it would reject every insert from 4.1, both from pods still serving during the rollout and after a rollback.

SQL Server has a trap for vendors who test on Developer edition. Developer edition includes every Enterprise feature, online index create and rebuild among them, and Standard edition lacks that feature in both SQL Server 2022 and 2025. A migration written with WITH (ONLINE = ON) passes QA and then fails on a customer's Standard instance in the middle of the window. SQL Server 2025 adds a Standard Developer edition with Standard's feature set, so a QA stack built on it fails the way the customer's server would, days before the window. Where Petrichor builds QA stacks from a configuration the customer's QA lead approves, that configuration is where to pin the SQL Server edition your customers license.

Time the backfill before choosing the window. A table of 40 million rows backfilled in batches of 5,000 takes 8,000 batches, and at 200 milliseconds a batch that comes to about 27 minutes of steady write load on the primary, before any retries. If the last restore test took 40 minutes, a one hour window cannot hold both the backfill and a restore. Run the backfill in its own window after the release, which works because 4.2 still reads customer_name.

Keep one row per customer deployment

Some software companies we host give each of their own customers a separate stack. Microsoft's architecture guidance calls this the deployment stamps pattern, and one reason it gives for using it is hosting some tenants on different versions at the same time. That freedom produces drift. After a year of hotfixes and customers who asked to wait, no two stacks match, and support starts every investigation by working out which version it is looking at.

Each release updates the row for every stack it touches: customer, stack, application digest, schema version, configuration checksum, operating system and database versions, approved exceptions with an expiry date, and the ticket for the last change. Read the schema version from the database itself. flyway info lists applied and pending migrations, and on EF Core with SQL Server, SELECT TOP 1 MigrationId FROM __EFMigrationsHistory ORDER BY MigrationId DESC returns the newest one applied.

An exception without a date turns into a second product line. "Harlow Freight stays on 4.1 until March 31, after its fiscal year closes" has an end. Written without the date, the same line commits you to supporting 4.1 for as long as Harlow Freight renews.

Retire stacks from the same record. A stack nobody has deployed to in six months keeps taking operating system patches and holding customer data, and it will appear in that customer's next security questionnaire.

Acceptance checks written as commands

Approving a window and accepting a release are separate decisions, made by different people. The customer approves the window. Infrastructure checks belong to the host's engineers: services started, disk and memory inside their limits, the release backup finished. Whether the application behaves is for the vendor, or the customer's own users, to confirm.

A running process proves little: a container can start, pass its liveness probe and fail the first request that touches the database. Petrichor's engineers run the same Salt, Ansible and Terraform tooling as everyone else in this business, and after a release one of them reads each check by hand, because automation can keep reporting a service as running after it has stopped working.

Agree the stop condition and its deadline before the window opens. "If the export check has not passed by 09:30, roll back" is a decision the release owner can make at 09:30. Without it, the call tends to come at 11:00, after users have written ninety minutes of data against the new schema.

Write each check as a command or a click path, with its expected result and the person who reads it. For billing-api 4.2.0:

• curl -fsS https://billing.example.com/healthz returns HTTP 200 and "version":"4.2.0". The version string catches old pods that are still receiving traffic from the load balancer.
• An invoice created in the customer's test account reaches the export queue, and the queue drains to zero within five minutes.
• The export job, started by hand, writes a file to the SFTP drop with the expected row count.

Recovery paths and the data each one loses

Each recovery path costs something different, and the release record says which ones stay open at each step.

Rollback redeploys the previous artifact and loses no data, provided the previous version runs against the current schema. helm rollback billing-api returns the Kubernetes objects to the previous revision and leaves the database alone. CRDs are the exception, because Helm does not upgrade or delete them, so a release that changed a CRD needs its own reverse step.

To roll forward, the vendor ships a fix on top of the release. Users keep the data they entered, and the plan depends on the vendor building and testing that fix inside the window.

A restore returns the database to a backup and discards every write made after it. Once a column drop or an irreversible data conversion has run, it is the only path left. In the billing_name rename, that line is the column drop; for a data conversion, it is the moment the conversion starts writing.

Take the release backup after writes stop, with the application in maintenance mode, so the backup and the migration start from the same data. Customers whose staff are developers file tickets with Petrichor for database work, backups and restores. For a release, ask for that ticket to name the engineer, the backup's identifier and the check that shows it completed.

Price the restore with your own numbers. Say the backup finishes at 07:55, the release goes live at 08:30, and a problem found at 10:00 forces a restore. If the last restore test took 40 minutes, the service is back around 10:40 at its 07:55 state, and every invoice entered between 08:30 and 10:00 has to be entered again or replayed from another system.

Rehearse the recovery path on a restored copy of production, with the vendor's developers watching and every step timed. We ran that kind of rehearsal for a point of sale software company before its cutover: our engineers designed a high availability environment, presented it, and tested the migration repeatedly with the company's developers.

When the pipeline already holds the record

Skip most of this when one team writes and runs the software, every customer runs the same version, and the deploy pipeline stores the artifact, the approvals and the check results. The pipeline history is the release record, and a second document would drift away from it.

A team whose rollback comes from a managed service already has most of the recovery section covered. Azure App Service deployment slots, available on Standard tier plans and above, warm the new version in a staging slot and swap it into production, and swapping again brings back the last known good version. AWS CodeDeploy's blue/green deployments on EC2 can route traffic back to the original instances as long as they have not been terminated; on its EC2/On-Premises platform, blue/green works with EC2 instances only, and servers outside AWS can only be updated in place. If one of these tools carries your rollback today, moving to any other host, Petrichor included, means rebuilding that swap before the first release. Put that work in the migration estimate.

Bring the plan for your upcoming release to a call with a consultant, and they will mark which of its steps Petrichor's engineers would take on.