When to move QA off AWS ahead of production

Five signs a QA environment should go first

Petrichor's engineers start most new customers on QA. QA stacks turn over more often than dev stacks, because every validation and regression run wants a fresh copy with a specific dataset. Because QA is usually the least developed stack, pulling back costs little. We host production too, and a customer who offers it on the first day gets slowed down while we learn how its systems run.

Frequent rebuilds make setup expensive wherever QA runs. Outside AWS, each rebuild also repeats the transfer out of AWS, so the rebuild count sets the transfer bill too.

Failing sign two or three raises the cost of the arrangement. With the database left on AWS, sign four fails on speed instead, because every query crosses the link. Measure the round trip from the QA side before anything moves. In this command, time_connect minus time_namelookup is the TCP handshake, which takes one round trip:

curl -so /dev/null -w '%{time_namelookup} %{time_connect}\n' https://sqs.us-east-1.amazonaws.com/

At a 30 ms round trip, an API call that issues 200 sequential queries spends 6 seconds waiting on the network, and client timeouts tuned inside the VPC start failing intermittently. Move the QA database with the application servers, and leave only bulk or asynchronous traffic on the link.

When sign five fails, the discussion is over, because a network path cannot stand in for a service QA exists to test. List each AWS service a QA component calls in the Depends on column of the workload sizing sheet before pricing anything.

Passing all five means QA can run outside AWS without a large new line on the AWS bill. Whether the total drops depends on what the QA side costs.

Split the bill by environment in Cost Explorer

One Petrichor customer employs a person whose only job is reviewing its AWS invoice, and that invoice work is one reason it began hosting dev and QA with us. Non-production environment costs come out of an AWS bill in two ways. Where QA has its own AWS accounts, group by linked account. End is exclusive, so this covers June through August 2026:

aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-09-01 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --group-by Type=DIMENSION,Key=LINKED_ACCOUNT

Where production and QA share accounts, swap the --group-by argument for Type=TAG,Key=environment. Tags count only from activation, and activation is slow. A new tag key can take up to 24 hours to appear for activation and up to 24 more to activate. Older months stay untagged unless the management account requests a backfill, which reaches back up to twelve months and fills only the months when a resource carried the tag. Shared QA spend usually sits in the untagged bucket.

Next, check how much internet transfer already leaves AWS, because the free 100 GB a month is shared across every AWS service and Region:

aws ce get-cost-and-usage \
  --time-period Start=2026-08-01,End=2026-09-01 \
  --granularity MONTHLY \
  --metrics UsageQuantity UnblendedCost \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --filter '{"Dimensions":{"Key":"USAGE_TYPE","Values":["Global-DataTransfer-Out-Bytes","DataTransfer-Out-Bytes","USW2-DataTransfer-Out-Bytes"]}}'

The free allowance bills as Global-DataTransfer-Out-Bytes. Internet transfer from US East (N. Virginia) is DataTransfer-Out-Bytes with no prefix, US West (Oregon) is USW2-DataTransfer-Out-Bytes, and every other Region has its own prefix, so list the Regions in use first. Use exact names: the Cost Explorer API honors match options such as ENDS_WITH only for cost category and anomaly subscription calls. When the global line reads 100 GB, the allowance is spent, and QA traffic from outside AWS bills from its first GB.

Three costs never show up in an environment split:

• Shared networking with no environment tag. A NAT gateway in US East (N. Virginia) bills $0.045 an hour plus $0.045 per GB processed, whichever environment sent the traffic.
• A tooling account holding the CI runners and the image registry. ECR storage bills $0.10 per GB-month, and every QA image lives there.
• Support, billed as a percentage of the whole bill. Business Support+ charges 9 percent of the first $10,000 of monthly AWS charges and 7 percent from $10,000 to $80,000, with a $29 minimum per account. A $60,000 month carries a $4,400 fee. The same month without a $12,000 QA line carries $3,560, so $840 of the monthly support fee belongs to QA.

An account split is cleaner than a tag split, and neither one shows engineer time.

What still calls back to AWS

A QA environment outside AWS can still reach back for the database, images, fixtures, queues, secrets, and sign-in. Each dependency has a direction and a price, and most have a failure mode to plan for before the first rebuild.

ECR tokens expire after 12 hours

Docker gets its token from this login, shown with a placeholder account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

A build agent that logs in once at boot passes its daytime jobs, then fails the overnight rebuild at the pull step with an authorization error. Run the login as the first step of every job, or install the ECR credential helper, which manages the token for Docker and does not support MFA.

A pull touches two services. The client fetches the manifest from ECR and the layers from an S3 bucket AWS keeps for ECR in each Region, so a firewall rule or private route written for the ECR hostname alone fails at the first layer. Mirroring each release to a registry on the QA side leaves one login and one set of rules per release, in the mirror job, and cuts the transfer as well.

S3 reads billed as internet transfer

Inside AWS, a read from an S3 bucket by any service in the same Region carries no transfer charge, even from another account. From outside AWS, every byte read is internet transfer. AWS's price list sets that at $0.09 per GB for the first 10 TB a month from US East (N. Virginia), after a free 100 GB aggregated across all AWS services and Regions. Writes into AWS are free to transfer, so results flowing back stay cheap.

If QA runs on access keys that never expire

The usual shortcut is an access key pasted into an environment file on the QA side to reach S3 or SQS, copied into CI, and never rotated. IAM Roles Anywhere replaces it: a server presents an X.509 certificate from a CA you registered as a trust anchor and receives temporary credentials. The trust boundary is the whole account. Certificates from any trust anchor in the account can assume any role in it unless the role's trust policy adds conditions, so tie each role to its anchor with the aws:SourceArn condition key. Keep everything QA touches in an AWS account that holds no production data, so a leaked credential reaches only test resources. The price is a certificate authority to run, or AWS Private CA to pay for.

Egress allowlists for self-hosted runners

Self-hosted GitHub Actions runners on the QA side need outbound HTTPS on port 443 to four groups of domains: github.com, api.github.com and *.actions.githubusercontent.com for jobs; codeload.github.com for downloading actions; results-receiver.actions.githubusercontent.com and *.blob.core.windows.net for artifacts, logs and caches; and a separate set, including objects.githubusercontent.com, for runner updates. An allowlist built by watching the first week of traffic misses that last set if no runner release shipped that week. The runner then cannot update itself, and GitHub stops queuing jobs to a runner that has gone 30 days without a software update. Weeks after the environment goes live, the outage lands far from the firewall change that caused it.

Why the test dataset belongs on the QA side

Where the dataset lives decides the transfer bill, and request charges barely register beside it. Take Kestrel Ledger, a hypothetical accounts payable SaaS company that rebuilds QA 40 times a month. Each build pulls a 120 GB dataset of 60,000 objects from S3 plus 6 GB of images and build artifacts, and production already spends the free 100 GB.

The GET requests come to 40 × 60,000 = 2.4 million a month, or $0.96 at $0.0004 per 1,000. Transfer carries the bill.

Petrichor's provisioning sequence builds a stack in order: servers with the right operating systems, the software, the environment configuration, the customer's application, and last the base users and the dataset, before anyone logs in. The dataset loads in every build, so the configuration your QA lead approves is the place to name the staged copy as its source, and every rebuild then reads from the QA side.

Your team controls which options appear in that configuration, and configurations stay editable. Petrichor builds on the automation tools any shop can run: Salt, Ansible and Terraform. Our engineers add the layer above them, setting up each customer's options around that team's own variables. We run the platform through an API with command line wrappers, and a person watches it, because automation can fail without saying so. On one software vendor's calls with customers who host their own applications on AWS or Azure, most of those customers ask which image they are supposed to use. A configuration settles that once, since the customer already chose what goes in it. Development and QA environments sets out which setup work Petrichor engineers take and which stays with your team.

The exit waiver will not cover the dataset transfer. AWS's EC2 pricing page offers eligible customers free internet transfer when they move all of their data off AWS, or all of their data off a particular AWS service. Moving only QA leaves production data in the same services, so on a plain reading the waiver does not apply. Budget the transfer.

Staged copies age between refreshes. A regression that depends on this week's production shape needs more of them, and a full refresh is another 120 GB out of AWS, or $10.80. Refresh with aws s3 sync, which copies only new and updated objects. If a tenth of Kestrel Ledger's objects are rewritten in a week, a refresh moves about 12 GB. A dataset re-exported in full, such as a nightly database dump, carries a new timestamp on every object, so sync copies all of it. Copies built from production data also need the data owner's approval for where they sit and who can read them, which Reset QA for the next test covers.

Site-to-Site VPN, Direct Connect or public endpoints?

QA traffic out of AWS is almost all S3 reads and ECR layer pulls. For that traffic, a Site-to-Site VPN costs less than a 1 Gbps Direct Connect port below 3,285 GB a month, before the port's cross-connect fees.

Direct Connect's $0.02 rate applies only to bytes that cross the port. S3 and ECR layer downloads resolve to public AWS addresses, so they keep taking the internet at $0.09 unless a public virtual interface carries them or an S3 interface endpoint answers for them. A gateway endpoint cannot do it, because gateway endpoints accept no traffic from on-premises networks. An interface endpoint bills $0.01 an hour in each Availability Zone plus $0.01 per GB processed, so two zones add $14.60 a month and raise each S3 byte over the port to $0.03.

The QA side also has to resolve S3 names to the endpoint, or pulls keep resolving to public addresses, cross the internet at $0.09, and the $0.03 rate never applies. A Route 53 Resolver inbound endpoint handles that at $0.125 an hour per network interface, with two at minimum, or $182.50 a month, which would move the break-even worked out below from 3,285 GB to about 6,327 GB. The figures here assume the cheaper route: a zone on the QA side's own DNS that answers for s3.us-east-1.amazonaws.com and its subdomains with the endpoint's private addresses, which the endpoint-specific name returns from public DNS. ECR redirects layer downloads to its own S3 bucket, so run one docker pull with the resolver's query log open and confirm every S3 name it asks for comes back private.

With the endpoint, the port costs $197.10 a month more than the VPN and saves $0.06 per GB, so it breaks even at 197.10 / 0.06 = 3,285 GB a month. Traffic addressed to the VPC itself, such as query results from a database left on AWS, crosses at $0.02 and breaks even sooner, at 182.50 / 0.07 = 2,607 GB. Applied to Kestrel Ledger's S3 and ECR traffic:

Two build patterns at Kestrel Ledger

Staging the dataset flips the answer to the VPN.

Setting the MSS on each tunnel

The tunnel does not support Path MTU Discovery. Without it, SSH sessions and small API calls work while image pulls and large query results stall. Set the MSS by algorithm and NAT traversal: 1406 for AES-GCM without NAT traversal, 1398 with it, and 1382 for AES-CBC with SHA2-256 and NAT traversal. A Linux gateway clamps it on the forward chain:

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1382

Fragment packets that still exceed the MTU on the customer gateway, before encryption. Bring up both tunnels as well. Each VPN connection has two, and when AWS takes one down for maintenance, traffic fails over only if the customer gateway has the other one established.

A standard tunnel also caps each dataset refresh. At 1.25 Gbps, 120 GB takes 768 seconds of pure transfer (120 × 8 / 1.25), about 13 minutes, and realized throughput runs below the ceiling. Large Bandwidth Tunnels raise the ceiling to 5 Gbps, and they attach only to a transit gateway or Cloud WAN.

Digest pinning across both providers

Builds stay comparable when both sides pull one artifact by digest and every QA stack comes from one approved configuration. Pull by digest in both places:

docker pull registry.example.com/kestrel/api@sha256:<digest>

docker buildx imagetools inspect registry.example.com/kestrel/api:2026.09.3 prints the digest a tag points to, with a separate digest for each platform when the image is built for several. Development, QA and staging explains why a moving tag breaks release evidence.

Check CPU architecture next. Graviton instances run arm64, so if production runs on Graviton and the QA side runs x86_64, one tag built for both platforms resolves to two different images, and a pass in QA covers the amd64 image while production runs the arm64 one. Record which platform QA ran, or build and test both. Match the database engine's major and minor version between RDS and the engine in QA, along with the parameters that change query plans. RDS is managed, so exact parity off AWS is out of reach, and the test report should name the gap.

Teams better served leaving QA in place

• QA that tests AWS itself: IAM and VPC endpoint policies, Lambda event sources, Step Functions, DynamoDB streams, or a Terraform plan against the AWS provider. An application built on managed services it cannot replace belongs where those services run.
• Staging that rehearses the release into production. A staging environment outside AWS cannot prove a deployment into AWS, so staging often stays after QA leaves.
• Regression suites that need a fresh copy of production at full size on every run. At 120 GB a copy, the dataset arithmetic above turns against leaving.
• A bill driven by idle hours, which a scheduler inside AWS can fix. AWS says Instance Scheduler can cut costs up to 70 percent for instances needed only during business hours, with weekly running time dropping from 168 hours to 50, and estimates $13.15 a month to run it in US East (N. Virginia). The limits sit on the database side: a stopped RDS instance starts itself after seven consecutive days, storage and backups keep billing while it is stopped, and RDS for SQL Server in a Multi-AZ deployment cannot be stopped.
• Nobody owning the path back. The tunnel, the runner allowlist and the credentials each need a named owner on one side or the other, or the arrangement adds a failure domain nobody watches.

Development servers, once the first stack holds up

The customer that pays someone to review its AWS invoice started with QA servers, proved them out, then added development servers, and it is expanding further with its leadership now involved. Before a team commits, a Petrichor engineer can assess its QA stack and stand up a version of it on hardware we own, so the team sees how it runs.

Book a call with a consultant and bring the QA environment you would move first.