How long to run a soak test

Average load, held for hours

Grafana's k6 documentation treats the soak test as an average load test with a longer plateau, where load rises to an average level and stays there for several hours or even days. Typical lengths on that page are 3, 4, 8, 12, 24, and 48 to 72 hours, and its sample script holds 100 virtual users for 8 hours. It names the defects that show only after extended use: response time degradation, memory or other resource leaks, data saturation and storage depletion.

Grafana also sets the order of work. Smoke and average load tests come first, because each test type uncovers different problems. After those pass, the soak reuses the average load script where possible, with a much longer duration.

Peak load belongs to a different test. A soak run at peak asks two questions at once, and a failure cannot be assigned to the load or to the hours.

Set the length from the slowest cycle in the system

Those typical values are a starting list. For one system, the number that fits is the longest gap between events that change its state, plus enough time afterward to watch the system settle.

One row of that table can be worked out ahead of time. Take a hypothetical booking product, Thornwick Booking, with a reservations table of 10 million rows. At the defaults, the threshold is 2,000,050 dead rows. If average load updates that table 40 times a second, that is 144,000 updates an hour, and the table crosses the threshold no sooner than hour 14. An 8 hour soak ends before autovacuum has touched it.

A run from Friday at 6:00 p.m. to Monday at 6:00 a.m. lasts 60 hours, inside the 48 to 72 hour range k6 lists. A job scheduled for 1:00 a.m. runs three times in that window, and a Sunday job runs once.

Readings to keep every ten minutes

Backend monitoring is especially important for soak tests, in the words of the k6 page. A load tool reports latency and errors, and the servers have to report everything else. Sampling every ten minutes is enough to draw a slope.

# Every ten minutes, appended to one file per host.
# The process name and the mount point are placeholders.
# The psql lines need your own connection settings.
date -u +%FT%TZ
ps -C app-server -o pid=,etimes=,rss=
df --output=used /var
psql -At -c "select count(*) from pg_stat_activity"
psql -At -c "select relname, n_dead_tup, last_autovacuum
             from pg_stat_user_tables order by n_dead_tup desc limit 5"

rss is the resident set size, the physical memory the process holds, in kibibytes. etimes is the number of seconds since the process started, so a restart during the night shows up as that number dropping. df prints used space in 1K blocks. The two queries count open database connections, then show the tables with the most dead rows and the last time autovacuum reached each one.

Read the file as slopes. At Thornwick Booking, suppose the application process holds 1,200 MiB at hour 1 and 1,800 MiB at hour 13. That is 50 MiB an hour, and the line reaches a limit of 4,096 MiB at about hour 59. An 8 hour test would have ended at 1,550 MiB with every request passing.

Latency gets the same treatment. Compare the p95 of the first full hour with the p95 of the last hour at one arrival rate. A rise between the two is a finding even when the last hour still meets the budget.

Write the pass conditions before Friday

A load test passes or fails against a latency budget. The conditions for a soak have to be written down before the run starts, and four cover most products.

• No process restarts on any host, which etimes shows by rising for the whole run.
• Memory, connection count and disk use each flatten, or their slopes put the limit further away than the longest gap between planned restarts.
• The last hour's p95 sits within a stated margin of the first hour's.
• Every scheduled job inside the window started and finished.

Decide ahead of time whether a failed condition stops the run. Once a leak has shown its slope the remaining hours add little, and stopping early hands the stack back sooner.

Run the script for two hours on Thursday first. That rehearsal shows whether the script survives a token expiry, whether the sampling file fills in, and how many rows the test itself writes each hour. A script that creates a booking on every pass grows the reservations table all weekend, and the team should decide beforehand whether that growth belongs in the test.

A stack nobody borrows for 60 hours

For the whole window, a soak needs a stack that nobody deploys to, resets or borrows. Shared QA is due back for Monday's regression cycle, and 60 hours of synthetic load on production lands on customers.

On Petrichor, we build a performance environment for the test window and tear it down when you have your numbers. We build it from your production configuration, write down every difference and size it for the traffic you name. How long the environment should live is a line in the written spec, so a 60 hour hold is agreed before we build anything.

When the readings point at a bottleneck, our engineers help tune the operating system, database and network around it. We then rerun the test on that same environment, so the second file of readings can be laid over the first.

Growth too slow for one weekend to show

A leak of 5 MiB an hour adds 300 MiB across 60 hours, which can hide inside the ordinary swing of a process's memory. Its slope is still in the samples, and a second run at twice the length separates growth from swing.

Data volume is the other limit. One weekend at average load adds a weekend of rows, while production adds years of them. Storage depletion and slow queries on large tables need a dataset loaded at the size the product will reach, which is a separate test.

A separate environment also leaves out whatever only the production platform does. If production's database is a managed AWS service, tests of that service's own behavior stay on AWS. When to move QA off AWS ahead of production lists the AWS cases.

Talk to a consultant and bring your scheduled jobs with their run times, plus the arrival rate from your last average load test.