Performance budgets per release

Four lines in a release budget

The term comes from front end work. Google's web.dev defines a performance budget as "a set of limits imposed on metrics that affect site performance", and its examples are page weights and load times. A service behind an API needs different lines. Take a hypothetical invoicing product, Farrowdale Billing, and the budget its performance engineer keeps for the endpoint that pays an invoice.

Hold the arrival rate fixed. k6's constant arrival rate executor starts iterations independently of how the system responds, and its documentation recommends it when iterations should stay constant whatever the performance of the system under test. With a fixed pool of virtual users, a slower release receives fewer requests, and the two releases end up measured under different load.

Measure the run to run spread first

Before setting a margin, run one build five times on one environment and write down the p95 from each run. Suppose Farrowdale's five runs of release 4.3 read 262, 255, 269, 258 and 266 ms. The lowest and highest are 14 ms apart, which is 5.5 percent of the lowest.

A budget of "no more than 5 percent slower than the last release" would fail that build against itself on an unlucky pair of runs. The margin has to sit above the measured spread, and Farrowdale sets 10 percent. A regression smaller than the spread stays invisible to a single run. Seeing it takes several runs of each build and a comparison of their medians.

A margin wide enough to clear the noise is also wide enough to hide a drift. This is Farrowdale's year, one p95 per release.

Every release passes a 10 percent margin against the one before it. So the budget carries a second comparison, against an anchor release. Farrowdale's anchor is 4.2, and the rule is 25 percent over its 240 ms, which is 300 ms. Release 4.4 measures 287 and passes. At 313, release 4.5 fails, with 187 ms of the fixed limit still unspent.

The last release and the candidate, back to back

On file for release 4.3 is a p95 of 262 ms, measured weeks ago. Since then the test environment has been patched or rebuilt, and the dataset has grown. Comparing the candidate with that stored number measures the environment's changes along with the code's.

Run both builds in one window. Install release 4.3, restore the dataset, run the script and save the results. Then do the same for the 4.4 candidate: install it, restore the dataset again and run the same script at the same arrival rate. The restore between runs matters because the first run leaves the database larger and its cache warm, and the second build would inherit both. If the window allows four runs, alternate the builds so neither one always goes second.

The baseline run also produces the numbers for the script. k6 calls its pass and fail criteria thresholds, and it exits with a non zero code when one fails, which is what stops a pipeline. The limits below are the baseline's 262 ms p95 and its 555 ms p99, each plus 10 percent.

export const options = {
  scenarios: {
    budget: {
      executor: "constant-arrival-rate",
      rate: 120,
      timeUnit: "1s",
      duration: "20m",
      preAllocatedVUs: 200,
      maxVUs: 400,
    },
  },
  thresholds: {
    http_req_failed: ["rate<0.01"],
    "http_req_duration{name:pay_invoice}": ["p(95)<288", "p(99)<610"],
  },
};

In the script, the payment request carries the tag name: pay_invoice, and k6 turns a tagged request into a sub metric that a threshold can name. Both numbers are written by the pipeline from the baseline run, so nobody edits them by hand between releases.

Queries per request, counted by the database

Latency carries noise. A count of statements carries almost none when the script and the dataset are fixed, which makes it the line that catches a small regression first.

PostgreSQL's pg_stat_statements module keeps the count. Its calls column is the number of times each statement was executed. The module has to be listed in shared_preload_libraries, which takes a server restart, so ask for it when the environment is specified.

-- Before the run. Superusers only by default.
select pg_stat_statements_reset();

-- After the run.
select sum(calls) from pg_stat_statements;
select calls, query from pg_stat_statements order by calls desc limit 10;

Twenty minutes at 120 requests a second is 144,000 requests. If release 4.3 executes 1,728,000 statements, that is 12 per request. A candidate that executes 2,160,000 is at 15. The last query lists statements by count, with constants replaced by parameter symbols, and the three new ones sit near the top. Run the count on a database that only the test touches, so a monitoring session's own statements stay out of the sum.

A named person accepts each overage

A line over budget stops the release, and from there it goes one of two ways. The team fixes the regression and reruns both builds, or a named person accepts the overage in writing, with the reason and the measured numbers saved beside the results. An accepted overage still counts against the anchor line, so the next release starts with less room.

The anchor moves in one direction without a meeting. When a release comes in faster than the anchor, that release becomes the anchor and the gain is kept. Raising the anchor is a decision for whoever owns the product's response time commitments, made with the table above in front of them.

One environment for both runs

Both runs need one set of servers with identical settings and one dataset, held for the hours the comparison takes. On Petrichor, we build a performance environment for the test window and tear it down when you have your numbers. We build it from your production configuration, write down every difference and size it for the traffic you name. The next release's test gets an environment built for its own window, which is the reason to run the last release again and leave the stored number in the file.

When a line fails and the cause sits below the application, our engineers help tune the operating system, database and network around the bottleneck. We then rerun the test on that same environment, so the change shows as a number beside the first run.

A budget measured this way gives the difference between two releases. The figure customers see in production differs by whatever is on the written list of differences, and What a passing test proves in dev, QA and staging starts that list with the database edition. A 20 minute run also says nothing about hour 14, when a slow leak or the first vacuum of a large table shows up. That takes a longer test, held at average load.

Talk to a consultant and bring your last three load test reports, with the build each one ran against and the environment it ran on.