An API test finishes with a mean response time of 97.5 ms. That sounds
respectable. It may describe an API where almost every request took 100 ms.
It may also describe 95 requests that took 50 ms and five that took a full
second.
The arithmetic is correct in both cases. The user experience is not the same.
Latency is a distribution produced by a workload, not a single property of an endpoint.
Once that idea lands, performance testing changes. The job is no longer to make one number smaller. It is to understand the shape of the system as demand, data, dependencies, and failure interact.
Begin with the user-visible question
“How fast is this API?” is too broad to test.
A useful performance question identifies the operation, caller, conditions, and acceptable outcome:
Can an authenticated user retrieve the first page of their recent activity in under
300 msfor 95% of requests while the service receives 40 requests per second, with fewer than 0.1% failed requests?
That statement is not automatically the right objective. It is testable. It also exposes several decisions that a tool cannot make:
- Is server time or client-observed time important?
- Does the response have to be correct, or merely return a successful status?
- Is the target about typical use, a busy period, or an exceptional peak?
- Which users, payloads, and data sizes belong in the workload?
- What happens to requests that time out?
Google’s Site Reliability Engineering guidance recommends choosing a small number of service indicators that reflect what users care about. For a user-facing service, those often include availability, latency, and throughput. Correctness belongs in the conversation even when it is harder to reduce to an infrastructure metric.
Averages compress the interesting part
Order the response times from fastest to slowest and percentiles become easier to reason about:
p50is the middle observation: a useful description of the typical request.p95marks the point at which 95% of observations are at or below the value.p99brings the slower tail into view.
No percentile is automatically “the right one”. A p99 based on a tiny sample is unstable, while p50 alone can ignore a painful minority of requests. Report several points alongside the request count, error rate, and test conditions.
The earlier hypothetical dataset illustrates another trap:
95 requests × 50 ms = 4,750 ms
5 requests × 1,000 ms = 5,000 ms
mean = 9,750 ms / 100 requests
= 97.5 msThe slowest 5% consumed more total waiting time than the other 95 requests combined. The mean notices that something happened but does not explain the two populations. Looking at the distribution gives the investigation somewhere to begin.
Load and latency belong on the same graph
A latency result without its offered load is incomplete.
Many services look stable until a shared resource approaches saturation. A connection pool, worker queue, processor, database lock, or downstream rate limit can create a performance cliff: a small increase in arrival rate produces a large increase in waiting time.
This is why the following tests answer different questions:
- Smoke: Does the script work and does one user receive a correct response?
- Baseline: What happens under a small, stable workload?
- Expected load: Can the service meet its objective under representative demand?
- Stress: Where does behaviour begin to degrade?
- Recovery: After overload stops, does the service return to normal without intervention?
The point of stress testing is not to boast about the largest number reached. It is to find the transition from useful service to queueing, error, or collapse— and to see whether recovery is controlled.
Concurrent users are not an arrival rate
“100 virtual users” does not necessarily mean 100 requests per second.
If each simulated user waits for a response and then pauses, a slower service can reduce the rate at which the test sends new work. The measured load becomes partly controlled by the system under test. This closed-loop model can be appropriate for an interactive workflow, but it may hide what happens when work arrives independently.
An open workload model schedules arrivals at a target rate. If the service cannot keep up, work queues or the load generator begins to miss its schedule. That behaviour may resemble webhooks, messages, or externally driven traffic more closely.
State which model the test uses. “100 users” and “100 requests per second” are different claims.
Decide where the clock starts
Server instrumentation can report time spent handling a request. A load generator observes network travel, connection management, TLS, proxies, and the server. A browser may also include application work before the result becomes useful.
None of these boundaries is universally correct. They answer different questions.
OpenTelemetry’s current HTTP semantic conventions define separate client and server request-duration metrics and recommend recording server duration as a histogram. That separation is useful: it prevents “API latency” from becoming an accidental mixture of incompatible clocks.
Record:
- where the measurement begins and ends;
- whether connections are reused;
- whether DNS and TLS setup are included;
- the load generator’s network position;
- clock and unit conventions;
- any proxy, gateway, or cache in the path.
If a server trace says 40 ms and the client sees 240 ms, neither number
invalidates the other. The difference is evidence.
Treat errors and timeouts as performance results
A system can improve its reported latency by failing slow requests early—or by excluding them from the calculation.
Always place latency beside:
- attempted and completed request counts;
- status-code or error-class breakdown;
- timeout count and timeout threshold;
- response validation failures;
- work abandoned by the load generator.
A fast 500 response does not satisfy a successful-request objective. A timed
out request has user-visible latency even if it never appears in the successful
request histogram.
Validate something meaningful in the response. A cached error page and the correct account data can share an HTTP status if the test only checks that the request completed.
Control the data, then disturb it deliberately
Performance depends on state.
A test using one account and one object may measure a hot cache. Randomising everything may instead create a workload unlike any real user. Large accounts, cold starts, recently written records, permission checks, and expensive search terms can each form distinct populations.
Begin with a documented dataset and representative request mix. Then isolate the conditions likely to produce slow paths:
- warm cache versus cold cache;
- small versus large result sets;
- read-only traffic versus concurrent writes;
- one tenant versus many;
- healthy dependencies versus a deliberately slow dependency.
Do not stir every variable at once. A workload should be realistic enough to matter and controlled enough to explain.
Publish a test contract
A result becomes reusable when another person can understand the contract that produced it:
- service and dependency versions;
- environment and resource limits;
- endpoint, payload, authentication, and dataset;
- load model, rate, concurrency, and request mix;
- warm-up, test duration, and number of observations;
- measurement location and timing boundary;
- latency percentiles, throughput, errors, and correctness checks;
- relevant server saturation or queue metrics;
- known differences from production.
Keep the test script with the result when possible. Grafana k6, for example, supports thresholds that can make a test fail when a metric crosses a defined criterion. The tool is useful; the explicit contract is what makes its output evidence.
The useful insight
An API is not “a 97.5 ms API”. It is a system that produces a distribution of outcomes under particular conditions.
The average is not wrong. It is simply too compressed to carry the whole argument. Put the workload, distribution, errors, correctness, and measurement boundary back beside it, and performance testing becomes an explanation rather than a screenshot.
