Layout
PART II · Measurement · CONCEPT 6 OF 24

The average buyer is not the buyer stuck waiting.

EXPLORE~7 min
Back to journey
  • A cache can make most requests faster while a few buyers wait through retries.
  • Mean latency, median latency, and p95 latency describe different parts of that experience.
Latency & tails: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
A percentile is a position, not an average.
  • p95 is a threshold at or below which about 95% of measured latencies lie.
  • It is not the average of the slowest 5%.
  • Define the event and sampling convention.

Northstar's p95 page latency is a threshold for roughly 95% of measured requests, not the average latency of the slowest 5%.

Timeouts must not vanish.
  • A successful-request-only chart can improve because the slowest requests now fail.
  • Track timeouts and errors alongside latency, and predefine how incomplete requests are represented.

If Northstar's slowest requests time out, a successful-request-only latency chart may improve while the full buyer experience gets worse.

Match inference to the metric.
  • A request-weighted p95 gives heavy shoppers more weight.
  • A buyer-weighted metric answers a different question.
  • Quantile uncertainty needs a suitable method, such as buyer-cluster bootstrap, not a mean’s standard-error formula.

Northstar must distinguish request-weighted p95 from a buyer-weighted measure and use uncertainty methods that respect repeated requests per buyer.

LAB 06

Latency & tails · Northstar

Sorted request latency in milliseconds-55.6583.81223.21862.6FastestMedianp95
Control · msTreatment · ms
Mean change-64 ms
Median change-100 ms
p95 change500 ms

Each line is 100 constructed requests sorted by latency, not time. Increase the tail penalty: most requests improve while the slowest deteriorate. Quantiles use the nearest-rank rule. No quantile confidence interval is estimated here; real inference must account for repeated requests per buyer and failures.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Successful requests got faster but timeouts doubled. What does that show?

Your learning note

Private · just for you