PART II · Measurement · CONCEPT 6 OF 24
The average buyer is not the buyer stuck waiting.
EXPLORE~7 min
Back to journey- A cache can make most requests faster while a few buyers wait through retries.
- Mean latency, median latency, and p95 latency describe different parts of that experience.
| Concept | What it means | Northstar example |
|---|---|---|
| A percentile is a position, not an average. |
| Northstar's p95 page latency is a threshold for roughly 95% of measured requests, not the average latency of the slowest 5%. |
| Timeouts must not vanish. |
| If Northstar's slowest requests time out, a successful-request-only latency chart may improve while the full buyer experience gets worse. |
| Match inference to the metric. |
| Northstar must distinguish request-weighted p95 from a buyer-weighted measure and use uncertainty methods that respect repeated requests per buyer. |
LAB 06
Latency & tails · Northstar
Control · msTreatment · ms
Mean change-64 ms
Median change-100 ms
p95 change500 ms
Each line is 100 constructed requests sorted by latency, not time. Increase the tail penalty: most requests improve while the slowest deteriorate. Quantiles use the nearest-rank rule. No quantile confidence interval is estimated here; real inference must account for repeated requests per buyer and failures.
Constructed teaching example · not Amazon data or a production experimentation engine
Successful requests got faster but timeouts doubled. What does that show?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗