精读预计 3 分钟

Alice is impatient

摘要

文章用 Alice（请求延迟）和 Alex（故障恢复）两个例子说明同一系统的两种时间感知差异：服务端按请求或事件计平均值，但用户按真实等待时长体验。由于 inspection paradox（检查悖论），用户感受到的是按时间加权后的分布，长请求/长故障会被更频繁 “遇到”，数学上对应 E [X^2]/E [X]，显著高于简单均值。作者进一步用模拟说明，即使 MTTR 或平均延迟看起来不高，只要尾部很重，用户体验的平均等待仍会大幅变长。文章强调尾延迟和恢复时间的右尾分布对体验影响更大，并指出 trimmed mean 等截断指标会丢失关键尾部信息，尤其在无法通过重试隐藏的恢复场景中更为明显。

荐读理由

能直接用上的方法 / 工具 / 架构（可迁移到我手上的项目）：根据正文摘要和完整摘录的实质，你能将检查悖论的思想应用到自身项目，用它揭示当前服务指标（如MTTR或均延迟）与客户真实感知时间（E[X^2]/E[X]）之间的高度不匹配，作为工程决策的直接支点。

原文

Marc's Blog

About Me

My name is Marc Brooker. I like to build things that work, and do cool stuff. I like building big things. I also dabble in machining, welding, cooking, and skiing.

I am an engineer at Amazon Web Services (AWS) in Seattle, where I work on agentic AI, especially safety and policy for agentic AI. Before that, I worked on EC2, EBS, databases, serverless, and serverless databases. All opinions are my own.

Links

My Publications and Videos @marcbrooker on Mastodon @MarcJBrooker on Twitter

Is this blog written by AI?

Meet Alice. Alice is impatient.

What do you mean?

Meet Alice. Alice uses your web service. Alice, like most humans, measures her time in seconds and minutes. Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s.

You’re both right.

Meet Alex. Alex uses your web service. Alex, like most humans, measures his time in seconds and minutes. Alex says that when you have outages, they last a long time and he gets really annoyed. You tell Alex that your MTTR is less than 1 minute. Alex says that he sees the mean outage lasting 1 hour.

Again, you’re both right.

What’s going on? What’s going on is that you’re measuring time in requests, or in outages, and Alex and Alice are measuring time in seconds and minutes. When you have a long request or a long outage, Alex and Alice count that as a long time, with a heavy weight. But you only count that as one.

More technically, what’s going on here is the inspection paradox. Alex and Alice don’t experience your latency distribution $f(t)$, they experience a t-weighted version of it. If you have a MTTR or mean request time of $\mathbb{E}[X]$, Alex and Alice experience $\mathbb{E}_a[X] = \frac{\mathbb{E}[X^2]}{\mathbb{E}[X]} = \mathbb{E}[X] + \frac{\mathrm{Var}(X)}{\mathbb{E}[X]}$.

Most of the time they’re waiting, they’re waiting for things that take a long time. This is (roughly) how humans experience time.

Let’s play with this with a little simulation. Plug in your median latency (or recovery time), and 99th percentile latency (or recovery time), we’ll fit a log-normal distribution to it, and then plot both what your service metrics see and what your customers see.

Median: ms p99: ms

What your service sees (mean): – ms. What your customers experience (mean): – ms.

For example, put in 30 as the median (let’s ignore the milliseconds and pretend these are minutes for now) for a 30 minute Median TTR (i.e. in half of your postmortems you see a recovery time of $\leq 30$ minutes), and 600 in as the p99 (one in every 100 events, recovery takes 10 hours). Your MTTR is just over an hour. Your customers experience a mean time to recovery of around 6 hours!

There are many arguments for why tail latency (and long recovery times) are so important to understand (e.g. multiple samples), but this is the one that I think is the least widely understood. For service times, timeout-and-retry can hide this latency some of the time (as long as the running request doesn’t hold locks or other exclusive resources). But, for recovery time, no such hiding is possible. The heaviness if the tail matters a great deal. This is also one of the reasons I don’t like trimmed measurements (like trimmed means) as a way of thinking about service latency or recovery time. They throw out some really critical context about the shape of the right tail that dominates the customer experience (the other reason is related to Little’s Law and capacity usage, which I’ve written about before).

A note on log-normal: I chose log-normal here for numerical convenience. It has the nice property that $\mathrm{lognormal}(\mu, \sigma^2)$ becomes $\mathrm{lognormal}(\mu + \sigma^2, \sigma^2)$. Also it’s well-behaved around 0. I don’t believe that log-normal is a particularly good choice of distribution for latency or recovery time metrics, and generally would approach these problems entirely non-parametrically.

« Back to the blog index