Find out where your latency actually goes

Something in your app feels slow. You have a theory about why. This guide replaces that theory with a number.

At the end you will have a per-hop latency budget for your own system: a table showing exactly how many milliseconds each layer contributes, summing to the delay your users actually feel. You will also have a physics check that tells you whether the remaining time is a routing problem you can fix or a distance problem you cannot.

The method took me about an hour on a system I had been mis-diagnosing for weeks. I was certain the bottleneck was rendering. Rendering turned out to be 0.25% of it.

Prerequisites

• An app with a round trip you can trigger on demand (a keystroke, a click, an API call)
• Shell access to at least one machine in the path
• ping, curl, traceroute, and dig on your local machine
• A browser with a JavaScript console, if the client is a browser
• Roughly one hour

You do not need a profiler, an APM vendor, or a tracing framework. Every number below comes from tools already on your machine.

The rule that makes this work

Measure the whole thing first, then subtract.

Most performance work runs the other way. You suspect a layer, you instrument that layer, you find something slow in it, and you optimize it. This works only if your suspicion was right. When it is wrong, you can spend a week making a 1 ms operation take 0.4 ms and change nothing a user notices.

Measuring the total first gives you a denominator. Every layer you time afterward becomes a percentage of something real.

The system in this walkthrough

The worked example is TermStream, a browser-based terminal I have been building. The details of the product do not matter. What matters is the shape, which is extremely common:

browser  ->  WebSocket  ->  app server  ->  child process  ->  back

Swap in your own stack. A React app calling an API. A game client talking to a server. Anything where a user action leaves the machine and comes back.

The symptom: typing felt heavy and laggy. The obvious suspect was the renderer, because the terminal used a DOM-based renderer instead of a GPU one, and every article about fast terminals is about GPU rendering.

Step 1: Measure what the user actually feels

Time the complete round trip, from input event to visible pixel change. Nothing else counts until you have this number.

In a browser, hook the input event and watch the DOM for the response:

const marks = { keys: [], paints: [] };
 
window.addEventListener('keydown', (e) => {
  if (e.isTrusted) marks.keys.push(performance.now());
}, true);
 
new MutationObserver(() => marks.paints.push(performance.now()))
  .observe(document.querySelector('#your-output-container'), {
    childList: true, subtree: true, characterData: true,
  });

Then type one character and read the pair.

Caution: use real input events, not synthetic ones. A dispatchEvent(new KeyboardEvent(...)) is not trusted, and many apps ignore it. I lost twenty minutes to a measurement that produced zero samples for exactly this reason. Drive it from the keyboard, or from a browser automation tool that uses the debugger protocol.

Verify: you should get one paint timestamp per keystroke. If paints outnumber keystrokes, your observer is catching unrelated DOM churn. Narrow the observed element.

What I got, three consecutive keystrokes:

keydown -> visible echo:   893 ms
                          1696 ms
                          2834 ms

That is the number to explain. Everything from here is subtraction.

Note the second thing those samples show: the values are not just large, they are inconsistent. Hold onto that. Spread matters as much as magnitude, and Step 6 is where it gets its own treatment.

Step 2: Time the client's own work

You control this code, so measure it directly. Patch the send call and compare against the input timestamp:

const origSend = WebSocket.prototype.send;
WebSocket.prototype.send = function (data) {
  marks.sends.push(performance.now());
  return origSend.call(this, data);
};

Verify: every keystroke should produce exactly one send. If you see extra sends, you are catching heartbeats or acks; filter by payload.

Result:

keydown -> WebSocket.send:   0.1 ms
                             0.2 ms
                             0.3 ms

The client turns a keypress into a network packet in a tenth of a millisecond. Whatever is wrong, it is not the input path.

Step 3: Time the render

Same instrumentation, different pair: measure from "response arrived" to "DOM changed."

In my case the response handler acknowledged data on the same tick it painted, which made this easy to read: the acknowledgement and the paint were 1 ms apart. Rendering cost about 1 ms.

Against an 893 ms total, that is 0.1%.

This is the moment the whole investigation turned. I had been planning a renderer migration. The data said a renderer that was infinitely fast would have made typing feel 0.1% better.

Verify: if your render time is a meaningful fraction of the total, stop here. You found it. Go optimize rendering. For most remote systems it will not be.

Step 4: Time your server without the network

Get onto the server and hit it over loopback. This removes the network entirely and leaves only your application code:

for i in 1 2 3 4 5; do
  curl -s -o /dev/null -w "ttfb=%{time_starttransfer}\n" http://127.0.0.1:PORT/health
done

Result:

ttfb=0.008095
ttfb=0.007543
ttfb=0.003037
ttfb=0.003317
ttfb=0.003913

3 to 8 ms. That is the app answering a real request with zero network in the path.

Verify: compare this against the same endpoint hit from outside. The difference is your network. If loopback is also slow, you have found a server-side problem and the rest of this guide is optional.

Worth noting: the box was at load 1.97 on 2 vCPU at the time, badly overloaded by a runaway process. It still answered in 8 ms. Server CPU pressure is very often not the thing making a request slow, even when the CPU graph looks alarming. Measure before you believe the graph.

Step 5: Measure the network per hop

Now the part everyone skips. Do not measure "the network" as one number. Measure each leg separately, because the fix is different for each.

Find the hops:

traceroute your-server.example.com

Then ping each interesting hop directly, with enough packets to see the distribution:

ping -c 20 FIRST_HOP
ping -c 20 YOUR_SERVER

Warning: read the standard deviation, not just the average. An average hides everything that matters here.

My path had two legs, because the app server and the process it talks to were in different countries:

you -> control plane:     min 271.2   avg 681.9   max 1333.1   stddev 402.4
control plane -> backend: min 125.8   avg 125.9   max  126.4   mdev    0.201

Look at those standard deviations. The second leg is a machine talking to a machine over good transit: 126 ms, and it varies by two tenths of a millisecond. The first leg averages 682 ms and swings by 402 ms.

Same network, two completely different problems. One leg needs nothing. The other is both far and unhealthy.

Verify: if any leg shows duplicate ICMP replies, note it. That is a real signal and Step 6 explains it.

Step 6: Subtract your own last mile

This is the step that produces the biggest surprise, and almost nobody runs it.

Ping your own router. One hop. No internet involved:

# macOS
ping -c 100 $(route -n get default | awk '/gateway/{print $2}')
 
# Linux
ping -c 100 $(ip route | awk '/default/{print $3}')

I expected sub-millisecond numbers. I got:

100 packets transmitted, 98 received, +14 duplicates, 2.0% packet loss
rtt min/avg/max/mdev = 2.15/63.5/488/93.3 ms

488 ms to my own router. 93 ms of jitter and 2% loss before a single packet reached the internet.

The cause was visible in the link state: 2.4 GHz, channel 9, with fourteen other networks nearby and three of them on the same channel. The adapter supported 5 GHz. The 5 GHz band was almost empty.

Those +14 duplicates also explain something I had misattributed earlier. I had seen duplicate ICMP replies from the remote server and assumed a routing anomaly. They were Wi-Fi. 802.11 retransmits at the link layer, and a frame whose acknowledgement is lost arrives twice. They showed up on every destination, local and remote, because they were generated on my own last hop.

Separating your jitter from the network's

You now have two jitter numbers that overlap: your Wi-Fi contributes some, the WAN contributes the rest. Independent variances add in quadrature, so you subtract the same way:

WAN-only jitter = sqrt(total_stddev^2 - local_stddev^2)

With a measured total of 113.6 ms to the far server and 26.0 ms on the local link:

sqrt(113.6^2 - 26.0^2) = 110.6 ms of WAN jitter

Run the same subtraction against a nearer candidate server and the numbers separate cleanly:

sqrt(34.3^2 - 26.0^2) = 22.4 ms of WAN jitter

That tells you the ordering of your fixes. Today the WAN dominates, so move the server first. After that, your own Wi-Fi becomes the largest remaining contributor, and the fix costs nothing but changing a band.

Verify: if your local link is clean (sub-millisecond, no loss, no duplicates), skip the subtraction and use the raw numbers. Wired connections usually are. Wi-Fi under contention usually is not.

Step 7: Build the budget

Put every measurement in one table, as a percentage of the total:

Rendered to scale, the argument makes itself:

keydown -> wire   |
app               |
render            |
network           |=================================================|
                  0                                             400 ms

That is the whole finding. Three layers I could edit accounted for under 2%. The layer I could not edit accounted for the rest.

Step 8: Check the remainder against physics

You now know the network owns the time. The next question decides what to do about it: is the path badly routed, or simply long?

Compute the great-circle distance between the endpoints, then the theoretical floor. Light in fibre travels at roughly 200,000 km/s, so a round trip cannot beat:

floor_ms = (2 * distance_km) / 200

Fibre does not run in straight lines. Real routes are roughly 1.4 times the great-circle distance, and protocol overhead adds a little more, so a well-routed path lands somewhere around 1.5 to 1.9 times the floor. Divide your measured minimum by the floor and you get an efficiency ratio:

This single column tells you which lever to pull:

• Roughly 1.5x to 1.9x means the path is already close to what fibre allows. There is no routing left to improve. The only remaining variable is distance, so move the endpoint or accept the number.
• Above about 2.5x means the packets are taking a bad path for the distance. A CDN, a different transit provider, or a smart-routing product can plausibly help.

My current server measured 1.92x. Reasonably routed, and 13,677 km away. Network tuning was not going to help, because there was little wrong with the routing. The distance was the problem.

The backend host at 3.31x was the opposite: only half the distance, yet slower in practice. That is a routing problem, and it is worth fixing separately from the distance problem.

Do the division yourself rather than trusting a number someone hands you. I originally wrote this table with ratios computed against an already-adjusted baseline instead of the raw floor, which made every path look better than it was, and it survived until I re-derived the column from the formula.

Warning: use the minimum RTT for this ratio, never the average. The minimum is the closest you get to pure propagation delay. Averages include queueing, and queueing is not distance.

Step 9: Validate the model before you trust it

Predict the total from the parts. If your model is right, the prediction matches the measurement.

The path traversed two legs in each direction, so:

one-way to control plane  = 682 / 2 = 341 ms
one-way control -> backend = 126 / 2 =  63 ms

predicted = 341 + 63 + 63 + 341 = 808 ms
measured  =                       893 ms
unexplained =                      85 ms

An 85 ms gap on an 893 ms total. That residual is the app, plus SSH encryption, plus the terminal multiplexer, and it lines up with the 6 ms loopback measurement once you account for the extra process hops.

This is the step that makes the whole exercise trustworthy. Anyone can produce a table of numbers. A model that predicts the total from its parts, within 10%, is a model you can make decisions with.

Verify: if your prediction misses badly, you have an unmeasured hop. Go find it. That gap is information, not noise.

Step 10: Now pick a layer

With the budget in hand, the decisions become boring, which is the point:

That last row is worth expanding, because it is the one people get wrong in both directions.

I put the app behind an edge network that terminates TLS near the user. The handshake improved enormously:

                before        after
TCP connect     262-288 ms    5.8-13.7 ms
TLS             555-572 ms    16.5-32.7 ms
time to first byte  824-888 ms   289-417 ms

Connection setup got roughly 25 times faster, because the handshake now completes 4 ms away instead of 13,000 km away.

Steady-state keystroke latency did not move at all. The edge cannot shorten the path to the origin, and after the handshake every keystroke still makes the full trip.

Both facts are true and they are not in tension. Edge termination fixes handshakes, page loads and reconnects. It does not fix round trips to an origin that is far away. Knowing which one you have is the entire value of the budget.

What you have now

A table that says where the milliseconds go, a physics check that says whether the remaining time is fixable, and a validated model that predicts the total from the parts.

The general result, which I did not expect going in: for anything that crosses a network, the layers you can edit are usually not the layers that matter. Client code, render pipelines and server handlers get the attention because they are the parts you own. In my case they summed to under 2%.

The layers that mattered were where the server sits, and the quality of the first thirty metres of the link.

Where to go next

Once the budget says the network owns your latency, you have exactly three options, and only three:

1. Shorten the path. Move the server. This is usually cheap and usually the largest win.
2. Stop waiting for the round trip. Predict the response locally and reconcile when the truth arrives. This is what Mosh does for SSH, and it makes typing feel instant regardless of distance. It is also genuinely hard to get right, because a wrong prediction that reaches the screen is worse than a slow correct one.
3. Accept it. Sometimes 250 ms is fine. The budget tells you what you are accepting, which beats not knowing.

What the budget rules out is the fourth option everyone tries first, which is optimizing the layer you happen to be looking at.

All measurements taken against a live deployment and from the client machine, using ping, curl, traceroute and the browser console. Server IPs and hostnames redacted; the timing figures are unmodified.