Latency Numbers Every Engineer Should Know (and What They Mean for Design)

Good system design starts with intuition for how long things take. Reading from memory versus disk, a call within a data center versus across an ocean: these differ by orders of magnitude, and those gaps explain why caches exist, why chatty services are slow, and why a CDN makes a site in another continent feel local.

The famous "latency numbers every programmer should know" list popularized this idea. Exact values vary with hardware generation and cloud environment, so treat the numbers below as orders of magnitude, not benchmarks.

#Key takeaways

• Memory is roughly 100× faster than SSD random reads, and SSDs are much faster than spinning disks for random I/O.
• A network round trip within a data center costs about as much as reading hundreds of kilobytes from memory.
• Cross-continent round trips (tens to hundreds of milliseconds) dwarf almost everything else.
• Design implications: cache hot data, batch and parallelize network calls, keep data close to compute, and move content close to users.

#The numbers (approximate orders of magnitude)

#Scaled to human time

If an L1 cache hit took 1 second:

• A RAM access would take about 1.5 minutes.
• An SSD random read would take several hours to a day.
• A round trip within a data center, roughly a few days.
• A transatlantic round trip, a few years.

That's why a single extra network hop in a hot loop matters more than almost any code micro-optimization.

#Lesson 1: Memory beats disk, and cache hot data

Reading from RAM is orders of magnitude faster than reading from storage. That's the entire premise of:

• Database buffer pools: size RAM so the working set of hot data and indexes stays in memory.
• Application and distributed caches (caching strategies).
• Indexes, which minimize the number of pages read (how database indexes work).

If your database's working set doesn't fit in memory, performance can fall off a cliff. Watch cache hit ratios.

#Lesson 2: Network round trips add up fast

A same-region round trip of ~0.5 ms seems tiny, until code makes hundreds of them:

textN+1 queries: 1 query for 100 orders + 100 queries for customers
= 101 round trips × 0.5 ms ≈ 50 ms of pure network waiting (before query time)

Fixes:

• Batch: fetch all customers in one query (WHERE id IN (...)) or use joins.
• Parallelize independent calls instead of making them sequentially.
• Reduce chattiness between microservices: coarser APIs, or aggregate data in a backend-for-frontend (REST vs GraphQL vs gRPC).
• Reuse connections: new TCP and TLS handshakes add extra round trips (connection pooling explained).

#Lesson 3: Distance is physics

Light in fiber travels at roughly 200,000 km/s, about 5 µs per kilometer one way. New York to London (~5,600 km) has a theoretical minimum round trip of about 56 ms; real routes add more. No software optimization beats geography. So:

• Put content close to users with CDNs (what is a CDN).
• Keep application servers close to their database: an app in one region querying a database in another pays cross-region latency on every query.
• Consider regional deployments for global latency-sensitive products (how to scale a web application).
• Minimize round trips per page load: HTTP/2 or HTTP/3, fewer blocking resources, connection reuse (what happens when you type a URL).

#Lesson 4: Sequential beats random

Sequential reads are dramatically faster than random reads, especially on spinning disks but also on SSDs. This shapes:

• Log-structured storage and append-only designs (Kafka's log, LSM-tree databases).
• Columnar analytics formats that read long runs of similar data.
• Batch processing of large files.

#Lesson 5: Tail latency matters at scale

If a page makes 50 backend calls and each has a 1% chance of taking over 1 second, then about 40% of page loads (1 − 0.99⁵⁰ ≈ 0.39) will hit at least one slow call. Fan-out amplifies tail latency. Mitigations:

• Measure and optimize p99, not averages.
• Set timeouts and use fallbacks.
• Hedged requests: send a duplicate request to another replica if the first is slow (carefully, to avoid overload).
• Reduce fan-out where possible.

#Lesson 6: Do the math before designing

Back-of-the-envelope estimates using these numbers quickly show whether a design is plausible (capacity planning):

textGoal: API p95 < 100 ms
Budget: 2 DB queries (~2–5 ms each incl. round trip), 1 cache read (~1 ms),
        1 call to another service in-region (~5–20 ms), app logic (~10 ms)
→ ~40 ms: fits comfortably.

Same API calling a service in another continent (~150 ms RTT): cannot meet p95 < 100 ms.

#FAQ

#Are these latency numbers still accurate?

The exact values change with hardware and cloud infrastructure: memory and SSDs have improved, and cloud networking varies by provider and region. The relative orders of magnitude have stayed remarkably stable, and that's what matters for design intuition.

#Why is a network call so much slower than a function call?

A function call happens in nanoseconds within the CPU and memory. A network call involves serialization, the operating system network stack, physical transmission, the remote server's processing and the return trip. Each step adds microseconds to milliseconds.

#How do I measure latency in my own system?

Use distributed tracing and metrics with latency histograms (p50/p95/p99) per endpoint and dependency (observability explained). Synthetic tests from different regions reveal geographic latency.

#What's the biggest latency win for most web apps?

Usually eliminating unnecessary round trips (N+1 queries, sequential service calls, uncached repeated reads) and serving static content from a CDN. These often matter more than CPU-level optimizations.