The Real Bottleneck at 1 Million RPS

Getting an API to survive 1,000,000 HTTP requests every second cost around $2,000 in real AWS billing. It also took five separate wrong guesses about what would break first. A backend engineer who runs the channel Cododev spent a video walking through the whole thing on camera, starting on a Mac Studio and ending on a cluster of AWS's largest network-optimized instances, and the pattern that emerges is more useful than the final number. Every time he fixed the bottleneck he'd identified, the ceiling moved somewhere else. The database, the thing almost every scaling article blames first, was the actual problem exactly once.

That is the real claim worth taking from this experiment: at extreme scale, the bottleneck is never the one you assumed on the way in, and finding the real one takes measurement, not intuition, the same rule that holds even at ordinary request volumes.

The baseline: three Node frameworks, one simple route

Before touching AWS, the test ran locally on a Mac Studio (12 CPU cores, 32 GB RAM, roughly a $60-a-month machine amortized over three years). The route was as simple as an API gets: GET /simple, returns {"message": "hi"}. Three Node.js frameworks, same route, same hardware:

Express lost by roughly 4x before a single line of business logic existed. That gap alone is worth knowing if you're picking a framework for anything latency-sensitive; it has nothing to do with your code and everything to do with how much the framework does before your handler even runs. The rest of the video runs on Cpeak, since it tracks close to raw Node performance and stays readable.

Clustering the same process across all 12 cores with PM2 took the realistic route (one with query params and a 30 KB JSON response, closer to a real API) from 8,000 requests per second to 42,000. No code changed. That's the first free lunch: most Node servers are running on one core by default, and nobody notices until someone benchmarks it.

Problem #1: the network was the ceiling before the CPU was

Moving to AWS meant a genuinely different machine: a C8i.32xlarge, 128 CPU cores, 256 GB of RAM, a 50 Gbit/s network card, $6 an hour to rent. On the trivial /simple route, this thing did 6,000,000 requests per second. The number is almost meaningless: a one-word JSON response tells you nothing about a real system.

Point it at the realistic 30 KB route instead, and throughput fell off a cliff: 100,000 requests per second, with the CPU sitting half idle. That's the tell. A server that isn't using its CPU isn't CPU-bound, whatever the naive assumption says. The actual math: 120 GB of data moved in 20 seconds is 6 GB/s, and the instance's 50 Gbit/s network card caps out at 6.25 GB/s. The server had almost nothing left to give. It wasn't slow. It was full.

The fix wasn't a framework change or a bigger instance. It was shrinking the payload from 30 KB to about 1 KB, which took the same route to 3,000,000 requests per second on the same hardware. Bandwidth is a physical ceiling that no amount of clever code gets around; the only lever is how many bytes you're actually pushing per response.

Problem #2: an O(n) query that took 43 seconds to answer

This is the moment that makes the "just don't have bugs at this scale" line from the video's intro concrete. The route needed a random row from a Postgres table seeded with 10 million records. The first version wrote it the way most people would:

SELECT * FROM codes ORDER BY RANDOM() LIMIT 1;

ORDER BY RANDOM() sorts the entire table by a freshly generated random value on every single call. It's an O(n) operation disguised as a five-word query. Under load, it crashed the database outright; when it did respond, one query took 43 seconds. A second attempt (SELECT COUNT(*), then a random offset) failed the same way, because COUNT(*) is O(n) too.

The fix was to grab the current max ID once (an indexed lookup, effectively O(1)), then pick a random ID within that range and fetch it directly. Same logical result, completely different complexity class: 40 milliseconds, 200,000 requests per second. A version that skipped even the max-ID check and just guessed an ID in a known range pushed that to 400,000. The gap between the first version and the last is roughly six orders of magnitude in latency, and none of it involved touching a config file or renting a bigger box. It was one query rewritten.

Problem #3: Redis is fast, and also single-threaded

With the query fixed, the database itself became the next wall, not because Postgres is slow, but because of what it costs to make it faster. Postgres was never the wrong tool here; it's genuinely good at what the rest of a real application needs from a database (transactions, joins, ad hoc queries a key-value store can't answer), and nothing in this experiment argues otherwise. It's specifically the wrong tool for one narrow job: serving a single hot read/write path at six-figure throughput, which is not what a relational database is optimized for. Pushing write throughput higher on that one path meant more IOPS and bigger instances, and the projected bill for the next tier up was $20,000 to $30,000 a month for a managed autoscaling setup. That's the real reason "the database is the bottleneck" is such common advice: fixing a database bottleneck by throwing hardware at it gets expensive fast, faster than almost anything else in the stack.

The actual fix was to stop asking Postgres to hold the hot path at all. The full 10-million-row table fit comfortably in the 200+ GB of RAM the instance already had, so a single Redis instance took over as the write target: 100,000 requests per second, at three times the throughput of the equivalent Postgres write, with the CPU sitting 80% idle. Then it hit its own wall at exactly 100,000, a flat ceiling that didn't move no matter how much idle CPU or RAM sat unused nearby, because a single Redis instance is single-threaded. All that spare hardware was irrelevant to a problem that lived entirely on one core.

The actual fix, once again, was not a bigger machine. It was a Redis Cluster: 30 nodes (15 masters, 15 replicas), with writes sharded across them by a crypto.randomUUID() key instead of a sequential ID. That combination, a real database-backed write path, not a toy route, crossed 1,000,000 requests per second for the first time in the whole video. (For anyone wondering about UUID collisions at that generation rate: the math works out to roughly 86,000 years before there's a 50% chance of even one.)

Problem #4: Node itself, not the hardware, was the ceiling

The next machine was close to the largest general-purpose instance AWS offers for this kind of workload: a C8gn.48xlarge, 192 cores, 384 GB of RAM, a 600 Gbit/s network card, about $11 an hour. On this machine, Express couldn't clear half a million requests per second on the full-size payload. Not because of a missing optimization; because Node's cluster model distributes incoming traffic through a single parent process to every worker, and at 180-plus workers, that distribution overhead becomes the bottleneck itself, independent of how much CPU or RAM sits behind it. Go, Rust, Java, and Python were all tried on the same box first. None of them reached 1 million either.

The fix that finally worked was a full rewrite of the hot route in C++, using the Drogon framework. The first version of that rewrite was slower than the Node one it replaced, because Drogon's default JSON parser turned out to be four times slower than V8's built-in parser. Swapping to RapidJSON fixed it. On a 60-second sustained run, the C++ service averaged 1,000,000 requests per second and peaked at 1,200,000, moving roughly 300 Gbit/s of data, using only about 70% of the CPU to do it. Node needed 100% CPU utilization to hit a lower number. That gap, not the raw RPS figure, is the actual argument for rewriting a hot path in C++: it isn't just faster, it leaves headroom the Node version never had.

Problem #5: the load balancer had a limit nobody had checked

Putting two of those beast machines behind an AWS Network Load Balancer should have roughly doubled throughput. It cratered instead, from roughly 38 GB/s on a single server down to 5 GB/s across two. The cause was a hard limit on Load Balancer Capacity Units, capped at 165 by default, that has to be pre-reserved through AWS's own NLB capacity calculator before a burst like this hits it. It's the kind of ceiling that doesn't show up until you're already past every other one.

AWS support's own AI assistant produced ten suggested fixes for the ticket. None of them worked. At a scale only a handful of companies in the world actually operate at, there isn't enough public documentation, forum history, or support-ticket data for a model to have learned the real answer. That's a genuinely useful data point about the current limits of AI-assisted troubleshooting, and it came from someone actually hitting the wall, not from a hot take about AI.

What the final proof run actually cost

The last validation run swapped one giant load-generating instance for sixty smaller ones (AWS's default account cap of 800 CPU cores per region ruled out the first two attempts, at 100 and 80 instances, before 60 turned out to be the number that would actually launch). Over 30 minutes, those 60 testers drove 2,000,000,000 requests into the C++ service and moved more than 60 terabytes of data, with 40 timeout errors and nothing else. One clean number, from a real sustained run, not a 20-second best-case sample.

The part every reader actually wants to know: billed across the actual hours the instances ran, rather than the eye-watering "$20,000 a month" sticker prices quoted for each machine along the way, the creator's total bill for the month covering this project (including off-camera testing, and by his own account a couple hundred dollars lost to avoidable mistakes) came to around $2,000, roughly $800 in databases and $1,200 in compute. The monthly sticker prices matter for capacity planning. They are not what building and testing this actually cost.

Where this leaves the stack decision

None of this argues for rewriting every route in C++. The creator's own framing, stated plainly on camera, is the right takeaway: reach for Node when you need development speed, and reach for something like Drogon only for the specific routes where raw throughput is the entire point, with a reverse proxy like Nginx in front deciding which request goes where. Everything below roughly 100,000 requests per second on a single path stayed comfortably inside Node the whole time. The Redis Cluster rewrite mattered once a single write path needed to clear six figures with real persistence behind it. The C++ rewrite only mattered once Node's own request-distribution model, not the CPU or the network under it, became the thing left standing in the way.

The database, the default villain in almost every article about scaling to a million requests a second, only left the story once: not because it got tuned harder, but because it got replaced with a tool built for a different shape of problem. The other four bottlenecks were never it in the first place.