In June 2025, a pair of LinkedIn engineers published a routine-looking blog post. Architecture diagrams, throughput numbers, the usual format. But the substance was not routine at all: the company that built Kafka — the software quietly running underneath half the Fortune 500 — announced that it had spent the last several years replacing it internally, and that over 90% of its own application traffic was no longer running on the system it gave the world.
This is the story of why, and how they pulled it off without anyone outside the company noticing.
What Kafka Actually Does
If you've never touched Kafka, here's the one paragraph you need. Think of it as a mail-sorting facility that never closes. Applications ("producers") drop messages into labeled bins called topics — one topic might track "someone liked a post," another "a payment just cleared." Other applications ("consumers") watch the bins they care about and react the moment something new arrives. To handle huge volumes, each topic is split into partitions — parallel lanes that spread the load across many machines. That's Kafka: a durable, ordered, replayable stream of events, sliced into lanes, moving from senders to receivers in near real time.
Three LinkedIn engineers built it in 2010, when the company had 90 million members and a Hadoop pipeline collapsing under its own file count. Kafka fixed that problem. LinkedIn then open-sourced it, and the rest of the industry built a good chunk of modern data infrastructure on top of it.
The Scale That Broke the Math
Fifteen years later, LinkedIn's own Kafka footprint looked like this: roughly 32 trillion records a day, 17 petabytes a day, spread across 400,000 topics on 150 separate clusters running on more than 10,000 machines, serving 1.2 billion members.
Distributed systems rarely degrade gracefully at extreme scale. A design that works fine at ten times its intended load might still limp along at a hundred times it. At a thousand times, it doesn't slow down politely — it runs into a wall baked into decisions made a decade earlier, one that no amount of additional hardware can fix. LinkedIn had hit that wall. Not because Kafka was badly built, but because certain 2010-era trade-offs had become load-bearing constraints at 2025-era scale.
Three of them stood out.
The controller bottleneck. Every Kafka cluster relies on a single node — the controller — to track metadata for every partition: who's the leader, who's in sync, what's changed. At LinkedIn's scale, with clusters holding hundreds of thousands of partitions, a slow metadata update on that one node could freeze an entire cluster for seconds. In a system moving 32 trillion records a day, a few frozen seconds is an incident, not a hiccup.
The rebalancing cost. When a broker fails, Kafka rebalances by physically moving terabytes of partition data across the network. Some machines sit idle while others get pinned at 100% CPU just copying data. The fix for one failure tends to manufacture a second, smaller one. Manageable at modest scale; a recurring operational headache at LinkedIn's.
Static partitioning. Kafka requires deciding upfront how many partitions a topic will ever need. Guess too low, you cap your own throughput permanently. Guess too high, you pay for capacity nobody uses. With 400,000 topics and traffic patterns constantly shifting, this wasn't a minor inconvenience — it was a tax paid by every team, indefinitely.
None of these three problems matter much for a company running Kafka at ordinary scale. They only become existential at the extreme edge of what the architecture was ever designed to handle — LinkedIn's own engineers describe this as the "0.1%" tier of scale.
Building the Replacement
LinkedIn engineers Onur Karaman and Xiongqi Wu designed a new log storage system from the ground up, written in C++, called Northguard. The core design philosophy runs in the opposite direction from Kafka's: instead of centralizing control in a single controller, Northguard shards both data and metadata and coordinates through a decentralized protocol, so no single node's failure can freeze the cluster.
A few specific design choices:
- Segments instead of rigid partitions. The basic unit is a record (key, value, headers), grouped into a segment — the smallest unit of replication. Segments split and merge dynamically, removing the need to guess capacity months in advance.
- Ranges instead of a fixed partition count. When a range splits under load, only the clients writing to that specific range are briefly affected — not the whole topic. Kafka-style rebalances tend to be all-or-nothing events; Northguard avoids that by design.
- Log striping, which spreads load evenly across the cluster instead of concentrating it on a handful of brokers.
- Metadata sharded via consistent hashing, backed by Raft-replicated state machines distributed across what LinkedIn calls "vnodes" — the direct fix for the controller bottleneck, since there's no longer a single controller to bottleneck.
- Millisecond-level durability with strong consistency — producers are acknowledged only after data is fsynced across replicas, a stronger guarantee than Kafka's default model.
Northguard isn't a Kafka fork. It's closer to what you'd get if you asked: given everything we now know about running at 1.2-billion-user scale, how would we build this from scratch? Same foundational idea as Kafka — durable, ordered, replayable streams — but re-engineered for a scale Kafka's original architecture was never meant to reach.
The Harder Problem: You Can't Turn It Off
Designing Northguard was, in some ways, the easier half. The harder constraint was this: LinkedIn cannot schedule downtime. There is no maintenance window where 1.2 billion users pause activity while engineers swap the plumbing underneath them. Whatever replaced Kafka had to go in live, under hundreds of thousands of topics and thousands of applications, without any team needing to rewrite its code.
To solve that, LinkedIn built a second system: Xinfra (pronounced "zin-frah"), a virtualization layer that sits between every application and whatever physical system actually stores its data. Applications stop talking to "a Kafka cluster" directly — they talk to a virtual topic, and Xinfra decides, transparently, where that topic actually lives at any given moment.
During a migration, Xinfra can dual-write a topic to both the old and new systems simultaneously, let consumers read from either transparently, and preserve ordering guarantees throughout. Once the migration completes, the application's code hasn't changed at all — it never had visibility into the fact that anything moved underneath it.
Where Things Stand
As of the most recent reporting, more than 90% of LinkedIn's applications are running on Xinfra clients, and thousands of topics — accounting for trillions of records a day — have already migrated from Kafka to Northguard with zero downtime for mission-critical workloads. Several hundred thousand topics remain, which says less about how far behind the project is and more about how deliberately it's being rolled out: gradually, in production, under full load, rather than through a single high-risk cutover.
LinkedIn has said Northguard and Xinfra aren't open source yet — the company is still finalizing internals and will consider it later. So this isn't infrastructure you can adopt this weekend. Its relevance is elsewhere.
Why This Matters Beyond LinkedIn
Nearly every company will never come close to LinkedIn's scale, and that's exactly why this story is more useful as a lesson than as a blueprint. Every architecture encodes an assumption about the ceiling you'll never hit — until you do. Kafka's single controller, its manual partitioning, its rebalance-by-copying-data approach were not mistakes; they were the right trade-offs for a company with 90 million users in 2010. LinkedIn's move away from Kafka isn't a story about Kafka failing. It's a case study in recognizing when the thing you built — even the thing you're famous for building — has quietly reached the edge of its own assumptions, and having the discipline to replace it live, without breaking what everyone else depends on.
Sources: LinkedIn Engineering Blog ("Introducing Northguard and Xinfra"); reporting and analysis from InfoQ, SiliconANGLE, BigDATAwire/HPCwire, and Webmobix.
Comments
…
Loading comments…