Introduction
If you sit in enough architecture meetings, you will hear Kafka brought up constantly. Someone usually describes it as a message queue, but that is only half the story. It is actually a distributed event streaming platform built to handle massive, real-time data pipelines. It pulls this off through a few very deliberate design choices.
The Pub/Sub Setup
Kafka operates on a publish-subscribe model. Producers fire data into “topics,” and consumers read from them. Think of a topic as a continuous feed. The beauty of this is total decoupling: producers and consumers operate completely independently of one another. Every record sent contains a key, a payload value, and a timestamp. The key dictates where the data physically lands, the payload is the actual data, and the timestamp marks when it happened.
Why Partitions Matter
Topics are just logical concepts. Physically, they get split into partitions. A partition is an ordered, append-only log where new records are tacked onto the end and given a sequential ID called an offset. Consumers use this offset to remember their place in the log.
Partitions are Kafka’s secret to scaling. If a topic has ten partitions, ten consumers can read from it simultaneously. These partitions are also distributed across different servers (brokers) to balance the load. By default, Kafka distributes records evenly. However, if you include a key with your message, Kafka guarantees that messages sharing the same key always hit the same partition. This is vital when you need strict ordering, like processing sequential events for a specific user.
How Storage Actually Works
This is where the mental model breaks for people used to traditional queues. Kafka does not delete a message just because a consumer read it. It dumps everything to disk and holds onto it for a configured amount of time.
Instead of building a complex in-memory cache, Kafka leans completely on the OS page cache. It writes sequentially to disk (which is incredibly fast), and the operating system automatically keeps the frequently accessed data in RAM. On the disk itself, every partition consists of a raw data log and an index file that makes looking up offsets fast.
Retention is a balancing act. You can configure Kafka to delete data after a certain amount of time or once the log hits a specific size. Whichever limit triggers first wins, and unread messages will be deleted just the same as read ones. You have to weigh the tradeoffs carefully. Storing data longer eats up disk space fast on busy topics. Huge logs also make recovery painful if a broker dies, because the new leader has a mountain of data to catch up on. And if you lack strict disk monitoring, a full drive will force Kafka to start quietly purging your oldest data earlier than expected.
Durability and “Acks”
For fault tolerance, partitions are replicated across multiple brokers. One acts as the leader to handle reads and writes, while the followers passively copy the data. If the leader dies, an “in-sync” follower automatically takes over.
Producers get to choose how safe they want their writes to be using acknowledgements.
- Acks = 0: You are firing into the void. It is fast, but you will absolutely lose data if something hiccups.
- Acks = 1: The leader confirms the write before the followers have copied it. If the leader crashes a millisecond later, that data is gone. This is the default setting and the culprit behind most Kafka data loss stories.
- Acks = all: The leader waits for every in-sync replica to confirm the write. It adds latency, but it is the only way to guarantee you won’t lose critical data.
Consumer Responsibilities
Kafka does not micromanage who has read what. Consumers have to commit their own offsets back to the cluster. If a consumer crashes mid-job, it restarts from its last committed offset. This is why you get “at-least-once” delivery, which means your consumer logic must be idempotent to handle occasional duplicate messages.
What if no one is listening? Kafka does not care. If producers are writing to a topic with zero consumers, Kafka just saves the data until the retention period expires. It is exactly like a radio station blasting a broadcast into an empty room. The broadcast happens, but if nobody ever tunes in, you should probably ask why you are spending money to run the station.
Is it a Database?
Not quite. You cannot query Kafka by primary key or run arbitrary reads. Its sole purpose is moving data reliably.
However, features like compacted topics blur the line. A compacted topic never deletes data based on time; it just indefinitely keeps the most recent value for every key, acting like a state changelog. Tools like ksqlDB push it even closer to database territory by allowing SQL-style queries on streams, but you are still fundamentally dealing with a streaming engine, not Postgres.
Ultimately, Kafka handles ridiculous throughput because it combines simple concepts: append-only logs, OS-level caching, and partition scaling. The operational tax is high, and a bad configuration will quickly ruin your day. But once you understand the actual mechanics, designing systems around it and debugging those inevitable issues becomes a lot more manageable.