In a previous blog post we have already covered the key features and migration possibilities to transition to Kafka KRaft mode. The Kafka community just hit a major milestone: with the release of Apache Kafka 4.0, ZooKeeper is officially gone. For over a decade, ZooKeeper has been a critical (and sometimes painful) part of running Kafka in production. Now, Kafka’s own built-in consensus mechanism — KRaft (Kafka Raft) — is the default and only option moving forward.
Goodbye ZooKeeper
For years, ZooKeeper managed Kafka’s metadata — tracking brokers, topics, partitions, and configuration. But it came with operational overhead: managing a separate cluster, dealing with connectivity issues, and aligning version compatibility.
This change has been a long time coming. KRaft mode was introduced in Kafka 2.8, declared production-ready in 3.3, and now becomes the only mode in 4.0. To migrate to 4.0 make sure you migrate from a version supporting Kraft and Zookeeper mode: see blogpost Kraft. It is now time to migrate to Kafka 4.0 with KRaft. Support and Bugfixes for Kafka 3.x versions will only be provided for a limited time.
New Features in Kafka 4.0
Besides the removal of ZooKeeper, Kafka 4.0 brings a few more improvements worth mentioning:
- Tiered Storage - KIP-405: This feature decouples compute and storage by allowing Kafka brokers to offload older log segments to remote storage solutions like Amazon S3, Google Cloud Storage, or Azure Blob Storage. This opens the door to infinite retention and makes Kafka more cost-effective for long-term storage use cases.
- New Consumer Group Protocol - KIP-848: Kafka 4.0 also brings the general availability of KIP-848, introducing a powerful new consumer group protocol designed to dramatically improve rebalance performance. This optimization significantly reduces downtime and latency, enhancing the reliability and responsiveness of consumer groups, especially in large-scale deployments.
- Metadata Snapshotting - KIP-866: Kafka now takes regular snapshots of its metadata state, drastically improving the time it takes for a new controller to catch up or recover. This enhances stability, especially in large clusters with frequent metadata changes.
- Queues for Kafka (early access) - KIP-932: Kafka has introduced support for shared consumer groups, aligning it with traditional message queue semantics. This feature allows multiple consumers to consume from the same partition in a round-robin fashion, enabling load-balanced message processing where strict message ordering is not required. This enhancement extends Kafka’s functionality and brings it closer to that of traditional queueing systems like RabbitMQ or ActiveMQ.
- Remove old client protocol API versions - KIP-896: Kafka 4.0 still supports older clients, the minimum client version protocol version is now 2.1. As for clients the minimal Java version was bumped to 11, on brokers the minimum will be Java 17.
- Cleaner Codebase: With ZooKeeper code removed, the Kafka codebase is now leaner and more focused on evolving the KRaft-based architecture.
For more features and release information, check the official post.
Final Thoughts
Kafka 4.0 represents a huge leap forward in simplifying the architecture of distributed messaging. It’s lighter, leaner, and more maintainable without ZooKeeper in the mix. If you’ve been holding off on migrating to KRaft, now’s the time. With the built-in tools and maturity of the feature, it’s never been easier.
Welcome to a ZooKeeper-free Kafka world.