Kafka Monthly Digest: June 2023

This 65th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in June 2023.

For last month’s digest, see Kafka Monthly Digest: May 2023.

Releases

Releases

There are two new releases: 3.5.0 and 3.4.1.

The release process for 3.6.0 and 3.5.1 also started.

3.5.0

I released Kafka 3.5.0 on June 15 and published an announcement on the Apache Kafka blog. As always, you can find the complete list of changes in the release notes or the release plan on the Kafka wiki.

This new minor release brings a lot of bug fixes and a few new interesting features, which I'll highlight in the next sections.

Kafka brokers and clients

Updates to the Kafka broker and clients include the following:

• The migration from ZooKeeper to KRaft is still available in early access. You can find the migration steps in the documentation.
• The built-in assignors now support rack-awareness. (KIP-881)
• There's a new ConfigProvider implementation, EnvVarConfigProvider, to retrieve configurations from environment variables. (KIP-887)
• The kafka-storage tool now supports configuring SCRAM for inter broker authentication with KRaft. (KIP-900)
• The inter broker replication protocolhas been improved to fix potential data loss in case of a broker failing while another broker had an unclean shutdown. (KIP-903)

Kafka Connect

Updates to Kafka Connect include the following:

• MirrorMaker in dedicated mode now supports running multiple instances and is able to handle automatic reconfigurations. (KIP-710)
• There's a new REST API endpoint for listing offsets for connectors. Endpoints for updating and deleting offsets will come in a future release. (KIP-875)
• MirrorMaker can now use the IncrementalAlterConfig API when mirroring topic configuration between clusters. (KIP-894)
• MirrorSourceConnector now supports the exactly once APIs from KIP-618. (KAFKA-14021)

Kafka Streams

Updates to Kafka Streams include the following:

• New method in the ProductionExceptionHandler interface, handleSerializationException(), to handle serialization errors encountered while producing records. (KIP-339)
• Support for versioned state stores to improve the accuracy of joins when out of order records are processed. (KIP-889)
• New built-in Serdes for Boolean primitive types. (KIP-907)

3.4.1

Luke Chen released 3.4.1 on June 6. This bugfix release fixes 58 issues. You can find more details in the release notes.

3.6.0

Satish Duggana volunteered to run the 3.6.0 release. You can find the release plan in the wiki.

3.5.1

On June 28, Divij Vaidya volunteered to be the release manager for 3.5.1. You can find the release plan in the wiki.

Kafka Improvement Proposals

Kafka Improvement Proposals

Last month, the community submitted 10 KIPs (KIP-938 to KIP-947). I'll highlight a few of them:

• KIP-939: Support Participation in 2PC: Two phase commit (2PC) is a protocol used to implement distributed transactions across multiple systems. This KIP proposes protocol and client changes to allow Kafka to participate in distributed transactions.
• KIP-940: Broker extension point for validating record contents at produce time: This KIP proposes a new type of broker plugin to validate records when they are received on brokers. For example, it could be used to reject records if they don't match a specific schema.
• KIP-945: Update threading model for Consumer: The internals of consumer client logic have increased in complexity over time and due to its asynchronous nature that logic can now be tricky to maintain. This KIP aims at simplifying the threading model while keeping the consumer public APIs identical.

Community Releases

Community Releases

• Debezium 2.3: Debezium is a Change Data Capture platform. This new release contains a lot of improvements and new features. This includes a new notification system so Debezium can emit events about its lifecyle and exactly once support for the PostgreSQL connector.

Blogs

Blogs

I selected some interesting blog articles that were published last month:

• Deep dive on Amazon MSK tiered storage

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.