Karafka framework 1.4.0 Release Notes (Ruby + Kafka)
source link: https://mensfeld.pl/2020/09/karafka-framework-1-4-0-release-notes-ruby-kafka/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
This release mostly solves problems related to message deserialization and normalizes some of the naming conventions to ease during the upgrade to the upcoming 2.0
version.
Note: This release is the last release with ruby-kafka
under the hood. We’ve already started the process of moving to rdkafka-ruby
.
Note: If you are using Sidekiq-Backend
plugin, please make sure that you’ve processed all the jobs from your Sidekiq queue before upgrading Karafka gems.
Table of Contents [show]
Changes (features, incompatibilities, etc)
consumer#metadata
is now consumer#batch_metadata
This change is trivial: if you use batch consuming mode and you use the Consumer#metadata
method, replace it with Consumer#batch_metadata
.
# Karafka 1.3
class
UsersConsumer < ApplicationConsumer
def
consume
puts metadata
end
end
# Karafka 1.4
class
UsersConsumer < ApplicationConsumer
def
consume
puts batch_metadata
end
end
Message metadata available under #metadata
method
Up to version 1.3
, all the message metadata would be directly available under the root scope of the params
object using both direct method reference as well as with #[]
accessor.
While it felt like “The Rails way”, it had several side-effects, amongst which the biggest were the need of having a hash like API, issues with accessing metadata without payload deserialization, and a lack of clear separation between payload and the metadata.
From now on, you can use the params.metadata
object to fetch all the metadata.
Note: we’ve preserved the direct metadata values fetching from the params
object to preserve backwards compatibility.
# 1.3
params[
'partition'
]
#=> 0
params.partition
#=> 0
# 1.4
params[
'partition'
]
#=> NoMethodError (undefined method '[]')
# This will work due to backward compatibility
params.partition
#=> 0
# This is the recommended way of accessing metadata
params.metadata.partition
#=> 0
# This will also work as metadata is a struct now
params.metadata[
:partition
]
#=> 0
params.metadata[
'partition'
]
#=> 0
Message metadata access allowed without message deserialization
When accessing metadata, the payload is not being deserialized until #payload
method is being used.
null message support in the default JSON deserializer
When the Kafka message payload is null
/ nil
, deserialization won’t fail. Support for it was added as some of the Karafka users use log compaction with a nil
payload. In case like that, #payload
will return nil
.
Karafka::Params::Params
no longer inherits from a Hash
Karafka::Params::Params
is now just a struct. This change is introduced to normalize the setup, limit the corner cases and simplify the interface only to methods that are really needed.
Documentation
Our Wiki has been updated accordingly to the 1.4
status. Please notify us if you find any incompatibilities.
Getting started with Karafka
If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:
git clone https:
//github
.com
/karafka/example-app
.
/example_app
then, just bundle install all the dependencies:
cd
.
/example_app
bundle
install
and follow the instructions from the example app Wiki.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK