GitHub - edenhill/librdkafka: The Apache Kafka C/C++ library
source link: https://github.com/edenhill/librdkafka
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
README.md
librdkafka - the Apache Kafka C/C++ client library
Copyright (c) 2012-2016, Magnus Edenhill.
https://github.com/edenhill/librdkafka
librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.
librdkafka is licensed under the 2-clause BSD license.
For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md
See the wiki for a FAQ.
NOTE: The master
branch is actively developed, use latest release for production use.
Overview
- High-level producer
- High-level balanced KafkaConsumer (requires broker >= 0.9)
- Simple (legacy) consumer
- Compression: snappy, gzip, lz4
- SSL support
- SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
- Broker version support: >=0.8 (see Broker version compatibility)
- Stable C & C++ APIs (ABI safety guaranteed for C)
- Statistics metrics
- Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
- RPM package: librdkafka and librdkafka-devel
- Gentoo package: dev-libs/librdkafka
- Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, ...
Language bindings
- C#/.NET: confluent-kafka-dotnet (based on rdkafka-dotnet)
- C++: cppkafka
- D (C-like): librdkafka
- D (C++-like): librdkafkad
- Erlang: erlkaf
- Go: confluent-kafka-go
- Haskell (kafka, conduit, avro, schema registry): hw-kafka
- Haskell: haskakafka
- Haskell: haskell-kafka
- Lua: luardkafka
- Node.js: node-rdkafka
- Node.js: node-kafka
- Node.js: kafka-native
- OCaml: ocaml-kafka
- PHP: phpkafka
- PHP: php-rdkafka
- Python: confluent-kafka-python
- Python: PyKafka
- Ruby: Hermann
- Ruby: rdkafka-ruby
- Rust: rust-rdkafka
- Tcl: KafkaTcl
- Swift: Perfect-Kafka
Users of librdkafka
- kafkacat - Apache Kafka swiss army knife
- Wikimedia's varnishkafka - Varnish cache web log producer
- Wikimedia's kafkatee - Kafka multi consumer with filtering and fanout
- rsyslog
- syslog-ng
- collectd
- logkafka - Collect logs and send to Kafka
- redBorder
- Headweb
- Produban's log2kafka - Web log producer
- fuse_kafka - FUSE file system layer
- node-kafkacat
- OVH - AntiDDOS
- otto.de's trackdrd - Varnish log reader
- Microwish has a range of Kafka utilites for log aggregation, HDFS integration, etc.
- aidp - kafka consumer embedded Lua scripting language in data process framework
- Yandex ClickHouse
- NXLog - Enterprise logging system, Kafka input/output plugin.
- large unnamed financial institutions
- and many more..
- Let me know if you are using librdkafka
Usage
Requirements
The GNU toolchain
GNU make
pthreads
zlib (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
Instructions
Building
./configure
make
sudo make install
NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.
NOTE: See CMake instructions for experimental CMake build (unsupported).
Usage in code
See examples/rdkafka_example.c for an example producer and consumer.
Link your program with -lrdkafka -lz -lpthread -lrt
.
Documentation
The public APIs are documented in their respective header files:
- The C API is documented in src/rdkafka.h
- The C++ API is documented in src-cpp/rdkafkacpp.h
To generate Doxygen documents for the API, type:
make docs
Configuration properties are documented in CONFIGURATION.md
For a librdkafka introduction, see INTRODUCTION.md
Examples
See the examples/
sub-directory.
Tests
See the tests/
sub-directory.
Support
File bug reports, feature requests and questions using GitHub Issues
Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.
Commercial support
Commercial support is available from Edenhill services
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK