38
GitHub - travisjeffery/jocko: Kafka implemented in Golang with built-in coordina...
source link: https://github.com/travisjeffery/jocko
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
README.md
JOCKO
Kafka/distributed commit log service in Go.
Goals of this project:
- Implement Kafka in Go
- Protocol compatible with Kafka so Kafka clients and services work with Jocko
- Make operating simpler
- Distribute a single binary
- Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
- Smarter configuration settings
- Able to use percentages of disk space for retention policies rather than only bytes and time kept
- Handling size configs when you change the number of partitions or add topics
- Learn a lot and have fun
TODO
- Producing
- Fetching
- Partition consensus and distribution
- Protocol
- Produce
- Fetch
- Metadata
- Create Topics
- Delete Topics
- Consumer group [current task]
- Discovery
- API versioning [more API versions to implement]
- Replication [first draft done - testing heavily now]
Reading
- How Jocko's built-in service discovery and consensus works
- How Jocko's (and Kafka's) storage internals work
Project Layout
├── broker broker subsystem
├── cmd commands
│ └── jocko command to run a Jocko broker and manage topics
├── commitlog low-level commit log implementation
├── examples examples running/using Jocko
│ ├── cluster example booting up a 3-broker Jocko cluster
│ └── sarama example producing/consuming with Sarama
├── protocol golang implementation of Kafka's protocol
├── prometheus wrapper around Prometheus' client lib to handle metrics
├── server API subsystem
└── testutil test utils
└── mock mocks of the various subsystems
Building
Local
-
Clone Jocko
$ go get github.com/travisjeffery/jocko
-
Build Jocko
$ cd $GOPATH/src/github.com/travisjeffery/jocko $ make
Docker
docker build -t travisjeffery/jocko:latest .
License
MIT
-
GitHub @travisjeffery
-
Twitter @travisjeffery
-
Medium @travisjeffery
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK