GitHub - cswinter/LocustDB: Massively parallel, high performance analytics datab...
source link: https://github.com/cswinter/LocustDB
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
README.md
LocustDB
An experimental analytics database aiming to set a new standard for query performance on commodity hardware. See How to Analyze Billions of Records per Second on a Single Desktop PC for an overview of current capabilities.
How to use
- Install Rust
- Clone the repository
git clone https://github.com/cswinter/LocustDB.git
cd LocustDB
- Run the repl!
RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo +nightly run --release --bin repl -- test_data/nyc-taxi.csv.gz
Instead of test_data/nyc-taxi.csv.gz
, you can also pass a path to any other .csv
or gzipped .csv.gz
file. The first line of the file will need to contain the names for each column. The datatypes for each column will be derived automatically, but things might break for columns that contain a mixture of numbers/strings/empty entries.
You can pass the magic strings nyc100m
or nyc
to load the first 5 files (100m records) or full 1.46 billion taxi rides dataset which you will need to download first (for the full dataset, you will need about 120GB of disk space and 60GB of RAM).
Running tests or benchmarks
cargo +nightly test
RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo +nightly bench
Goals
A vision for LocustDB.
Fast
Query performance for analytics workloads is best-in-class on commodity hardware, both for data cached in memory and for data read from disk.
Cost-efficient
LocustDB automatically achieves spectacular compression ratios, has minimal indexing overhead, and requires less machines to store the same amount of data than any other system. The trade-off between performance and storage efficiency is configurable.
Low latency
New data is available for queries within seconds.
Scalable
LocustDB scales seamlessly from a single machine to large clusters.
Flexible and easy to use
LocustDB should be usable with minimal configuration or schema-setup as:
- a highly available distributed analytics system continuously ingesting data and executing queries
- a commandline tool/repl for loading and analysing data from CSV files
- an embedded database/query engine included in other Rust programs via cargo
Non-goals
Until LocustDB is production ready these are distractions at best, if not wholly incompatible with the main goals.
Strong consistency and durability guarantees
- small amounts of data may be lost during ingestion
- when a node is unavailable, queries may return incomplete results
- results returned by queries may not represent a consistent snapshot
High QPS
LocustDB does not efficiently execute queries inserting or operating on small amounts of data.
Full SQL support
- All data is append only and can only be deleted/expired in bulk.
- LocustDB does not support queries that cannot be evaluated independently by each node (large joins, complex subqueries, precise set sizes, precise top n).
Support for cost-inefficient or specialised hardware
LocustDB does not run on GPUs.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK