README.md

Slim - surprisingly space efficient data types in Golang

Slim is collection of surprisingly space efficient data types, with corresponding serialisation APIs to persisting them on-disk or for transport:

• SlimIndex: is a common index structure, building on top of SlimTrie.

  

• SlimTrie is the underlying index data structure, evolved from trie.

  

  Features:

  • Minimised: requires only 6 byte per key(even smaller than a 8-byte pointer!!).

  • Stable: memory consumption is stable in various scenario. The Worst case converges to average consumption tightly. See benchmark.

  • Unlimited key size: You can have VERY long key, without bothering yourself with any waste of memory(and money). Do not waste your life writing another prefix compression:). (aws-s3 limits key size to 1024 bytes). Memory consumption only relates to key count, not to key size.

  • Ordered: like btree, keys are stored in alphabetic order, but faster to access. Range-scan is supported!(under development)

  • Fast: time complexity for a get is O(log(n) + k); n: key count; k: key size. With comparison to btree, which is O(log(n) * k)(tree-like). And golang-map is O(k)(hash-table-like).

  • Ready for transport: SlimTrie has no gap between its in-memory layout and its on-disk layout or transport presentation. Unlike other data-structure such as the popular red-black-tree, which is design for in-memory use only and requires additional work to persisting it or transport it.

  • Loosely coupled design: index logic and data storage is completely separated. Piece of cake using SlimTrie to index huge data.

Memory overhead benchmark

Comparison of SlimTrie and native golang-map.

• Key size: 512 byte, different key count:

• Key count: 16384, different key size:

SlimTrie memory does not increase when key become longer.

Performance benchmark: get

Time(in nano second) spent on a get operation with SlimTrie, golang-map and btree by google.

Smaller is better.

See: benchmark-get-md.

It is about 60% faster than the btree by google.

Status

• 0.1.0(Current): index structure.

Synopsis

Use SlimIndex to index external data

package main
import (
        "fmt"
        "strings"
        "github.com/openacid/slim/index"
)

type Data string

func (d Data) Read(offset int64, key string) (string, bool) {
        kv := strings.Split(string(d)[offset:], ",")[0:2]
        if kv[0] == key {
                return kv[1], true
        }
        return "", false
}

func main() {
        // `data` is a sample external data.
        // In order to let SlimIndex be able to read data, `data` should have
        // a `Read` method:
        //     Read(offset int64, key string) (string, bool)
        data := Data("Aaron,1,Agatha,1,Al,2,Albert,3,Alexander,5,Alison,8")

        // keyOffsets is a prebuilt index that stores key and its offset in data accordingly.
        keyOffsets := []index.OffsetIndexItem{
                {Key: "Aaron",     Offset: 0},
                {Key: "Agatha",    Offset: 8},
                {Key: "Al",        Offset: 17},
                {Key: "Albert",    Offset: 22},
                {Key: "Alexander", Offset: 31},
                {Key: "Alison",    Offset: 43},
        }

        // Create an index
        st, err := index.NewSlimIndex(keyOffsets, data)
        if err != nil {
                fmt.Println(err)
        }

        // Lookup by SlimIndex
        v, found := st.Get2("Alison")
        fmt.Printf("key: %q\n  found: %t\n  value: %q\n", "Alison", found, v)

        v, found = st.Get2("foo")
        fmt.Printf("key: %q\n  found: %t\n  value: %q\n", "foo", found, v)

        // Output:
        // key: "Alison"
        //   found: true
        //   value: "8"
        // key: "foo"
        //   found: false
        //   value: ""
}

Getting started

Install

go get github.com/openacid/slim

All dependency packages are included in vendor/ dir.

Prerequisites

• For users (who'd like to build cool stuff with slim):

  Nothing.

• For contributors (who'd like to make slim better):

  • dep: for dependency management.
  • protobuf: for re-compiling *.proto file if on-disk data structure changes.

  Max OS X:

  brew install dep protobuf

  On other platforms you can read more: dep-install, protoc-install.

Who are using slim

baishancloud

Roadmap

• 2019 Mar 08: SlimIndex, SlimTrie
• Marshalling support
• SlimArray

Feedback and contributions

Feedback and Contributions are greatly appreciated.

At this stage, the maintainers are most interested in feedback centred on:

• Do you have a real life scenario that slim supports well, or doesn't support at all?
• Do any of the APIs fulfil your needs well?

Let us know by filing an issue, describing what you did or wanted to do, what you expected to happen, and what actually happened:

• bug-report
• improve-document
• feature-request

Or other type of issue.

Authors

• 刘保海 marshalling
• 吴义谱 array
• 张炎泼 slimtrie design
• 李文博 trie-compressing, trie-search
• 李树龙 marshalling

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE file for details.