8

Writing UDFs in Golang

 1 year ago
source link: https://www.splitgraph.com/blog/seafowl-udf-golang
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client
Aug 24, 2023 · By Patrick Skinner
READING TIME: 3 min

Writing UDFs in Golang

We return to the wide world of WASM. In our previous round we implemented a UDF using Rust; today we're porting it to Golang.

Seafowl has supported UDFs for some time now, and they continue to offer an exciting way for Seafowl users to significantly extend functionality in a fully sandboxed environment. Recently, in response to user request, we added a Golang example.

See the UDF docs.

To learn more about compiling Golang to WASM, read on.

Handling UDF requirements

There are certain requirements incumbent on the UDF implementer when it comes to running UDFs in Seafowl, and Golang is included. Today we cover a partial list:

  • reading data in from the host through linear raw memory
  • returning the computed result as a msgpack-encoded buffer
  • working around Golang default syntax (e.g. TitleCase function exports)
  • compiling Golang into WASM

Reading data in

As discussed previously, when you register a Seafowl UDF it's required to specify certain parameters including e.g. wasm_export (e.g. addi64), return_type (e.g. BIGINT), input_types (e.g. BIGINT).

This is important because raw memory is the medium for passing data between Seafowl and the UDF. By specifying the parameter types upfront at UDF registration time, both Seafowl host and UDF know how to properly encode/decode the data they share with each other. That's why the function addi64, expects a pointer to a buffer vs conventional Go function params.

Writing data out

Conversely, when we return the result to Seafowl, we need to return a buffer that includes the length of the result first, with the actual msgpack-encoded result appended afterwards.

This allows Seafowl's query engine to return the result of the UDF in the typical way.

TitleCase function export workaround

Golang conventionally uses TitleCase to indicate when a function should be exported from its module scope. Some may find it cute and this convention is presumably fine in traditional Golang, but in certain contexts like UDFs, we are required to provide alloc/dealloc functions in lower case - which poses a fun hurdle.

Fortunately, while researching how tinygo works, I learned it supports annotating functions regardless of casing via //export $functionName

For example:

//export alloc
func alloc(size uintptr) unsafe.Pointer {
    ...
}

And Seafowl will be able to call this function in the required way.

Compiling Golang into WASM

In August 2023 the Golang team shipped WASI/WASM support, so the following approach we used may no longer be strictly necessary these days.

When we tackled this project, Golang did not yet support WASM natively so looking elsewhere, we found Tinygo.

It turns out we can generate a WASM module ready for import into Seafowl via: tinygo build -o seafowl-udf-go.wasm -target=wasi

NOTE: it's necessary to include -target=wasi.

Recap

In this post we reviewed some of the steps taken to port the Rust-based UDF into Golang. This was my first Golang project and I appreciated the opportunity to work at the lower level, learn about byteslice manipulation, msgpack serialization, WASI and more.

If you build a cool UDF, please consider letting us know!

Image credits

[liantomtit](https://unsplash.com/@liantomtit), [evanescentlight](https://unsplash.com/@evanescentlight)

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK