How code generation wrote our API and CLI.
source link: https://pace.dev/blog/2020/07/27/how-code-generation-wrote-our-api-and-cli.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
We recently released the Pace API, along with an accompanying command line tool, and various client libraries, all of which enable programmatic integration with Pace.
We use Oto to describe our RPC (Remote Procedure Call) API using Go interfaces and structs, and a variety of templates to generate the code we need.
All methods in the API use a input and output struct for the request and response, supporting all JSON data types; strings, numbers, bools, objects and arrays.
Describing the API
We describe our API using Go interfaces and structs.
Here’s a sample from our CardsService
that describes the GetCard
method:
package pace
// CardsService allows you to programmatically manage cards in Pace.
type CardsService interface {
// GetCard gets a card by ID.
GetCard(GetCardRequest) GetCardResponse
}
// GetCardRequest is the input object for GetCard.
type GetCardRequest struct {
// OrgID is the ID of the org.
// example: "your-org-id"
OrgID string
// CardID is the ID of the card to get.
// example: "123"
CardID string
}
// GetCardResponse is the output object for GetCard.
type GetCardResponse struct {
// Card is the card.
Card Card
}
type Card struct {
ID string
CTime string
Title string
Body string
BodyHTML string
// simplified to save your eyes
}
Since this is real Go code, our IDEs build and lint this along with the rest of our code.
How Oto parses the definition
Oto uses the golang.org/x/tools/go/packages
package (among others from the standard library) to understand the Go code, and turn the definition files into a workable data structure.
The following types are taken from the parser inside Oto:
// Service describes a service, akin to an interface in Go.
type Service struct {
Name string `json:"name"`
Methods []Method `json:"methods"`
Comment string `json:"comment"`
}
// Method describes a method that a Service can perform.
type Method struct {
Name string `json:"name"`
NameLowerCamel string `json:"nameLowerCamel"`
InputObject FieldType `json:"inputObject"`
OutputObject FieldType `json:"outputObject"`
Comment string `json:"comment"`
}
// etc
Now that we have Go data that describes the interfaces, methods and structs, we can use templates to generate code.
Code generation
Oto is essentially a code generation tool, because most of the work needed to wire up an implementation to an endpoint is predictable boilerplate that we would prefer not to write.
When generics lands in Go, we will be able to reduce the amount of boilerplate code generated in favour of generic methods. In this world, we may not need to generate much plumbing code at all.
Generating server-side plumbing
The first thing we generate is the server-side plumbing for the implementation of the service.
- The template that was used to generate this code is available as part of
otohttp
package api
// CardsService allows you to programmatically manage cards in Pace.
type CardsService interface {
// GetCard gets a card by ID.
GetCard(context.Context, GetCardRequest) (*GetCardResponse, error)
}
type cardsServiceServer struct {
server *otohttp.Server
cardsService CardsService
}
// RegisterCardsService registers the CardsService implementation with the otohttp Server.
func RegisterCardsService(server *otohttp.Server, cardsService CardsService) {
handler := &cardsServiceServer{
server: server,
cardsService: cardsService,
}
server.Register("CardsService", "GetCard", handler.handleCardsServiceGetCard)
}
func (s *cardsServiceServer) handleCardsServiceGetCard(w http.ResponseWriter, r *http.Request) {
var request GetCardRequest
if err := otohttp.Decode(r, &request); err != nil {
s.server.OnErr(w, r, fmt.Errorf("CardsService GetCard %s", err))
return
}
response, err := s.cardsService.GetCard(r.Context(), request)
if err != nil {
s.server.OnErr(w, r, fmt.Errorf("CardsService GetCard %s", err))
return
}
if err := otohttp.Encode(w, r, http.StatusOK, response); err != nil {
s.server.OnErr(w, r, fmt.Errorf("CardsService GetCard %s", err))
return
}
}
In the above code, we generate:
- A new
CardsService
Go interface that describes the methods of the service. Notice thatcontext.Context
anderror
types were added to the signature of the methods - The
cardsServiceServer
struct couples theotohttp.Server
with the service implementation, and has realhttp.Handler
methods likehandleCardsServiceGetCard
that handles the RPC calls - The
RegisterCardsService
method uses theRegister
method on theotohttp.Server
to bind the routeCardsService.GetCard
to the appropriate handler
All we have to do is write a struct that implements the new CardsService
interface, and the generated code, along with the code inside otohttp
, does the rest for us.
Generating clients
A client is used to access the services on the remote server.
They aren’t always necessary if you’re working with plain-old JSON/HTTP but due to the security measures in Pace, we need to securely sign each request with a secret cryptographic key so we can be sure it is being made by a trusted partner. This can be tricky, so we generate clients that do it for you.
The first client we generate is a Go package, which we use in our tests.
By dog-fooding the generated Go client in our tests, we can not only ensure the services are functioning as we expect them to, but also that the client itself works.
The generated code for the GetCard
client (generated from a template file in the github.com/pacedotdev/pace
project) looks like this:
// GetCard gets a card.
func (s *CardsService) GetCard(ctx context.Context, r GetCardRequest) (*GetCardResponse, error) {
requestBodyBytes, err := json.Marshal(r)
if err != nil {
return nil, errors.Wrap(err, "CardsService.GetCard: marshal GetCardRequest")
}
signature, err := generateSignature(requestBodyBytes, s.client.secret)
if err != nil {
return nil, errors.Wrap(err, "CardsService.GetCard: generate signature GetCardRequest")
}
url := s.client.RemoteHost + "/api/CardsService.GetCard"
s.client.Debug(fmt.Sprintf("POST %s", url))
s.client.Debug(fmt.Sprintf(">> %s", string(requestBodyBytes)))
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(requestBodyBytes))
if err != nil {
return nil, errors.Wrap(err, "CardsService.GetCard: NewRequest")
}
req.Header.Set("X-API-KEY", s.client.apiKey)
req.Header.Set("X-API-SIGNATURE", signature)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept-Encoding", "gzip")
req = req.WithContext(ctx)
resp, err := s.client.HTTPClient.Do(req)
if err != nil {
return nil, errors.Wrap(err, "CardsService.GetCard")
}
defer resp.Body.Close()
var response struct {
GetCardResponse
Error string
}
var bodyReader io.Reader = resp.Body
if strings.Contains(resp.Header.Get("Content-Encoding"), "gzip") {
decodedBody, err := gzip.NewReader(resp.Body)
if err != nil {
return nil, errors.Wrap(err, "CardsService.GetCard: new gzip reader")
}
defer decodedBody.Close()
bodyReader = decodedBody
}
respBodyBytes, err := ioutil.ReadAll(bodyReader)
if err != nil {
return nil, errors.Wrap(err, "CardsService.GetCard: read response body")
}
if err := json.Unmarshal(respBodyBytes, &response); err != nil {
if resp.StatusCode != http.StatusOK {
return nil, errors.Errorf("CardsService.GetCard: (%d) %v", resp.StatusCode, string(respBodyBytes))
}
return nil, err
}
if response.Error != "" {
return nil, errors.New(response.Error)
}
return &response.GetCardResponse, nil
}
- Notice that this code explicitly mentions the input and output objects
GetCardRequest
andGetCardResponse
- this is an example of code that would be replaced when Go gets generics.
JavaScript/TypeScript clients
It isn’t just Go code we can generate, in fact, the templates don’t really know what they’re producing.
We generate a TypeScript client from a slightly modified version of the TypeScript template in Oto, that yields something like this for the GetCard
method:
// GetCard gets a card by ID.
async getCard(getCardRequest: GetCardRequest = null) {
if (getCardRequest == null) {
getCardRequest = new GetCardRequest();
}
const headers: HeadersInit = new Headers();
headers.set('Accept', 'application/json');
headers.set('Content-Type', 'application/json');
await this.client.headers(headers);
const response = await fetch(this.client.basepath + 'CardsService.GetCard', {
method: 'POST',
headers: headers,
body: JSON.stringify(getCardRequest),
})
return response.json().then((json) => {
if (json.Error) {
throw new Error(json.Error);
}
return new GetCardResponse(json);
})
}
Documentation
Our documentation is a Svelte JS app, available to browse at https://pace.dev/docs/api.
We generate the .svelte
components that make up the documentation front-end.
The comments extracted from the original definition files are preserved throughout, and we also use them in the docs.
Extending our API
If we want to make changes to our clients, we only have to modify the tempalte and regenerate the code. We do not have to make the same changes over and over again for every method.
Similarily, if we want to add a new service or method, we only have to:
- Update the definition package code
- Use the oto command to generate server stubs, and client code
- The build fails because the implementation struct no longer matches the generated Go interface (since we added new methods). So we implement the methods to satisfy the interface
Frequently asked questions
A few questions come up again and again, so we’ve answered some of them here.
Why not use gRPC?
gRPC has some benefits over our approach, and also some key disadvantages which ultimately pushed us to build Oto.
gRPC uses a binary protocol, which results in smaller data payload sizes when compared to text-based JSON. In systems with lots of messages flying around, this can make a big difference but isn’t (yet?) a high priority for us due to the nature of the app we’re building. Instead, API discoverability and developer friendliness are more important.
The primary disadvantage we encountered was that the gRPC tooling requires you to open a port for the binary comms, which was not possible to do on Google App Engine (standard environment), where Pace is deployed.
Secondly, rather than obfuscate the message data (humans struggle reading binary) we wanted a more user friendly JSON API which was more familiar to developers. There are a range of great packages that allow you to expose JSON services alongside the binary gRPC ones, but they work by proxying to the binary port, rather than providing a standalone solution within themselves.
So in our case, developer comfortability and familiarity (ours and our future API consumers) is more important than most of the technical arguments that you might make in favour of gRPC.
- Oto uses Go interfaces to describe the API, and in all honesty, we generally try to use Go for as much as we can.
Finally, it’s worth mentioning that Oto has a JSON/HTTP implementation (called otohttp) but that this isn’t the only possible implementation. For example, it would be fairly trivial to add binary support to Oto.
Learn more about what we're doing at Pace.
A lot of our blog posts come out of the technical work behind a project we're working on called Pace.
We were frustrated by communication and project management tools that interrupt your flow and overly complicated workflows turn simple tasks, hard. So we decided to build Pace.
Pace is a new minimalist project management tool for tech teams. We promote asynchronous communication by default, while allowing for those times when you really need to chat.
We shift the way work is assigned by allowing only self-assignment, creating a more empowered team and protecting the attention and focus of devs.
We're currently live and would love you to try it and share your opinions on what project management tools should and shouldn't do.
What next? Start your 14 day free trial to see if Pace is right for your team
First published on 27 Jul 2020 by
or you can share the URL directly:
https://pace.dev/blog/2020/07/27/how-code-generation-wrote-our-api-and-cli.html
Thank you, we don't do ads so we rely on you to spread the word.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK