gRPC + REST API on AWS

Showing a setup of gRPC service which is also exposed as a REST API. It’s a setup that happens to work for us. No alternatives will be discussed in this post.

This is a concise blog post.

Architecture

1. ALB with HTTPS listener (trivially configured, out of scope of this post)
2. ECS running a task with 3 containers:
   • API Gateway. Implemented by Envoy. does:
     • requests authorization using the service in next container
     • proxies gRPC requests
     • proxies REST requests (converting them to upstream gRPC requests).
   • authorization service implemented with OPA
   • Our gRPC application

Notes

Health checks are not in very good shape yet

ECS Configuration (Simplified Excerpt)

In case the reader is not familiar, it CloudFormation below.

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        - Name: apigw
          Image: !Ref ApiGwImage
          PortMappings:
            - ContainerPort: !Ref ContainerPort
        - Name: opa
          Image: !Ref OpaImage
          PortMappings:
            - ContainerPort: 9191
        - Name: app
          Image: !Ref AppImage
          PortMappings:
            - ContainerPort: 4000

  Service:
    DependsOn:
      - GrpcListenerRule
      - RestListenerRule
      - GrpcTargetGroup
      - RestTargetGroup
    Type: AWS::ECS::Service
    Properties:
      ServiceName: !Ref ServiceName
      Cluster: !Ref Cluster
      TaskDefinition: !Ref TaskDefinition
      LoadBalancers:
        - ContainerName: apigw
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref GrpcTargetGroup
        - ContainerName: apigw
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref RestTargetGroup

  GrpcTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckTimeoutSeconds: 5
      Matcher:
        GrpcCode: "0-99"
      UnhealthyThresholdCount: 2
      HealthyThresholdCount: 2
      Port: !Ref ContainerPort
      Protocol: HTTP
      ProtocolVersion: GRPC
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 60 # default is 300
      TargetType: ip
      VpcId: !ImportValue VpcId

  RestTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /rest/not-found
      HealthCheckTimeoutSeconds: 5
      Matcher:
        HttpCode: 404
      UnhealthyThresholdCount: 2
      HealthyThresholdCount: 2
      Port: !Ref ContainerPort
      Protocol: HTTP
      ProtocolVersion: HTTP1
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 60 # default is 300
      TargetType: ip
      VpcId: !ImportValue VpcId

  GrpcListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref GrpcTargetGroup
      Conditions:
        - Field: path-pattern
          PathPatternConfig:
            Values:
              - '/censored.v1.CensoredService/*'
              - '/censored.v1.CensoredAdminService/*'
              - '/censored.v1.CensoredSystemService/*'
      ListenerArn: ...
      Priority: 1000

  RestListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref RestTargetGroup
      Conditions:
        - Field: path-pattern
          PathPatternConfig:
            Values:
              - '/rest/v1/*'
      ListenerArn: ...
      Priority: 1001

Envoy Configuration (Simplified Excerpt)

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 8000
      filter_chains:
        - filters:
            - name: Connection Manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                via: CensoredGW
                route_config:
                  name: Static response for tests
                  virtual_hosts:
                    - name: backend
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/test/static"
                          direct_response:
                            status: 200
                            body:
                              inline_string: "Static response for tests"
                        # Reference: https://envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/grpc_json_transcoder_filter#route-configs-for-transcoded-requests
                        - match:
                            prefix: "/"
                          route:
                            cluster: upstream
                            timeout: 60s
                http_filters:
                  - name: envoy.filters.http.grpc_json_transcoder
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_json_transcoder.v3.GrpcJsonTranscoder
                      # maybe disable later:
                      auto_mapping: true
                      proto_descriptor: "../path/to/proto_descriptor.bin" ### See next heading in this post
                      services:
                        - censored.v1.CensoredService
                        - censored.v1.CensoredAdminService
                        - censored.v1.CensoredSystemService
                      print_options:
                        add_whitespace: true
                        always_print_primitive_fields: true
                      request_validation_options:
                        reject_unknown_method: true
                        reject_unknown_query_parameters: true
                  - name: envoy.filters.http.cors
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
                  - name: envoy.ext_authz
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
                      failure_mode_allow: false
                      with_request_body:
                        max_request_bytes: 10485760 # 10M
                        allow_partial_message: false
                        pack_as_bytes: true
                      transport_api_version: V3
                      grpc_service:
                        envoy_grpc:
                          cluster_name: opa-agent
                        timeout: 10s
                  - name: envoy.filters.http.router
                    # https://github.com/envoyproxy/envoy/issues/21464
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                always_set_request_id_in_response: true
                access_log:
                  - typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                      # https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-default-format

  # Based on https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/grpc_json_transcoder_filter
  clusters:
    - name: opa-agent
      connect_timeout: 0.25s
      type: STRICT_DNS
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: { }
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 9191
    - name: upstream
      type: STRICT_DNS
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      load_assignment:
        cluster_name: grpc
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 4000

proto_descriptor.bin

GrpcJsonTranscoder must have the proto descriptor file in order to know how to transcode. The file contains:

1. proto definitions of your services, including extension that describes how to expose the services as REST
2. dependencies of the above proto definitions

The descriptor file is generated using a command similar to the following:

buf build -o proto_descriptor.bin --as-file-descriptor-set --path path/to/my.proto

buf is a way to manage .proto files and their dependencies (very imprecise definition, sorry)

If I remember correctly, you can generate the descriptor with protoc (without buf) but I don’t remember how.

grpcurl

Same descriptor file is used with grpcurl when you later test your service from the command line:

grpcurl -H "Authorization: Bearer ..." -protoset proto_descriptor.bin "example.com:443" censored.service.name/MyFunc

my.proto

This is how a protobuf definition with REST extension looks like (excerpt):

import "google/api/annotations.proto";

service Censored {
  rpc MyCreate(CreateRequest) returns (CreateResponse){
    option (google.api.http) = { post: "/rest/v1/my-objs" };
  }
  rpc MyGet(GetRequest) returns (GetResponse) {
    option (google.api.http) = { get: "/rest/v1/my-objs/{id}" };
  }
}

Excerpt from buf.yaml corresponding to the import above:

version: v1

deps:
  - buf.build/googleapis/googleapis

---

Hope this helps.

Sorry, I was in a rush to get this out. If anything is unclear or missing, please let me know.