Implement Rate-limiting with Redis and Golang on a Linux Server

<?xml encoding="utf-8" ??>

Introduction

Rate-limiting is the concept of blocking users, third-party applications, and bots to prevent abuse in your application. In this security model, you put a cap on the maximum times a client can execute a specific task in your application within a given time frame.

In a production environment, you'll implement rate-limiting in login forms, API endpoints, money transfer scripts, and more. In most cases, you'll be doing this to prevent brute force attacks, DoS and DDoS attacks, Web scraping, and duplicate actions.

While it is possible to implement rate-limiting using relational database servers like MySQL or PostgreSQL, this approach only works for small-scale applications. Using the disk-based databases for that purpose usually results in scalability issues when your applications' user-base grows to thousands or millions of clients.

The best approach is to use an in-memory database like Redis as the primary database for your rate-limiting modules. Redis saves and analyzes rate-limiting data using your server's RAM, making it highly responsive and scalable for such use-case.

In this guide, you'll use Golang to create a custom HTTP API that returns the current date and time from your Linux server. This tool may be used to synchronize the date/time for IoT devices. Then, to prevent abuse to the API's endpoint, you'll implement the rate-limiting concept with a Redis server.

Prerequisites

To complete this tutorial, ensure you have the following:

A Linux server configured with a basic level security.
A non-root user with sudo privileges.
A Redis server.
A Golang package.

1. Create a `main.go` File

The Golang package ships with a very powerful net/http library that provides HTTP functionalities in your application. Your application will listen for incoming requests on port 8080 for this tutorial. You'll create these functionalities under a main.go file. Then, your tool will query a function to retrieve the current date/time and return the response to any HTTP client (for instance, curl or a web browser) in JSON format.

Before you start coding, you need to create a separate directory for your source codes. Then, SSH to your server and execute the following steps.

Create a project directory.
```
$ mkdir project
```
Switch to the new project directory.
```
$ cd project
```
Then, create a main.go file. This file runs when you execute the application.
```
$ nano main.go
```

Enter the following information into the main.go file.

package main

import (

"encoding/json"
"fmt"
"net/http"

)

func main() {
    http.HandleFunc("/test", httpHandler)            
    http.ListenAndServe(":8080", nil)
}

func httpHandler(w http.ResponseWriter, req *http.Request) { 

    var err error
    resp := map[string]interface{}{}           

    resp, err = getTime() 

    enc := json.NewEncoder(w)
    enc.SetIndent("", "  ") 

    if err != nil {
        resp = map[string]interface{}{"error": err.Error(),}     
    }

    if err := enc.Encode(resp); err != nil {
        fmt.Println(err.Error())
    }

}

Save and close the file. In the above file, you're importing 3 packages. The encoding/json package allows you to output responses back to the client in JSON format. Then, you're using the fmt library to format and output basic strings. You're also implementing the HTTP functions using the net/http package.
Under the main(){ ... } function , you're using the statement http.HandleFunc("/test", httpHandler) to register a handler function(httpHandler(...){...}) for incoming HTTP requests. Then, you've used the statement http.ListenAndServe(":8080", nil) to listen and serve HTTP content.
Under the httpHandler(...){...} function, you're calling a getTime(){...} function, which you'll code later in a new file to output the current date/time. Then, you're outputting the response in JSON format using the statement if err := enc.Encode(resp)....
Your main.go file is now ready. Next, you'll create the timeapi.go file which holds the getTime(){...} function that you've referenced in the main.go file.

2. Create a `timeapi.go` File

Before you write the actual source code that rate-limits your app to promote fair usage, here are a few terms that are commonly used with this technology.

Maximum limit: This is the total number of requests or retries a client can make in a given time. For instance, you may want to limit users to a maximum of 5 requests in a minute. Software vendors use different names for this variable—for instance, maximum retries, maximum requests, and more.
Limit time: This is the time interval that you can define in seconds, minutes, hours, or even days within which requests should pass through before the limit (defined by the Maximum limit variable) is reached and an error returned by the application. Some companies might use ban time, time intervals, or other terms depending on their use-cases.

To understand the above variables better, the settings for an application that allows users to execute only 5 API requests within 60 seconds may be similar to the following code snippet.

  maxLimit  := 5
  limitTime := 60

Having understood those variables, you can now proceed to code the actual application.

Use nano to create the timeapi.go file
```
$ nano timeapi.go
```

Then, enter the following information into the timeapi.go file.

package main

import (

    "context"
"errors"
"time"

"github.com/go-redis/redis"

)

func getTime()(map[string]interface{}, error){

    var maxLimit  int64
    var limitTime int64

    maxLimit  = 5
    limitTime = 60

    systemUser := "john_doe"

    uniqueKey  := systemUser

    ctx := context.Background()

    redisClient := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
        Password: "",
        DB: 0,
    })

    var counter int64

    counter, err := redisClient.Get(ctx, uniqueKey).Int64()

    if err == redis.Nil {

        err = redisClient.Set(ctx, uniqueKey, 1, time.Duration(limitTime) * time.Second).Err()

        if err != nil {
            return nil, err
        }

        counter = 1

    } else if err != nil {

        return nil, err

    } else {

        if counter >= maxLimit {
            return nil, errors.New("Limit reached.")
        }

        counter, err = redisClient.Incr(ctx, uniqueKey).Result()

    if err != nil {
    return nil, err
    }                
    }

    dt := time.Now()

    res := map[string]interface{}{
               "data" : dt.Format("2006-01-02 15:04:05"),   
           } 

    return res, nil
}

Save and close the file when you're through with editing.
In the above file, you've implemented the context package and the ctx := context.Background() statement to pass requests to the Redis server without any deadline/timeout. Next, you're using the errors package to provide custom errors in your application. The time library allows you to set the limit time for the HTTP clients and to provide the current date/time to the API users. To implement Redis functions, you're using the github.com/go-redis/redis package.
In the getTime(...)(...){...} function, you've defined two variables. That is, maxLimit = 5 and limitTime = 60. This means your application will only accept 5 requests within a timeframe of 60 seconds.
Then, you've defined a systemUser variable and assigned it a value of john_doe. You're using this as the unique key(uniqueKey := systemUser) to track users in your application. In a production environment, you should retrieve the usernames once the users get authenticated into the system for applications that require a form of authentication.
Next, you're initializing a new Redis client using the statement redisClient := redis.NewClient(&redis.Options{ Addr: "localhost:6379", Password: "", DB: 0, }).
Then, you've initialized a zero-based counter(var counter int64). This counter tracks the user-based Redis key(uniqueKey) to retrieve the number of times a client has made requests to your HTTP resource.
You're using the statement if err == redis.Nil {...} to check if a client/API user has previously made any requests to your application. In case this is a first-time client, you're creating a Redis key with a value of 1 using the statement err = redisClient.Set(ctx, uniqueKey, 1, time.Duration(limitTime) * time.Second).Err(). However, if a key for the user exists in the database, you're incrementing the counter value using the statement ...counter, err = redisClient.Incr(ctx, uniqueKey).Result()... You're only incrementing the counter if the application's limit is not violated. You're checking this using the statement ...if counter >= maxLimit { return nil, errors.New("Limit reached.")}....
Towards the end, you're returning the current date/time to the calling function using the statement ...dt := time.Now() res := map[string]interface{}{ "data" : dt.Format("2006-01-02 15:04:05"), }.... This code only fires if there are no rate-limiting errors encountered.
Your application is now ready for testing.

3. Test the Golang Rate-limiting Functionality

With all the source codes in place, you can start sending HTTP requests to your API endpoint to test if the application works as expected.

Before you run the application, download the Redis package for Golang.
```
$ go get github.com/go-redis/redis
```
Then, run the application by executing the following statement. Please note: The below command has a blocking function. Don't enter any other command on your current SSH terminal window.
```
$ go run ./
```
Next, connect to your server in a new terminal window through SSH and use curl to send 20 requests to the http://localhost:8080/test endpoint.
```
$ curl http://localhost:8080/test?[1-20]
```

You should get the following output.

[1/20]: http://localhost:8080/test?1 --> <stdout>
--_curl_--http://localhost:8080/test?1
{
  "data": "2022-01-20 10:08:13"
}
...
[5/20]: http://localhost:8080/test?5 --> <stdout>
--_curl_--http://localhost:8080/test?5
{
  "data": "2022-01-20 10:08:13"
}

[6/20]: http://localhost:8080/test?6 --> <stdout>
--_curl_--http://localhost:8080/test?6
{
  "error": "Limit reached."
}
...

The above output shows that your application is working as expected, and the 6th request was blocked. Wait 60 seconds(1 minute) and run the curl command below.
```
$ curl http://localhost:8080/test
```
You should be able to get the time in JSON format since the 60 seconds ban time has already expired.
```
{
  "data": "2022-01-20 10:11:12"
}
```
Your Redis rate-limiting application with Golang is working as expected.

Conclusion

You've created an API that returns the current server's date and time in JSON format in this guide. You've then used the Redis server to implement a rate-limiting model in the application to promote fair usage of your application.

Follow the guides below to read more about Golang and Redis.

Want to contribute?

You could earn up to $600 by adding new articles

Submit your article Suggest an update Request an article

Introduction

Prerequisites

1. Create a `main.go` File

2. Create a `timeapi.go` File

3. Test the Golang Rate-limiting Functionality

Conclusion

Want to contribute?

Recommend

How to Install Portainer on Ubuntu 20.04 with Docker

How to Install Mailtrain Newsletter Application on Debian 9

Configuring BGP on Vultr

A Quick Guide to Node.js in 2019

How to Install ExpressionEngine CMS on Ubuntu 18.04 LTS

Reset Passwords and SSH Keys in Fedora CoreOS

Install YetiForce on Ubuntu 20.04

How to Migrate Mailboxes Easily With Imapsync on Ubuntu

Configure Rocky Linux with Multiple IP Addresses

How to Install Automad CMS on Fedora 30

About Joyk

Implement Rate-limiting with Redis and Golang on a Linux Server

Introduction

Prerequisites

1. Create a main.go File

2. Create a timeapi.go File

3. Test the Golang Rate-limiting Functionality

Conclusion

Want to contribute?

Recommend

About Joyk

1. Create a `main.go` File

2. Create a `timeapi.go` File