5

Write your AWS DevOps tool in Haskell for Great Good

 2 years ago
source link: https://blog.oyanglul.us/devops/write-aws-tool-in-haskell
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Write your AWS DevOps tool in Haskell for Great Good

Write your AWS DevOps tool in Haskell for Great Good

Table of Contents

Let's just simply compare AWS CLI, Haskell Amazonka, and official Go SDK with the following very simple tasks:

List files in one S3 bucket

AWS CLI https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

aws s3 ls s3://mybucket --recursive --page-size 100

Haskell Amazonka https://hackage.haskell.org/package/amazonka-s3-1.6.1/docs/Network-AWS-S3-ListObjects.htm

send $ listObjects "mybucket" & loMaxKeys ~? 100

Of course the Haskell code is pure and lazy, so send just returns a data that describes the command and produce no side effect, to actually exec the command we can create a function awsRun to execute the command:

awsRun = runResourceT . runAWS (newEnv Discover & envLogger .~ newLogger Info stdout) . within Sydney

It is similar to the step that Go SDK need to create a session first:

sess := session.Must(session.NewSession(&aws.Config{
    Region: aws.String(endpoints.ApSoutheast2RegionID),
}))

Go SDK https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.ListObjects

svc := s3.New(session.New())
input := &s3.ListObjectsInput{
    Bucket:  aws.String("mybucket"),
    MaxKeys: aws.Int64(100),
}

result, err := svc.ListObjects(input)
if err != nil {
 ...

List ALL FILES in a bucket

This time we need all, not just one page of files, let's compare

AWS CLI

Can't really do it…if there are more than 1000 files 🤷‍♂

–page-size (integer) The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed) https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html#options

Haskell Amazonka

paginate $ listObjects "mybucket" & loMaxKeys ~? 100

Simply replace send with paginate, amazonka will automatically pull objects into a Conduit https://github.com/snoyberg/conduit stream, you can simply map over the stream, .| takeC 3 to go over only first 3 pages, or even .| sinkList to load the very large list into memory.

Go SDK https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.ListObjectsPagesWithContext

objects := []string{}
err := svc.ListObjectsPagesWithContext(ctx, &s3.ListObjectsInput{
    Bucket: aws.String(myBucket),
}, func(p *s3.ListObjectsOutput, lastPage bool) bool {
    for _, o := range p.Contents {
        objects = append(objects, aws.StringValue(o.Key))
    }
    return true // continue paging
})
if err != nil {
    panic(fmt.Sprintf("failed to list objects for bucket, %s, %v", myBucket, err))
}

fmt.Println("Objects in bucket:", objects)

Go SDK uses callback function to go over pages, it is less efficient then stream and more verbose.

Query a DynamoDB table

AWS CLI https://docs.aws.amazon.com/cli/latest/reference/dynamodb/query.html

aws dynamodb query \
    --expression-attribute-values file://put-a-json-here \
    --key-condition-expression "Artist = :v1" \
    --projection-expression "SongTitle" \
    --table-name Music

Haskell Amazonka

send $ query "Music"
  & qExpressionAttributeValues .~ HashMap.singleton ":v1" (attributeValue & avS ?~ "No One You Know")
  & qKeyConditionExpression ?~ "Artist = :v1"
  & qProjectionExpression ?~ "SongTitle"

Go SDK

svc := dynamodb.New(session.New())
input := &dynamodb.QueryInput{
    ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
        ":v1": {
            S: aws.String("No One You Know"),
        },
    },
    KeyConditionExpression: aws.String("Artist = :v1"),
    ProjectionExpression:   aws.String("SongTitle"),
    TableName:              aws.String("Music"),
}

result, err := svc.Query(input)
if err != nil {
    ...

You should get the idea by now.

I guess anyone even can't read Haskell at all can identify the Haskell version is basically the same as AWS CLI, with some simple syntax mapping you can instantly translate any AWS CLI command into Haskell code.

  • -- to &
  • kebab-case to CamelCase
  • connect option name and value with .~ instead of space, or ?~ when the option is optional

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK