9

Infrastructure as Code on AWS using Go and Pulumi

 3 years ago
source link: https://dev.to/aws-builders/infrastructure-as-code-on-aws-using-go-and-pulumi-gn5
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Infrastructure as Code on AWS using Go and Pulumi

When we talk about Infrastructure as Code or IaC, the first tool that comes to mind is Terraform. Terraform, created by HashiCorp, has become the standard for documentation and infrastructure management, but its declarative language, HCL (HashiCorp Configuration Language), has some limitations. The main limitation is not being a programming language but a configuration one.

Some alternatives have been emerging to fulfill these needs, such as:

  • AWS Cloud Development Kit, Amazon's solution that allows us to use TypeScript, Python, and Java to program the infrastructure using the cloud provider's solutions;

  • Pulumi, which allows us to use TypeScript, JavaScript, Python, Go, and C# to program infrastructures using solutions from AWS, Microsoft Azure, Google Cloud, and Kubernetes installations.
    I will introduce Pulumi, using the Go language to create some infrastructure examples on AWS.

Installation

To make use of Pulumi, we first need to install its command-line application. Following the documentation, I installed it on my macOS using the command:

brew install pulumi
Enter fullscreen modeExit fullscreen mode

On the website, you can see how to install it on Windows and Linux.

Configure AWS Account Access

Since I will use AWS in this example, the next necessary step is to configure the credentials. For that, I got my access key and secret from the AWS dashboard and set the required environment variables:

export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
Enter fullscreen modeExit fullscreen mode

Creating the project

With the initial dependencies configured, we can now create the project:

mkdir post-pulumi
cd post-pulumi
pulumi new aws-go
Enter fullscreen modeExit fullscreen mode

One of the creation steps requires setting up an account on the Pulumi website. For that, the command-line application opens the browser for this step to be completed. So I logged in with my Github account, completed the registration, returned to the terminal, and continued the project creation without any problems.

You can see the result of running the command can at this link. In addition, at the end of the process, it installs all the necessary dependencies for creating the project in Go.

Files created

Looking at the directory contents, we can see that some configuration files and a main.go were created.

Pulumi.yaml

name: post-pulumi
runtime: go
description: A minimal AWS Go Pulumi program
Enter fullscreen modeExit fullscreen mode

Pulumi.dev.yaml

config:
  aws:region: us-east-1
Enter fullscreen modeExit fullscreen mode

main.go

package main

import (
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create an AWS resource (S3 Bucket)
        bucket, err := s3.NewBucket(ctx, "my-bucket", nil)
        if err != nil {
            return err
        }

        // Export the name of the bucket
        ctx.Export("bucketName", bucket.ID())
        return nil
    })
}
Enter fullscreen modeExit fullscreen mode

When running

pulumi up
Enter fullscreen modeExit fullscreen mode

The bucket was created in S3, as the code indicates.

And the command:

pulumi destroy
Enter fullscreen modeExit fullscreen mode

Destroy all the resources, in this case, the S3 bucket.

First example - creating a static page in S3

Now let's do some more complex examples.

The first step is to create a static page, which we are going to deploy:

mkdir static
Enter fullscreen modeExit fullscreen mode

Inside this directory, I created the file:

static/index.html

<html>
    <body>
        <h1>Hello, Pulumi!</h1>
    </body>
</html>
Enter fullscreen modeExit fullscreen mode

I changed main.go to reflect the new structure:

package main

import (
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create an AWS resource (S3 Bucket)
        bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
            Website: s3.BucketWebsiteArgs{
                IndexDocument: pulumi.String("index.html"),
            },
        })
        if err != nil {
            return err
        }

        // Export the name of the bucket
        ctx.Export("bucketName", bucket.ID())

        _, err = s3.NewBucketObject(ctx, "index.html", &s3.BucketObjectArgs{
            Acl:         pulumi.String("public-read"),
            ContentType: pulumi.String("text/html"),
            Bucket:      bucket.ID(),
            Source:      pulumi.NewFileAsset("static/index.html"),
        })
        if err != nil {
            return err
        }
        ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))
        return nil
    })
}
Enter fullscreen modeExit fullscreen mode

To update run:

pulumi up
Enter fullscreen modeExit fullscreen mode

And confirm the change.

The code snippet:

ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))
Enter fullscreen modeExit fullscreen mode

Generate as output the address to access index.html:

Outputs:
  + bucketEndpoint: "http://my-bucket-357877e.s3-website-us-east-1.amazonaws.com"
Enter fullscreen modeExit fullscreen mode

The case above is a straightforward example, but it already demonstrates the power of the tool. So let's make things a little more complex and fun now.

Second example - a site inside a container

Let's create a Dockerfile with a web server to host our static content:

static/Dockerfile

FROM golang

ADD . /go/src/foo

WORKDIR /go/src/foo
RUN go build -o /go/bin/main

ENTRYPOINT /go/bin/main

EXPOSE 80
Enter fullscreen modeExit fullscreen mode

Let's now create the static/main.go file, which will be our web server:

package main

import (
    "log"
    "net/http"
)

func main() {
    r := http.NewServeMux()
    fileServer := http.FileServer(http.Dir("./"))
    r.Handle("/", http.StripPrefix("/", fileServer))
    s := &http.Server{
        Addr:    ":80",
        Handler: r,
    }
    log.Fatal(s.ListenAndServe())
}
Enter fullscreen modeExit fullscreen mode

Let's change main.go to include the infrastructure of an ECS cluster and everything else needed to run our container:

package main

import (
    "encoding/base64"
    "fmt"
    "strings"

    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
    elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
    "github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Read back the default VPC and public subnets, which we will use.
        t := true
        vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
        if err != nil {
            return err
        }
        subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
        if err != nil {
            return err
        }

        // Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
        webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
            VpcId: pulumi.String(vpc.Id),
            Egress: ec2.SecurityGroupEgressArray{
                ec2.SecurityGroupEgressArgs{
                    Protocol:   pulumi.String("-1"),
                    FromPort:   pulumi.Int(0),
                    ToPort:     pulumi.Int(0),
                    CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
                },
            },
            Ingress: ec2.SecurityGroupIngressArray{
                ec2.SecurityGroupIngressArgs{
                    Protocol:   pulumi.String("tcp"),
                    FromPort:   pulumi.Int(80),
                    ToPort:     pulumi.Int(80),
                    CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
                },
            },
        })
        if err != nil {
            return err
        }

        // Create an ECS cluster to run a container-based service.
        cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
        if err != nil {
            return err
        }

        // Create an IAM role that can be used by our service's task.
        taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
            AssumeRolePolicy: pulumi.String(`{
    "Version": "2008-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}`),
        })
        if err != nil {
            return err
        }
        _, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
            Role:      taskExecRole.Name,
            PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
        })
        if err != nil {
            return err
        }

        // Create a load balancer to listen for HTTP traffic on port 80.
        webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
            Subnets:        toPulumiStringArray(subnet.Ids),
            SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
        })
        if err != nil {
            return err
        }
        webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
            Port:       pulumi.Int(80),
            Protocol:   pulumi.String("HTTP"),
            TargetType: pulumi.String("ip"),
            VpcId:      pulumi.String(vpc.Id),
        })
        if err != nil {
            return err
        }
        webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
            LoadBalancerArn: webLb.Arn,
            Port:            pulumi.Int(80),
            DefaultActions: elb.ListenerDefaultActionArray{
                elb.ListenerDefaultActionArgs{
                    Type:           pulumi.String("forward"),
                    TargetGroupArn: webTg.Arn,
                },
            },
        })
        if err != nil {
            return err
        }

        //create a new ECR repository
        repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
        if err != nil {
            return err
        }

        repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
            creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
                RegistryId: rid,
            })
            if err != nil {
                return nil, err
            }
            data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
            if err != nil {
                fmt.Println("error:", err)
                return nil, err
            }

            return strings.Split(string(data), ":"), nil
        }).(pulumi.StringArrayOutput)
        repoUser := repoCreds.Index(pulumi.Int(0))
        repoPass := repoCreds.Index(pulumi.Int(1))

        //build the image
        image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
            Build: docker.DockerBuildArgs{
                Context: pulumi.String("./static"),
            },
            ImageName: repo.RepositoryUrl,
            Registry: docker.ImageRegistryArgs{
                Server:   repo.RepositoryUrl,
                Username: repoUser,
                Password: repoPass,
            },
        })
        if err != nil {
            return err
        }

        containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
            fmtstr := `[{
                "name": "my-app",
                "image": %q,
                "portMappings": [{
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp"
                }]
            }]`
            return fmt.Sprintf(fmtstr, name), nil
        }).(pulumi.StringOutput)

        // Spin up a load balanced service running NGINX.
        appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
            Family:                  pulumi.String("fargate-task-definition"),
            Cpu:                     pulumi.String("256"),
            Memory:                  pulumi.String("512"),
            NetworkMode:             pulumi.String("awsvpc"),
            RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
            ExecutionRoleArn:        taskExecRole.Arn,
            ContainerDefinitions:    containerDef,
        })
        if err != nil {
            return err
        }
        _, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
            Cluster:        cluster.Arn,
            DesiredCount:   pulumi.Int(5),
            LaunchType:     pulumi.String("FARGATE"),
            TaskDefinition: appTask.Arn,
            NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
                AssignPublicIp: pulumi.Bool(true),
                Subnets:        toPulumiStringArray(subnet.Ids),
                SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
            },
            LoadBalancers: ecs.ServiceLoadBalancerArray{
                ecs.ServiceLoadBalancerArgs{
                    TargetGroupArn: webTg.Arn,
                    ContainerName:  pulumi.String("my-app"),
                    ContainerPort:  pulumi.Int(80),
                },
            },
        }, pulumi.DependsOn([]pulumi.Resource{webListener}))
        if err != nil {
            return err
        }
        // Export the resulting web address.
        ctx.Export("url", webLb.DnsName)
        return nil
    })
}

func toPulumiStringArray(a []string) pulumi.StringArrayInput {
    var res []pulumi.StringInput
    for _, s := range a {
        res = append(res, pulumi.String(s))
    }
    return pulumi.StringArray(res)
}
Enter fullscreen modeExit fullscreen mode

Complex? Yes, but this complexity is inherent to AWS features and not Pulumi. We would have similar complexity if we were using Terraform or CDK.

Before running our code, we need to download the new dependencies:

go get github.com/pulumi/pulumi-docker
go get github.com/pulumi/pulumi-docker/sdk/v3/go/docker
Enter fullscreen modeExit fullscreen mode

Now just run the command:

pulumi up
Enter fullscreen modeExit fullscreen mode

The execution output will generate the URL of the load balancer, which we will use to access the contents of our container in execution.

Reorganizing the code

Now we can start making use of the advantages of a complete programming language like Go. For example, we could use language features like functions, concurrency, conditionals, etc. In this example, we are going to organize our code better. For this, I created the iac directory and the iac/fargate.go file. After that, I moved most of the logic from main.go to the new file:

package iac

import (
    "encoding/base64"
    "fmt"
    "strings"

    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
    elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
    "github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func FargateRun(ctx *pulumi.Context) error {

    // Read back the default VPC and public subnets, which we will use.
    t := true
    vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
    if err != nil {
        return err
    }
    subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
    if err != nil {
        return err
    }

    // Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
    webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
        VpcId: pulumi.String(vpc.Id),
        Egress: ec2.SecurityGroupEgressArray{
            ec2.SecurityGroupEgressArgs{
                Protocol:   pulumi.String("-1"),
                FromPort:   pulumi.Int(0),
                ToPort:     pulumi.Int(0),
                CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
            },
        },
        Ingress: ec2.SecurityGroupIngressArray{
            ec2.SecurityGroupIngressArgs{
                Protocol:   pulumi.String("tcp"),
                FromPort:   pulumi.Int(80),
                ToPort:     pulumi.Int(80),
                CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
            },
        },
    })
    if err != nil {
        return err
    }

    // Create an ECS cluster to run a container-based service.
    cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
    if err != nil {
        return err
    }

    // Create an IAM role that can be used by our service's task.
    taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
        AssumeRolePolicy: pulumi.String(`{
    "Version": "2008-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}`),
    })
    if err != nil {
        return err
    }
    _, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
        Role:      taskExecRole.Name,
        PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
    })
    if err != nil {
        return err
    }

    // Create a load balancer to listen for HTTP traffic on port 80.
    webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
        Subnets:        toPulumiStringArray(subnet.Ids),
        SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
    })
    if err != nil {
        return err
    }
    webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
        Port:       pulumi.Int(80),
        Protocol:   pulumi.String("HTTP"),
        TargetType: pulumi.String("ip"),
        VpcId:      pulumi.String(vpc.Id),
    })
    if err != nil {
        return err
    }
    webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
        LoadBalancerArn: webLb.Arn,
        Port:            pulumi.Int(80),
        DefaultActions: elb.ListenerDefaultActionArray{
            elb.ListenerDefaultActionArgs{
                Type:           pulumi.String("forward"),
                TargetGroupArn: webTg.Arn,
            },
        },
    })
    if err != nil {
        return err
    }

    repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
    if err != nil {
        return err
    }

    repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
        creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
            RegistryId: rid,
        })
        if err != nil {
            return nil, err
        }
        data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
        if err != nil {
            fmt.Println("error:", err)
            return nil, err
        }

        return strings.Split(string(data), ":"), nil
    }).(pulumi.StringArrayOutput)
    repoUser := repoCreds.Index(pulumi.Int(0))
    repoPass := repoCreds.Index(pulumi.Int(1))

    image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
        Build: docker.DockerBuildArgs{
            Context: pulumi.String("./static"),
        },
        ImageName: repo.RepositoryUrl,
        Registry: docker.ImageRegistryArgs{
            Server:   repo.RepositoryUrl,
            Username: repoUser,
            Password: repoPass,
        },
    })
    if err != nil {
        return err
    }

    containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
        fmtstr := `[{
                "name": "my-app",
                "image": %q,
                "portMappings": [{
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp"
                }]
            }]`
        return fmt.Sprintf(fmtstr, name), nil
    }).(pulumi.StringOutput)

    // Spin up a load balanced service running NGINX.
    appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
        Family:                  pulumi.String("fargate-task-definition"),
        Cpu:                     pulumi.String("256"),
        Memory:                  pulumi.String("512"),
        NetworkMode:             pulumi.String("awsvpc"),
        RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
        ExecutionRoleArn:        taskExecRole.Arn,
        ContainerDefinitions:    containerDef,
    })
    if err != nil {
        return err
    }
    _, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
        Cluster:        cluster.Arn,
        DesiredCount:   pulumi.Int(5),
        LaunchType:     pulumi.String("FARGATE"),
        TaskDefinition: appTask.Arn,
        NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
            AssignPublicIp: pulumi.Bool(true),
            Subnets:        toPulumiStringArray(subnet.Ids),
            SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
        },
        LoadBalancers: ecs.ServiceLoadBalancerArray{
            ecs.ServiceLoadBalancerArgs{
                TargetGroupArn: webTg.Arn,
                ContainerName:  pulumi.String("my-app"),
                ContainerPort:  pulumi.Int(80),
            },
        },
    }, pulumi.DependsOn([]pulumi.Resource{webListener}))
    if err != nil {
        return err
    }
    // Export the resulting web address.
    ctx.Export("url", webLb.DnsName)
    return nil
}

func toPulumiStringArray(a []string) pulumi.StringArrayInput {
    var res []pulumi.StringInput
    for _, s := range a {
        res = append(res, pulumi.String(s))
    }
    return pulumi.StringArray(res)
}
Enter fullscreen modeExit fullscreen mode

The next step was to configure the iac directory to be a Go language module:

cd iac
go mod init github.com/eminetto/post-pulumi/iac
cd ..
go mod edit -replace github.com/eminetto/post-pulumi/iac=./iac
go mod tidy
Enter fullscreen modeExit fullscreen mode

Our main.go can now be simplified:

package main

import (
    "github.com/eminetto/post-pulumi/iac"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        return iac.FargateRun(ctx)
    })
}
Enter fullscreen modeExit fullscreen mode

That way, we can better manage the structure of the code that will handle AWS resources. We can reuse this code in other projects, use environment variables, write tests, or whatever else our imagination allows.

Conclusion

Using a tool like Pulumi significantly increases the range of options that we can use in building a project's infrastructure while maintaining readability, code reuse and organization.


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK