Forward systemd service logs to AWS Cloudwatch

I often find myself ssh'ing into my servers and checking my systemd service logs with $ journalctl -f -u {name}.service. One day I got tired of this and wanted all of my important logs in once place (Amazon AWS Cloudwatch). To my dismay, there weren't any real good tutorials on how to do so. So, voilà.

Overall, it's a fairly simple process consisting of the following few steps.

Open the service file with $ sudo vi /lib/systemd/system/{name}.service

Modify the [Service] section:

[Service]
...
StandardOutput=file:/var/log/{name}/logs.log
StandardError=file:/var/log/{name}/logs.log

Next, create the directory $ sudo mkdir /var/log/{name}

Finally, restart the service:

sudo systemctl daemon-reload
sudo systemctl stop {name}.service
sudo systemctl start {name}.service

Wait a little bit, and confirm logs are being written $ cat /var/log/{name}/logs.log

$ mkdir /tmp/cloudwatch-logs && cd /tmp/cloudwatch-logs
$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
$ sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Be sure to download the appropriate agent for your OS

See, here (copied below for convenience).

1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
2. In the navigation pane on the left, choose Roles and then Create role.
3. For Choose the service that will use this role, choose EC2 Allows EC2 instances to call AWS services on your behalf. Choose Next: Permissions.
4. In the list of policies, select the check box next to CloudWatchAgentServerPolicy. If necessary, use the search box to find the policy.
5. Choose Next: Review.
6. Confirm that CloudWatchAgentServerPolicy appears next to Policies. In Role name, enter a name for the role, such as CloudWatchAgentServerRole. Optionally give it a description. Then choose Create role.

See, here (copied below for convenience)

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the navigation pane, choose Instances.
3. Select the instance, choose Actions, Instance Settings, Attach/Replace IAM role.
4. Select the IAM role to attach to your instance, and choose Apply.

1. Navigate to https://console.aws.amazon.com/cloudwatch
2. Click Logs from the left menu. Then click Actions > Create log group. Name it /{service}/.
3. Click on the newly greated log group. Then click Create Log Stream. Name it logs.

$ mkdir ~/cloudwatch && cd ~/cloudwatch
$ vi config.json

Copy/paste the below json being sure to change settings as appropriate:

{
        "agent": {
                "metrics_collection_interval": 60,
                "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/{service}/logs.log",
                                                "log_group_name": "/{service}/",
                                                "log_stream_name": "logs",
                                                "timezone": "UTC"
                                        }
                                ]
                        }
                },
                "log_stream_name": "logs",
                "force_flush_interval" : 60
        }
}

Finally, start the cloudwatch agent $ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/home/ubuntu/cloudwatch/config.json -s