GitHub - ellimilial/sqs-s3-logger: Automated serverless logging to S3 via SQS.
source link: https://github.com/ellimilial/sqs-s3-logger
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
sqs-s3-logger
A library to persist messages on S3 using serverless architecture. It is mainly targeted at cheaply archiving low-volume, sporadic events from applications without a need to spin additional infrastructure.
What it’s not
Not a replacement for general logging systems or libraries. Provides no filtering or aggregation.
AWS Alternatives
Usage
Configure boto3
’s credentials as per:
http://boto3.readthedocs.io/en/latest/guide/quickstart.html#configuration
Make sure you setup:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
(optionally)
Take a look at main.py
.
For help: python3 main.py -h
For example (backup at midnight each Saturday from app-logs
queue to
app-logs-archive
bucket):
sqs-s3-logger create -b app-logs-archive -q app-logs -f app-logs-backup -s 'cron(0 0 ? * SAT *)'
Sending messages to a queue
Ideally you should use another AWS IAM user with permissions restricted to getting SQS queues and writing messages.
import boto3 sqs = boto3.resource('sqs') queue = sqs.get_queue_by_name(QueueName='<QUEUE_NAME>') queue.send_message(MessageBody='<MSG_BODY_NOT_EXCEEDING_256KB>')
Limitations
- Maximum SQS message size is limited to 256 KB
- There could be no more than 120,000 messages in a queue at a time.
- SQS messages cannot persist for longer than 14 days.
- Lambda environment has up to 512MB of ephemeral disk capacity.
- By default it does not guarantee correct time-based ordering
You may need to adjust your CRON settings depending on your volume.
Testing
python3 setup.py test
These will use your AWS account to instantiate a temporary integration environment.
Recommend
-
92
Zero Allocation JSON Logger The zerolog package provides a fast and simple logger dedicated to JSON output. Zerolog's API is designed to provide both a great developer experience and stunning
-
54
README.md wlog Simple log level based Go logger. Provides an io.Writer that filters log messages based on a log level prefix. Valid log levels are: DEB...
-
25
README.md pino
-
5
Working with Logs in Ruby Logging is one of the primary tasks an application usually addresses. Logs are used when you need to, for example, see what’s happening inside of your apps, monitor them, or c...
-
6
Copy link Contributor casperisfine c...
-
9
New issue set log level for global zap logger #870 Closed ...
-
7
SQS Message Mover sqsmover is a tool for moving AWS SQS messages from one queue to another. Useful when you need to move deadletter queue messages back into the original queue. Features Reliable deli...
-
5
API with NestJS #86. Logging with the built-in logger when using raw SQL We value your privacy
-
4
sqs-to-sns sqs-to-sns is an utility written in Go to forward messages from AWS SQS Queues to AWS SNS Topics. Created by gh-md-toc Build and r...
-
2
@luca1iuLuca Liua Business Intelligence Developer with a passion for all things dataReceive Stories fro...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK