Three Ways to Count the Objects in an AWS S3 Bucket
source link: https://fuzzyblog.io/blog/aws/2019/10/24/three-ways-to-count-the-objects-in-an-aws-s3-bucket.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Three Ways to Count the Objects in an AWS S3 Bucket
Oct 24, 2019
AWS S3, "simple storage service", is the classic AWS service. It was the first to launch, the first one I ever used and, seemingly, lies at the very heart of almost everything AWS does.
Given that S3 is essentially a filesystem, a logical thing is to be able to count the files in an S3 bucket. Illustrated below are three ways.
Method 1: aws s3 ls
S3 is fundamentally a filesystem and you can just call ls on it. Yep – ls in the cloud. blink
aws s3 ls s3://adl-ohi/ --recursive --summarize | grep "Total Objects:"
Total Objects: 444803
Method 2: aws s3api
And since S3 is a modern filesystem, it actually has an API that you can call. Yep – a json api. blink blink
aws s3api list-objects --bucket adl-ohi --output json --query "[length(Contents[])]"
[
448444
]
Method 3: A Python Example
Naturally you can just run code to do all this. I started with an example from the Stack Overflow link below that was written for boto and upgraded it to boto3 (as still a Python novice, I feel pretty good about doing this successfully; I remember when Ruby went thru the same AWS v2 to v3 transition and it sucked there too). I also learned how to dynamically introspect methods from Python objects as part of this debugging cycle.
#!/usr/local/bin/python
import sys
import boto3
s3 = boto3.resource('s3')
s3bucket = s3.Bucket(sys.argv[1])
size = 0
totalCount = 0
for key in s3bucket.objects.all():
totalCount += 1
size += key.size
print('total size:')
print("%.3f GB" % (size*1.0/1024/1024/1024))
print('total count:')
print(totalCount)
which gives output like this:
python3 scratch/count_s3.py adl-ohi
total size:
0.298 GB
total count:
486468
Note: I have a live upload happening on another machine so the numbers do change and that's actually fine.
References
Recommend
-
33
S3 (Simple Storage Service) is used to store objects and flat files in 'buckets' in the Cloud. There is unlimited storage available, across 100 buckets, and files can be from 0 bytes to 5TB. St...
-
14
How to Backup Heroku Postgres Database to an Encrypted AWS S3 Bucket Updated May 12, 2020 16 comments 8 minute read
-
10
有时候需要通过 S3 给外部用户交付数据，可通过这种方式实现：创建一个新的 IAM 用户和 S3 bucket，给该用户赋予对应的读写权限。 创建 IAM 用户 创建新的 IAM 用户，不赋予任何权限，生成 access key. 假设新用户的 ARN 是 arn:aws-cn:i...
-
11
Hi Sparta! In this article I will share with you a file storage module that can help you store your user's files (profil's pictures, PDF documents...) in your app with a scalable architecture. We could have stored all those d...
-
8
AWS CloudFormation example for S3 bucket Typical Cloudformation for an S3 bucket with block all public access enabled: Resources: S3BucketExample: Type: AWS::S3::Bucket Properties: BucketName: s3-...
-
3
My own personal AWS S3 bucket I’ve just rediscovered two digital assets that I’d mostly forgotten about. 1. The Reddit username judell, which I created in 2005 and never used. When you visit
-
5
Kuba Martin for Spacelift...
-
8
Create an S3 Bucket on AWS using Terraform In this article, I will show you how to use Terraform to create an S3 bucket on AWS. Before proceeding, I assume that you are familiar with S3 bucket if not then you can click
-
4
How to create an S3 Bucket using Python Boto3 on AWS In this article, we will learn to create an S3 bucket using the Python Boto3 library. We will also see the steps to delete the bucket we created. We will use the "create_bucket" &...
-
14
How to Manage Terraform State in an AWS S3 Bucket In this article, we will see what a Terraform state is and how to manage it on an S3 Bucket. We will also see what "lock" is in Terraform and how to implement it. To implement this,...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK