Join the community!
Visit our GitHub or LinkedIn page to join the Tilburg Science Hub community, or check out our contributors' Hall of Fame!
Want to change something or add new content? Click the Contribute button!
AWS S3 Buckets
Amazon Simple Storage Service (AWS S3) is an industry-leader object storage service that allows you to store and retrieve any kind of files fast, securely and anywhere in the world - basically Dropbox on steroids. It can be used for a range of use cases, from websites to backup and archives. It can also be particularly useful for research projects - storing a huge amount of raw data, for instance.
If you are using AWS S3 to store data necessary for your research and want to incorporate downloading and uploading your data files to AWS S3 into a make script, it is useful to use a code script to interact with your AWS S3 bucket instead of the command line. Keep reading to learn how to do it.
TipSome terminology
You will often read about AWS S3 buckets. What are they? Put simply, a bucket is a container for objects (files) stored in Amazon S3. Every object must be contained in a bucket. A single user can have as many buckets as they want. Inside a bucket, you can create as many folders you want.
WarningYou need to have access to an AWS S3 bucket and have the credentials to run the following code.
Whenever you interact with AWS, you must specify your security credentials to verify who you are and whether you have permission to access the resources that you are requesting. For instance, if you want to download a file from an AWS S3 bucket, your credentials must allow that access.
Learn more about AWS credentials here.
Code
We show you how to very simply upload and download files to a S3 bucket with boto3.
First off, install boto3 - the AWS SDK for Python - via pip:
pip install boto3
Establish a connection
# Import the necessary packages
import boto3
# Define credentials and make sure you have reading and writing privileges on AWS user settings
your_aws_access_key_id = "" # Enter your AWS Access Key
your_aws_secret_access_key = "" # Enter your AWS Secret Key
your_aws_region_name = "" # Enter your AWS bucket region
# Create an S3 client
s3 = boto3.client("s3",
region_name = your_aws_region_name,
aws_access_key_id = your_aws_access_key_id,
aws_secret_access_key = your_aws_secret_access_key)
your_aws_bucket_name = "" # Specify the bucket you use
List objects in your AWS bucket and print them
# List objects in your AWS bucket and print them
objects = s3.list_objects(Bucket = your_aws_bucket_name)
print(objects)
List the names of the objects in bucket contents
# List the names of the objects in bucket contents
for object in objects["Contents"]:
print(object["Key"])
Upload a file
# To upload a file
# Source: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html
filetoupload = open("hello.txt", "w")
# Upload the file to S3
s3.upload_file(filetoupload, your_aws_bucket_name, "remote-object-name.txt")
Download a file
# Download a file
# Source: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html
s3.download_file(your_aws_bucket_name, "remote-object-name.txt", "local-file-name.txt")
See also
- Learn more about AWS S3
- Learn more about
boto3, the AWS SDK for Python - Learn more on how to upload files with
boto3 - Learn more on how to download files with
boto3
SummaryIn this article, we showed you how to use Python to automate your uploads to and downloads from AWS S3. Hopefully, this is helpful to you.