
geralt / Pixabay
List all files in AWS S3 was one of the most recent requirement that I had to work on. Also, write them into a file and the easiest solution that I found was using the python boto.
Below is the code that I used to make this work. While I worked on this, it also made me realize the extensive capabalities of boto. Look at its documentation, you can do a lot more. Hope this helps someone.
#!/usr/bin/python # -*- coding: utf-8 -*- from boto.s3.connection import S3Connection import requests # Used to get the temporary credentials r = requests.get('URL which provides the temporary Credentials') data = r.json() # Extracting the required fields AccessKeyId = data['AccessKeyId'] SecretAccessKey = data['SecretAccessKey'] Token = data['Token'] my_file = open('results.txt', 'w') # making a connectiont o S3 conn = S3Connection(aws_access_key_id=AccessKeyId, aws_secret_access_key=SecretAccessKey, security_token=Token) # Accessing the required bucket bucket = conn.get_bucket('your-bucket-name') # Choosing the right prefix for key in bucket.list(prefix='foldername1/foldername2/foldername3/'): # print key.name.encode('utf-8') my_file.write(key.name.encode('utf-8') + '\n') my_file.close()