Read file content from S3 bucket with boto3

Posted on

Question :

Read file content from S3 bucket with boto3

I read the filenames in my S3 bucket by doing

objs = boto3.client.list_objects(Bucket='my_bucket')
    while 'Contents' in objs.keys():
        objs_contents = objs['Contents']
        for i in range(len(objs_contents)):
            filename = objs_contents[i]['Key']

Now, I need to get the actual content of the file, similarly to a open(filename).readlines(). What is the best way?

Asked By: mar tin


Answer #1:

boto3 offers a resource model that makes tasks like iterating through objects easier. Unfortunately, StreamingBody doesn’t provide readline or readlines.

s3 = boto3.resource('s3')
bucket = s3.Bucket('test-bucket')
# Iterates through all the objects, doing the pagination for you. Each obj
# is an ObjectSummary, so it doesn't contain the body. You'll need to call
# get to get the whole body.
for obj in bucket.objects.all():
    key = obj.key
    body = obj.get()['Body'].read()
Answered By: Jordon Phillips

Answer #2:

You might also consider the smart_open module, which supports iterators:

from smart_open import smart_open

# stream lines from an S3 object
for line in smart_open('s3://mybucket/mykey.txt', 'rb'):

and context managers:

with smart_open('s3://mybucket/mykey.txt', 'rb') as s3_source:
    for line in s3_source:
         print(line.decode('utf8'))  # seek to the beginning
    b1000 =  # read 1000 bytes

Find smart_open at

Answered By: caffreyd

Answer #3:

When you want to read a file with a different configuration than the default one, feel free to use either directly or the copy-pasted code:

def s3_read(source, profile_name=None):
    Read a file from an S3 source.

    source : str
        Path starting with s3://, e.g. 's3://bucket-name/key/'
    profile_name : str, optional
        AWS profile

    content : bytes

        Botocore is not able to find your credentials. Either specify
        profile_name or add the environment variables AWS_ACCESS_KEY_ID,
    session = boto3.Session(profile_name=profile_name)
    s3 = session.client('s3')
    bucket_name, key =
    s3_object = s3.get_object(Bucket=bucket_name, Key=key)
    body = s3_object['Body']
Answered By: Martin Thoma

Answer #4:

If you already know the filename, you can use the boto3 builtin download_fileobj

import boto3

from io import BytesIO

session = boto3.Session()
s3_client = session.client("s3")

f = BytesIO()
s3_client.download_fileobj(bucket_name, filename, f)
Answered By: reubano

Answer #5:

Using the client instead of resource:

s3 = boto3.client('s3')
result = s3.list_objects(Bucket = bucket, Prefix='/something/')
for o in result.get('Contents'):
    data = s3.get_object(Bucket=bucket, Key=o.get('Key'))
    contents = data['Body'].read()
Answered By: Climbs_lika_Spyder

Leave a Reply

Your email address will not be published. Required fields are marked *