AWS S3 - Connecting and Getting a List of Objects

Recently I have decided that I wanted to integrate Amazon Web Services (AWS) Simple Storage Solution (S3) into my site. Currently Amazon is offering limited service for free for twelve months (Current as of August 2nd, 2014) at this page. Creating an account does require a credit card at the time.


TL;DR

Why Use any Service?

This comes down to separating user content from the system content. But keeping these entities separated, I can move and redeploy either system without effecting the other very much.

For Example, I have a Turtle Image:
Turtle Image

this Image has nothing to do with the layout or functionality of my site.

Lets now take the example of my server going down, and I need to relocate things quickly. By keeping the file application deployment small by not having the user content integrated with it, I can deploy to a new location very quickly. If the user content system is backup correctly, you can do the same for the content server.

How to

First thing you need is the AWS account. Then follow these steps:
1. Create an Amazon User in the Identity and Account Management (IAM) Portal
2. Create your S3 Bucket
3. Set Permissions on your S3 Bucket to allow public access

IAM Setup

Create a User on IAM and save the credentials, you will need to the account ID and the Secret Key to connect to AWS from your site.

You will need to setup a permission set for the user you just created as well. I am using the current custom permissions for my user.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::[Bucket Name]",
                "arn:aws:s3:::[Bucket Name]/*"
            ]
        }
    ]
}

this allows for my user to connect to my S3 Bucket and subsequent folders and files.

S3 Setup

Next we are going create a bucket in the S3 Portal. once you have created a bucket, you'll want to select the properties screen.

Under Permissions, select the option to add more permissions, and set your the Grantee to Everyone, and allow them the permission to the List items.

Create a second permission for Authenticated Users and allow them to List and Upload/Delete.

Next we want to set up CORS with following settings

<?xml version="1.0" encoding="UTF-8"?>  
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">  
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
    <CORSRule>
        <AllowedOrigin>[YOUR WEBSITE ADDRESS]</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>  

This will allow your site and users the ability to view images you upload, but it will also allow your website to connect and make changes to everything in the bucket.

That should be all you need to do for setup on the AWS side.

The Javascript

Now that you have your server setup, we can use the AWS SDK's. I am using the Javsscript SDK to connect to AWS and get information out of my S3 Bucket.

First we need to connect to AWS, then we can get our bucket information:

$.getScript('https://sdk.amazonaws.com/js/aws-sdk-2.0.11.min.js', function() {
    AWS.config.update({
        accessKeyId: "[AWS KEY]",
        secretAccessKey: "[AWS SECRET]"
    });
    AWS.config.region = '[REGION BUCKET]';
    var s3bucket = new AWS.S3({ params: {Bucket: _s3Bucket} });
    s3bucket.listObjects({}, function(err, data) {
        if (err)  {
            console.log(err, err.stack); // an error occurred
        } else { 
            console.log(data); // dump the bucket data
        }
    });
});

A couple of things to note about the data being returned:
1. Data is actually returned as XML, not JSON. The AWS service converts the XML to JSON for you.
2. You will get a full list of all your items in the bucket, including the folders as separate array items. I used javascript to restructure the data into a file/folder format for my ease of use.

Conclusion

AWS S3 Service is an easy way to serve items on your site. It prevents content data from mixing with application data, and is fairly easy to interact with.

Next on the list, I am going to start Uploading Items into S3 directly from my site.