Amazon Simple Storage Service Developer Guide

553 Pages·2014·4.54 MB·English

Amazon Simple Storage Service
Developer Guide
API Version 2006-03-01
Amazon Simple Storage Service Developer Guide
Amazon Simple Storage Service: Developer Guide
Copyright © 2014 Amazon Web Services, Inc. and/or its affiliates. All rights reserved.
The following are trademarks of Amazon Web Services, Inc.: Amazon, Amazon Web Services Design, AWS, Amazon CloudFront,
Cloudfront, Amazon DevPay, DynamoDB, ElastiCache, Amazon EC2, Amazon Elastic Compute Cloud, Amazon Glacier, Kindle, Kindle
Fire, AWS Marketplace Design, Mechanical Turk, Amazon Redshift, Amazon Route 53, Amazon S3, Amazon VPC. In addition,
Amazon.com graphics, logos, page headers, button icons, scripts, and service names are trademarks, or trade dress of Amazon in
the U.S. and/or other countries. Amazon's trademarks and trade dress may not be used in connection with any product or service that
is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits
Amazon.
All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected
to, or sponsored by Amazon.
Amazon Simple Storage Service Developer Guide
Table of Contents
What Is Amazon S3? ..................................................................................................................... 1
How Do I...? ......................................................................................................................... 1
Introduction .................................................................................................................................. 2
Overview of Amazon S3 and This Guide.................................................................................... 2
Advantages to Amazon S3...................................................................................................... 2
Amazon S3 Concepts ............................................................................................................ 3
Buckets ....................................................................................................................... 3
Objects........................................................................................................................ 3
Keys ........................................................................................................................... 4
Regions ....................................................................................................................... 4
Amazon S3 Data Consistency Model................................................................................ 5
Features .............................................................................................................................. 7
Reduced Redundancy Storage ........................................................................................ 7
Bucket Policies.............................................................................................................. 7
AWS Identity and Access Management............................................................................. 8
Access Control Lists ...................................................................................................... 8
Versioning .................................................................................................................... 8
Operations ................................................................................................................... 8
Amazon S3 Application Programming Interfaces (API)................................................................. 9
The REST Interface....................................................................................................... 9
The SOAP Interface....................................................................................................... 9
Paying for Amazon S3............................................................................................................ 9
Related Services ................................................................................................................ 10
Making Requests......................................................................................................................... 11
About Access Keys .............................................................................................................. 11
AWS Account Access Keys ........................................................................................... 11
IAM User Access Keys ................................................................................................. 12
Temporary Security Credentials...................................................................................... 12
Request Endpoints .............................................................................................................. 13
Making Requests Using the AWS SDKs................................................................................... 14
Using AWS Account or IAM User Credentials................................................................... 15
Using IAM User Temporary Credentials........................................................................... 20
Using Federated User Temporary Credentials................................................................... 31
Making Requests Using the REST API.................................................................................... 47
Virtual Hosting of Buckets ............................................................................................. 48
Request Redirection and the REST API........................................................................... 52
Buckets...................................................................................................................................... 55
Restrictions and Limitations................................................................................................... 56
Rules for Naming......................................................................................................... 56
Configuration Options........................................................................................................... 57
Buckets and Regions.................................................................................................... 57
Bucket Website Configuration ................................................................................................ 59
Using the AWS Management Console............................................................................. 60
Using the SDK for Java................................................................................................. 60
Using the SDK for .NET................................................................................................ 63
Using the SDK for PHP................................................................................................. 66
Using the REST API..................................................................................................... 68
Requester Pays Buckets ....................................................................................................... 69
Configure Requester Pays by Using the Amazon S3 Console.............................................. 69
Configure Requester Pays with the REST API.................................................................. 70
DevPay and Requester Pays.......................................................................................... 72
Charge Details ............................................................................................................ 72
Buckets and Access Control.................................................................................................. 73
Billing and Reporting of Buckets............................................................................................. 73
Cost Allocation Tagging................................................................................................. 73
API Version 2006-03-01
iii
Amazon Simple Storage Service Developer Guide
Configuration Errors ............................................................................................................. 74
Objects ...................................................................................................................................... 75
Object Key and Metadata...................................................................................................... 76
Object Keys ................................................................................................................ 76
Object Metadata.......................................................................................................... 78
Subresources ..................................................................................................................... 80
Versioning .......................................................................................................................... 80
Lifecycle Management.......................................................................................................... 83
Overview.................................................................................................................... 84
Lifecycle Configuration Elements.................................................................................... 85
Object Archival............................................................................................................ 93
Object Expiration ......................................................................................................... 95
Specifying a Lifecycle Configuration................................................................................ 95
Cross-Origin Resource Sharing............................................................................................ 107
Cross-Origin Resource Sharing: Examples..................................................................... 108
How Do I Enable CORS on My Bucket?......................................................................... 108
How Does Amazon S3 Evaluate the CORS Configuration On a Bucket?.............................. 110
Using the AWS Management Console........................................................................... 111
Using the SDK for Java............................................................................................... 111
Using the SDK for .NET .............................................................................................. 116
Using the REST API................................................................................................... 122
Troubleshooting CORS Issues...................................................................................... 122
Operations on Objects ........................................................................................................ 122
Multipart Upload Overview........................................................................................... 123
Getting Objects.................................................................................................................. 127
Related Resources..................................................................................................... 127
Using the SDK for Java............................................................................................... 128
Using the SDK for .NET .............................................................................................. 131
Using the SDK for PHP............................................................................................... 134
Using the REST API................................................................................................... 136
Share an Object with Others........................................................................................ 136
Uploading Objects.............................................................................................................. 142
Related Resources..................................................................................................... 142
Uploading Objects in a Single Operation........................................................................ 143
Uploading Objects Using Multipart Upload API................................................................ 151
Uploading Objects Using Pre-Signed URLs.................................................................... 187
Copying Objects ................................................................................................................ 193
Related Resources..................................................................................................... 194
Copying Objects in a Single Operation........................................................................... 195
Copying Objects Using the Multipart Upload API............................................................. 204
Listing Object Keys............................................................................................................. 212
Iterating Through Multi-Page Results............................................................................. 213
Using a Prefix and Delimiter......................................................................................... 213
Using the SDK for Java............................................................................................... 214
Using the SDK for .NET .............................................................................................. 217
Using the SDK for PHP............................................................................................... 219
Using the REST API................................................................................................... 222
Related Resources..................................................................................................... 222
Deleting Objects ................................................................................................................ 222
Deleting Objects from a Version-Enabled Bucket............................................................. 222
Deleting Objects from an MFA-Enabled Bucket................................................................ 223
Related Resources..................................................................................................... 223
Deleting One Object Per Request................................................................................. 223
Deleting Multiple Objects Per Request........................................................................... 233
Restoring Objects .............................................................................................................. 255
Using the Console...................................................................................................... 256
Using the SDK for Java............................................................................................... 257
Using the SDK for .NET .............................................................................................. 259
API Version 2006-03-01
iv
Amazon Simple Storage Service Developer Guide
Using the REST API................................................................................................... 262
Managing Access ...................................................................................................................... 263
Introduction....................................................................................................................... 263
Overview .................................................................................................................. 264
How Amazon S3 Authorizes a Request.......................................................................... 269
Guidelines for Using the Available Access Policy Options.................................................. 274
Example Walkthroughs: Managing Access...................................................................... 277
Using Bucket Policies and User Policies................................................................................. 305
Access Policy Language Overview................................................................................ 305
Bucket Policy Examples .............................................................................................. 329
User Policy Examples................................................................................................. 335
Managing Access with ACLs ............................................................................................... 356
Access Control List (ACL) Overview.............................................................................. 356
Managing ACLs ......................................................................................................... 361
Data Protection.......................................................................................................................... 373
Data Encryption................................................................................................................. 373
Server-Side Encryption ............................................................................................... 374
Client-Side Encryption ................................................................................................ 397
Reduced Redundancy Storage............................................................................................. 409
Setting the Storage Class of an Object You Upload.......................................................... 410
Changing the Storage Class of an Object in Amazon S3................................................... 410
Versioning ........................................................................................................................ 411
Enabling a Bucket's Versioning State............................................................................. 412
Adding Objects to Versioning-Enabled Buckets................................................................ 417
Listing the Objects in a Versioning-Enabled Bucket.......................................................... 417
Retrieving Object Versions........................................................................................... 419
Deleting Object Versions ............................................................................................. 421
Transitioning Object Versions ....................................................................................... 426
Restoring Previous Versions ........................................................................................ 426
Versioned Object Permissions and ACLs........................................................................ 427
Working with Versioning-Suspended Buckets.................................................................. 428
Hosting a Static Website ............................................................................................................. 433
Website Endpoints ............................................................................................................. 434
Key Differences Between the Amazon Website and the REST API Endpoint......................... 435
Configure a Bucket for Website Hosting................................................................................. 436
Overview .................................................................................................................. 436
Syntax for Specifying Routing Rules.............................................................................. 438
Index Document Support ............................................................................................ 442
Custom Error Document Support ................................................................................. 443
Configuring a Redirect ................................................................................................ 445
Permissions Required for Website Access...................................................................... 447
Example Walkthroughs ....................................................................................................... 447
Example: Setting Up a Static Website............................................................................ 448
Example: Setting Up a Static Website Using a Custom Domain.......................................... 449
Setting Up Notification of Bucket Events......................................................................................... 457
Request Routing........................................................................................................................ 461
Request Redirection and the REST API................................................................................. 461
Overview .................................................................................................................. 461
DNS Routing............................................................................................................. 461
Temporary Request Redirection.................................................................................... 462
Permanent Request Redirection ................................................................................... 464
DNS Considerations........................................................................................................... 465
Performance Optimization ........................................................................................................... 466
Request Rate and Performance Considerations ...................................................................... 466
Workloads with a Mix of Request Types ......................................................................... 467
GET-Intensive Workloads ............................................................................................ 469
TCP Window Scaling.......................................................................................................... 469
TCP Selective Acknowledgement.......................................................................................... 470
API Version 2006-03-01
v
Amazon Simple Storage Service Developer Guide
BitTorrent.................................................................................................................................. 471
How You are Charged for BitTorrent Delivery........................................................................... 471
Using BitTorrent to Retrieve Objects Stored in Amazon S3........................................................ 472
Publishing Content Using Amazon S3 and BitTorrent................................................................ 473
Amazon DevPay ........................................................................................................................ 474
Amazon S3 Customer Data Isolation..................................................................................... 474
Example................................................................................................................... 475
Amazon DevPay Token Mechanism....................................................................................... 475
Amazon S3 and Amazon DevPay Authentication..................................................................... 475
Amazon S3 Bucket Limitation............................................................................................... 476
Amazon S3 and Amazon DevPay Process.............................................................................. 477
Additional Information ......................................................................................................... 477
Error Handling........................................................................................................................... 478
The REST Error Response.................................................................................................. 478
Response Headers .................................................................................................... 479
Error Response ......................................................................................................... 479
The SOAP Error Response.................................................................................................. 480
Amazon S3 Error Best Practices........................................................................................... 480
Retry InternalErrors.................................................................................................... 480
Tune Application for Repeated SlowDown errors.............................................................. 480
Isolate Errors ............................................................................................................ 481
Server Access Logging ............................................................................................................... 482
Overview .......................................................................................................................... 482
Log Object Key Format ............................................................................................... 483
How are Logs Delivered? ............................................................................................ 483
Best Effort Server Log Delivery..................................................................................... 483
Bucket Logging Status Changes Take Effect Over Time.................................................... 484
Related Topics................................................................................................................... 484
Enabling Logging Using Console .......................................................................................... 484
Enabling Logging Programmatically....................................................................................... 486
Enabling logging ........................................................................................................ 486
Granting the Log Delivery Group Write Permission........................................................... 486
Example: AWS SDK for .NET....................................................................................... 487
Log Format ....................................................................................................................... 489
Custom Access Log Information................................................................................... 491
Programming Considerations for Extensible Server Access Log Format............................... 491
Additional Logging for Copy Operations ......................................................................... 491
Deleting Log Files .............................................................................................................. 493
AWS SDKs and Explorers ........................................................................................................... 494
Specifying Signature Version in Request Authentication............................................................ 495
Using the AWS SDK for Java............................................................................................... 496
The Java API Organization.......................................................................................... 497
Testing the Java Code Examples.................................................................................. 497
Using the AWS SDK for .NET .............................................................................................. 498
The .NET API Organization.......................................................................................... 498
Testing the .NET Code Examples.................................................................................. 499
Using the AWS SDK for PHP and Running PHP Examples........................................................ 499
AWS SDK for PHP Levels............................................................................................ 499
Running PHP Examples.............................................................................................. 500
Related Resources..................................................................................................... 501
Using the AWS SDK for Ruby............................................................................................... 501
The Ruby API Organization ......................................................................................... 501
Testing the Ruby Script Examples................................................................................. 501
Using the AWS SDK for Python (Boto)................................................................................... 502
Appendices............................................................................................................................... 503
Appendix A: Using the SOAP API......................................................................................... 503
Common SOAP API Elements ..................................................................................... 503
Authenticating SOAP Requests .................................................................................... 504
API Version 2006-03-01
vi
Amazon Simple Storage Service Developer Guide
Setting Access Policy with SOAP.................................................................................. 505
Appendix B: Authenticating Requests (AWS Signature Version 2)............................................... 506
Authenticating Requests Using the REST API................................................................. 508
Signing and Authenticating REST Requests ................................................................... 509
Browser-Based Uploads Using POST............................................................................ 520
Resources ................................................................................................................................ 537
Document History ...................................................................................................................... 539
AWS Glossary........................................................................................................................... 546
API Version 2006-03-01
vii
Amazon Simple Storage Service Developer Guide
How Do I...?
What Is Amazon S3?
Amazon Simple Storage Service is storage for the Internet. It is designed to make web-scale computing
easier for developers.
Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of
data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable,
reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of
web sites.The service aims to maximize benefits of scale and to pass those benefits on to developers.
This guide explains the core concepts of Amazon S3, such as buckets and objects, and how to work with
these resources using the Amazon S3 application programming interface (API).
How Do I...?
Information Relevant Sections
General product overview and pricing Amazon Simple Storage Service (Amazon S3)
Get a quick hands-on introduction to Amazon Simple Storage Service Getting Started Guide
Amazon S3
Learn about Amazon S3 key terminology Introduction to Amazon S3 (p.2)
and concepts
How do I work with buckets? Working with Amazon S3 Buckets (p.55)
How do I work with objects? Working with Amazon S3 Objects (p.75)
How do I make requests? Making Requests (p.11)
How do I manage access to my Managing Access Permissions to Your Amazon S3
resources? Resources (p.263)
API Version 2006-03-01
1
Amazon Simple Storage Service Developer Guide
Overview of Amazon S3 and This Guide
Introduction to Amazon S3
This introduction to Amazon Simple Storage Service is intended to give you a detailed summary of this
web service. After reading this section, you should have a good idea of what it offers and how it can fit
in with your business.
Topics
• Overview of Amazon S3 and This Guide (p.2)
• Advantages to Amazon S3 (p.2)
• Amazon S3 Concepts (p.3)
• Features (p.7)
• Amazon S3 Application Programming Interfaces (API) (p.9)
• Paying for Amazon S3 (p.9)
• Related Services (p.10)
Overview of Amazon S3 and This Guide
Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of
data, at any time, from anywhere on the web.
This guide describes how you send requests to create buckets, store and retrieve your objects, and
manage permissions on your resources.The guide also describes access control and the authentication
process. Access control defines who can access objects and buckets within Amazon S3, and the type of
access (e.g., READ and WRITE).The authentication process verifies the identity of a user who is trying
to access Amazon Web Services (AWS).
Advantages to Amazon S3
Amazon S3 is intentionally built with a minimal feature set that focuses on simplicity and robustness.
Following are some of advantages of the Amazon S3 service:
• Create Buckets – Create and name a bucket that stores data. Buckets are the fundamental container
in Amazon S3 for data storage.
API Version 2006-03-01
2
Amazon Simple Storage Service Developer Guide
Amazon S3 Concepts
• Store data in Buckets – Store an infinite amount of data in a bucket. Upload as many objects as you
like into an Amazon S3 bucket. Each object can contain up to 5 TB of data. Each object is stored and
retrieved using a unique developer-assigned key.
• Download data – Download your data or enable others to do so. Download your data any time you
like or allow others to do the same.
• Permissions – Grant or deny access to others who want to upload or download data into your Amazon
S3 bucket. Grant upload and download permissions to three types of users. Authentication mechanisms
can help keep data secure from unauthorized access.
• Standard interfaces – Use standards-based REST and SOAP interfaces designed to work with any
Internet-development toolkit.
Note
SOAP support over HTTP is deprecated, but it is still available over HTTPS. New Amazon S3
features will not be supported for SOAP.We recommend that you use either the REST API
or the AWS SDKs.
Amazon S3 Concepts
Topics
• Buckets (p.3)
• Objects (p.3)
• Keys (p.4)
• Regions (p.4)
• Amazon S3 Data Consistency Model (p.5)
This section describes key concepts and terminology you need to understand to use Amazon S3 effectively.
They are presented in the order you will most likely encounter them.
Buckets
A bucket is a container for objects stored in Amazon S3. Every object is contained in a bucket. For example,
if the object named photos/puppy.jpg is stored in the johnsmith bucket, then it is addressable using
the URL http://johnsmith.s3.amazonaws.com/photos/puppy.jpg
Buckets serve several purposes: they organize the Amazon S3 namespace at the highest level, they
identify the account responsible for storage and data transfer charges, they play a role in access control,
and they serve as the unit of aggregation for usage reporting.
You can configure buckets so that they are created in a specific Region. For more information, see
Buckets and Regions (p.57).You can also configure a bucket so that every time an object is added to
it, Amazon S3 generates a unique version ID and assigns it to the object. For more information, see
Versioning (p.411).
For more information about buckets, see Working with Amazon S3 Buckets (p.55).
Objects
Objects are the fundamental entities stored in Amazon S3. Objects consist of object data and metadata.
The data portion is opaque to Amazon S3.The metadata is a set of name-value pairs that describe the
object.These include some default metadata, such as the date last modified, and standard HTTP metadata,
such as Content-Type.You can also specify custom metadata at the time the object is stored.
API Version 2006-03-01
3