Table Of Content

Amazon Simple Storage Service User Guide API Version 2006-03-01 Amazon Simple Storage Service User Guide Amazon Simple Storage Service: User Guide Amazon Simple Storage Service User Guide Table of Contents What is Amazon S3? ........................................................................................................................... 1 Features of Amazon S3 ............................................................................................................... 1 Storage classes.................................................................................................................. 1 Storage management......................................................................................................... 1 Access management........................................................................................................... 2 Data processing................................................................................................................. 2 Storage logging and monitoring.......................................................................................... 2 Analytics and insights......................................................................................................... 3 Strong consistency............................................................................................................. 3 How Amazon S3 works ............................................................................................................... 3 Buckets............................................................................................................................. 4 Objects............................................................................................................................. 4 Keys................................................................................................................................. 5 S3 Versioning.................................................................................................................... 5 Version ID ......................................................................................................................... 5 Bucket policy..................................................................................................................... 5 S3 Access Points................................................................................................................ 5 Access control lists (ACLs)................................................................................................... 6 Regions............................................................................................................................. 6 Amazon S3 data consistency model .............................................................................................. 6 Concurrent applications...................................................................................................... 7 Related services......................................................................................................................... 8 Accessing Amazon S3 ................................................................................................................. 9 Amazon Web Services Management Console ......................................................................... 9 Amazon Command Line Interface......................................................................................... 9 Amazon SDKs.................................................................................................................... 9 Amazon S3 REST API.......................................................................................................... 9 Paying for Amazon S3 .............................................................................................................. 10 PCI DSS compliance.................................................................................................................. 10 Getting started ................................................................................................................................ 11 Setting up............................................................................................................................... 11 Sign up for an Amazon Web Services account...................................................................... 12 Secure IAM users .............................................................................................................. 12 Step 1: Create a bucket ............................................................................................................. 12 Step 2: Upload an object .......................................................................................................... 14 Step 3: Download an object...................................................................................................... 15 Using the S3 console ........................................................................................................ 15 Step 4: Copy an object ............................................................................................................. 15 Step 5: Delete the objects and bucket ........................................................................................ 16 Deleting an object ............................................................................................................ 16 Emptying your bucket ....................................................................................................... 17 Deleting your bucket ........................................................................................................ 17 Next steps............................................................................................................................... 17 Understand common use cases .......................................................................................... 18 Control access to your buckets and objects .......................................................................... 18 Manage and monitor your storage ...................................................................................... 19 Develop with Amazon S3 .................................................................................................. 19 Learn from tutorials ......................................................................................................... 20 Explore training and support ............................................................................................. 21 Access control.......................................................................................................................... 22 Creating a new bucket ...................................................................................................... 22 Storing and sharing data................................................................................................... 23 Sharing resources............................................................................................................. 24 Protecting data................................................................................................................ 24 API Version 2006-03-01 iii Amazon Simple Storage Service User Guide Tutorials.......................................................................................................................................... 27 Getting started........................................................................................................................ 20 Optimizing storage costs........................................................................................................... 20 Hosting videos and websites...................................................................................................... 20 Data processing ........................................................................................................................ 20 Protecting data........................................................................................................................ 20 Transforming data with S3 Object Lambda.................................................................................. 28 Prerequisites.................................................................................................................... 29 Step 1: Create an S3 bucket .............................................................................................. 30 Step 2: Upload a file to the S3 bucket ................................................................................ 31 Step 3: Create an S3 access point ....................................................................................... 31 Step 4: Create a Lambda function ...................................................................................... 32 Step 5: Configure an IAM policy for your Lambda function's execution role ............................... 36 Step 6: Create an S3 Object Lambda access point ................................................................. 36 Step 7: View the transformed data..................................................................................... 37 Step 8: Clean up.............................................................................................................. 39 Next steps....................................................................................................................... 41 Detecting and redacting PII data ................................................................................................ 42 Prerequisites: Create an IAM user with permissions ............................................................... 43 Step 1: Create an S3 bucket .............................................................................................. 44 Step 2: Upload a file to the S3 bucket ................................................................................ 45 Step 3: Create an S3 access point ....................................................................................... 45 Step 4: Configure and deploy a prebuilt Lambda function ...................................................... 46 Step 5: Create an S3 Object Lambda access point ................................................................. 47 Step 6: Use the S3 Object Lambda access point to retrieve the redacted file ............................. 48 Step 7: Clean up.............................................................................................................. 48 Next steps....................................................................................................................... 51 Hosting video streaming........................................................................................................... 52 Prerequisites: Register and configure a custom domain with Route 53 ..................................... 52 Step 1: Create an S3 bucket .............................................................................................. 53 Step 2: Upload a video to the S3 bucket ............................................................................. 54 Step 3: Create a CloudFront origin access identity................................................................ 54 Step 4: Create a CloudFront distribution .............................................................................. 54 Step 5: Access the video through the CloudFront distribution ................................................. 56 Step 6: Configure your CloudFront distribution to use your custom domain name ...................... 57 Step 7: Access the S3 video through the CloudFront distribution with the custom domain name .. 60 (Optional) Step 8: View data about requests received by your CloudFront distribution ................ 60 Step 9: Clean up.............................................................................................................. 61 Next steps....................................................................................................................... 64 Batch-transcoding videos.......................................................................................................... 64 Prerequisites.................................................................................................................... 65 Step 1: Create an S3 bucket for the output media files ......................................................... 65 Step 2: Create an IAM role for MediaConvert ....................................................................... 67 Step 3: Create an IAM role for your Lambda function ............................................................ 67 Step 4: Create a Lambda function for video transcoding ........................................................ 69 Step 5: Configure Amazon S3 Inventory for your S3 source bucket .......................................... 81 Step 6: Create an IAM role for S3 Batch Operations .............................................................. 83 Step 7: Create and run an S3 Batch Operations job .............................................................. 86 Step 8: Check the output media files from your S3 destination bucket ..................................... 89 Step 9: Clean up.............................................................................................................. 89 Next steps....................................................................................................................... 91 Configuring a static website...................................................................................................... 92 Step 1: Create a bucket ..................................................................................................... 92 Step 2: Enable static website hosting .................................................................................. 92 Step 3: Edit Block Public Access settings ............................................................................. 93 Step 4: Add a bucket policy that makes your bucket content publicly available .......................... 94 Step 5: Configure an index document ................................................................................. 95 API Version 2006-03-01 iv Amazon Simple Storage Service User Guide Step 6: Configure an error document .................................................................................. 96 Step 7: Test your website endpoint .................................................................................... 97 Step 8: Clean up.............................................................................................................. 97 Configuring a static website using a custom domain ..................................................................... 97 Before you begin .............................................................................................................. 98 Step 1: Register a custom domain with Route 53 .................................................................. 98 Step 2: Create two buckets ................................................................................................ 98 Step 3: Configure root Domain bucket ................................................................................ 99 Step 4: Configure subdomain bucket for redirect ................................................................ 100 Step 5: Configure logging ................................................................................................ 101 Step 6: Upload index and website content ......................................................................... 101 Step 7: Upload an error document .................................................................................... 102 Step 8: Edit Block Public Access ....................................................................................... 103 Step 9: Attach a bucket policy ......................................................................................... 104 Step 10: Test your domain endpoint ................................................................................. 105 Step 11: Add alias records ............................................................................................... 106 Step 12: Test the website ................................................................................................ 109 Speeding up your website with Amazon CloudFront ............................................................ 109 Cleaning up example resources ........................................................................................ 112 Working with buckets ..................................................................................................................... 114 Buckets overview.................................................................................................................... 114 About permissions.......................................................................................................... 115 Managing public access to buckets ................................................................................... 115 Bucket configuration....................................................................................................... 116 Naming rules......................................................................................................................... 118 Example bucket names .................................................................................................... 119 Creating a bucket ................................................................................................................... 119 Default settings for new S3 buckets FAQ ................................................................................... 125 Viewing bucket properties ....................................................................................................... 126 Methods for accessing a bucket ................................................................................................ 127 Virtual-hosted–style access.............................................................................................. 127 Path-style access............................................................................................................ 127 Accessing an S3 bucket over IPv6 ..................................................................................... 128 Accessing a bucket through S3 access points ...................................................................... 128 Accessing a bucket using S3:// ......................................................................................... 128 Emptying a bucket ................................................................................................................. 129 Deleting a bucket ................................................................................................................... 130 Setting default bucket encryption ............................................................................................ 133 Using SSE-KMS encryption for cross-account operations ...................................................... 134 Using default encryption with replication .......................................................................... 134 Using Amazon S3 Bucket Keys with default encryption ........................................................ 135 Enabling default encryption ............................................................................................. 135 Monitoring default encryption .......................................................................................... 137 Configuring Transfer Acceleration ............................................................................................. 138 Why use Transfer Acceleration? ........................................................................................ 138 Requirements for using Transfer Acceleration ..................................................................... 138 Getting Started .............................................................................................................. 139 Enabling Transfer Acceleration ......................................................................................... 140 Speed Comparison tool................................................................................................... 145 Using Requester Pays .............................................................................................................. 145 How Requester Pays charges work .................................................................................... 146 Configuring Requester Pays ............................................................................................. 146 Retrieving the requestPayment configuration ..................................................................... 148 Downloading objects in Requester Pays buckets ................................................................. 148 Restrictions and limitations ...................................................................................................... 149 Working with objects ...................................................................................................................... 151 Objects.................................................................................................................................. 151 API Version 2006-03-01 v Amazon Simple Storage Service User Guide Subresources.................................................................................................................. 152 Creating object keys ............................................................................................................... 152 Object key naming guidelines .......................................................................................... 153 Working with metadata ........................................................................................................... 155 System-defined object metadata ...................................................................................... 156 User-defined object metadata .......................................................................................... 157 Editing object metadata .................................................................................................. 159 Uploading objects................................................................................................................... 160 Using multipart upload ........................................................................................................... 169 Multipart upload process ................................................................................................. 169 Checksums with multipart upload operations ..................................................................... 170 Concurrent multipart upload operations ............................................................................ 171 Multipart upload and pricing ........................................................................................... 171 API support for multipart upload ..................................................................................... 171 Amazon Command Line Interface support for multipart upload ............................................ 172 Amazon SDK support for multipart upload ........................................................................ 172 Multipart upload API and permissions ............................................................................... 172 Configuring a lifecycle policy ........................................................................................... 174 Uploading an object using multipart upload ...................................................................... 176 Uploading a directory ..................................................................................................... 189 Listing multipart uploads ................................................................................................. 191 Tracking a multipart upload ............................................................................................. 193 Aborting a multipart upload ............................................................................................ 195 Copying an object .......................................................................................................... 199 Multipart upload limits .................................................................................................... 204 Copying objects...................................................................................................................... 204 To copy an object ........................................................................................................... 205 Downloading an object ........................................................................................................... 211 Checking object integrity ......................................................................................................... 218 Using supported checksum algorithms .............................................................................. 218 Using Content-MD5 when uploading objects ...................................................................... 224 Using Content-MD5 and the ETag to verify uploaded objects ............................................... 224 Using trailing checksums................................................................................................. 225 Using part-level checksums for multipart uploads ............................................................... 225 Deleting objects..................................................................................................................... 226 Programmatically deleting objects from a version-enabled bucket ........................................ 227 Deleting objects from an MFA-enabled bucket .................................................................... 227 Deleting a single object ................................................................................................... 227 Deleting multiple objects ................................................................................................. 234 Organizing and listing objects .................................................................................................. 246 Using prefixes................................................................................................................ 247 Listing objects................................................................................................................ 248 Using folders................................................................................................................. 258 Viewing an object overview ............................................................................................. 261 Viewing object properties ................................................................................................ 261 Using presigned URLs ............................................................................................................. 262 Limiting presigned URL capabilities ................................................................................... 262 Who can create a presigned URL ...................................................................................... 263 When does Amazon S3 check the expiration date and time of a presigned URL? ...................... 264 Sharing objects.............................................................................................................. 264 Uploading objects........................................................................................................... 268 Deleting an object .......................................................................................................... 282 Transforming objects.............................................................................................................. 284 Creating Object Lambda access points .............................................................................. 286 Using Amazon S3 Object Lambda Access Points ................................................................. 295 Security considerations.................................................................................................... 296 Writing Lambda functions ............................................................................................... 300 API Version 2006-03-01 vi Amazon Simple Storage Service User Guide Using Amazon built functions .......................................................................................... 321 Best practices and guidelines for S3 Object Lambda ........................................................... 322 S3 Object Lambda tutorials ............................................................................................. 324 Debugging S3 Object Lambda .......................................................................................... 324 Working with access points .............................................................................................................. 325 Configuring IAM policies.......................................................................................................... 325 Access point policy examples ........................................................................................... 326 Condition keys............................................................................................................... 328 Delegating access control to access points ......................................................................... 329 Granting permissions for cross-account access points .......................................................... 330 Creating access points ............................................................................................................. 330 Rules for naming Amazon S3 access points ........................................................................ 330 Creating an access point .................................................................................................. 331 Creating access points restricted to a VPC ......................................................................... 332 Managing public access ................................................................................................... 334 Using access points ................................................................................................................. 335 Monitoring and logging................................................................................................... 335 Managing access points ................................................................................................... 337 Using a bucket-style alias for your access point .................................................................. 339 Using access points with Amazon S3 operations ................................................................. 340 Restrictions and limitations ...................................................................................................... 342 Working with Multi-Region Access Points ........................................................................................... 344 Creating Multi-Region Access Points .......................................................................................... 346 Rules for naming Amazon S3 Multi-Region Access Points ..................................................... 347 Rules for choosing buckets for Amazon S3 Multi-Region Access Points ................................... 348 Blocking public access with Amazon S3 Multi-Region Access Points ....................................... 348 Creating Amazon S3 Multi-Region Access Points ................................................................. 349 Configuring Amazon PrivateLink ....................................................................................... 350 Using a Multi-Region Access Point ............................................................................................ 352 Multi-Region Access Point hostnames ................................................................................ 353 Multi-Region Access Points and Amazon S3 Transfer Acceleration ......................................... 354 Multi-Region Access Point permissions .............................................................................. 354 Request routing.............................................................................................................. 358 Failover configuration..................................................................................................... 359 Bucket replication........................................................................................................... 364 Supported operations..................................................................................................... 367 Managing Multi-Region Access Points ........................................................................................ 373 Monitoring and logging........................................................................................................... 373 Monitoring and logging requests made to Multi-Region Access Point management APIs ........... 374 Using CloudTrail............................................................................................................. 375 Restrictions and limitations ...................................................................................................... 375 Security......................................................................................................................................... 378 Data protection...................................................................................................................... 379 Data encryption..................................................................................................................... 379 Server-side encryption.................................................................................................... 380 Using client-side encryption ............................................................................................. 424 Internetwork privacy............................................................................................................... 428 Traffic between service and on-premises clients and applications .......................................... 428 Traffic between Amazon resources in the same Region ........................................................ 429 Amazon PrivateLink for Amazon S3 .......................................................................................... 429 Types of VPC endpoints .................................................................................................. 429 Restrictions and limitations of Amazon PrivateLink for Amazon S3 ........................................ 430 Creating a VPC endpoint ................................................................................................. 430 Accessing Amazon S3 interface endpoints .......................................................................... 430 Accessing buckets and S3 access points from S3 interface endpoints ..................................... 431 Updating an on-premises DNS configuration ...................................................................... 434 Creating a VPC endpoint policy ........................................................................................ 435 API Version 2006-03-01 vii Amazon Simple Storage Service User Guide Identity and access management .............................................................................................. 438 Overview....................................................................................................................... 438 Access policy guidelines ................................................................................................... 444 Request authorization..................................................................................................... 448 Bucket policies and user policies ....................................................................................... 456 Amazon managed policies ............................................................................................... 611 Managing access with ACLs .............................................................................................. 613 Using CORS ................................................................................................................... 632 Blocking public access ..................................................................................................... 643 Reviewing bucket access .................................................................................................. 653 Verifying bucket ownership .............................................................................................. 658 Controlling object ownership ................................................................................................... 661 Object Ownership settings ............................................................................................... 663 Changes introduced by disabling ACLs ............................................................................... 664 Prerequisites for disabling ACLs ........................................................................................ 665 Object Ownership permissions ......................................................................................... 666 Disabling ACLs for all new buckets ................................................................................... 666 Replication and Object Ownership .................................................................................... 667 Setting Object Ownership ................................................................................................ 667 Prerequisites for disabling ACLs ........................................................................................ 668 Creating a bucket ........................................................................................................... 676 Setting Object Ownership ................................................................................................ 679 Viewing Object Ownership settings ................................................................................... 682 Disabling ACLs for all new buckets ................................................................................... 683 Troubleshooting............................................................................................................. 684 Logging and monitoring.......................................................................................................... 687 Compliance Validation............................................................................................................. 688 Resilience.............................................................................................................................. 689 Backup encryption.......................................................................................................... 691 Infrastructure security............................................................................................................. 692 Configuration and vulnerability analysis .................................................................................... 693 Security Best Practices ............................................................................................................ 694 Amazon S3 preventative security best Practices .................................................................. 694 Amazon S3 Monitoring and auditing best practices ............................................................. 697 Managing storage........................................................................................................................... 699 Using S3 Versioning ................................................................................................................ 699 Unversioned, versioning-enabled, and versioning-suspended buckets ..................................... 700 Using S3 Versioning with S3 Lifecycle ............................................................................... 700 S3 Versioning ................................................................................................................. 701 Enabling versioning on buckets ........................................................................................ 704 Configuring MFA delete ................................................................................................... 708 Working with versioning-enabled objects ........................................................................... 709 Working with versioning-suspended objects ....................................................................... 728 Using Amazon Backup for Amazon S3 ....................................................................................... 730 Working with archived objects ................................................................................................. 731 Archive retrieval options .................................................................................................. 732 Restoring an archived object ............................................................................................ 734 Using Object Lock .................................................................................................................. 738 S3 Object Lock ............................................................................................................... 739 Configuring Object Lock on the console ............................................................................ 743 Managing Object Lock .................................................................................................... 744 Managing storage classes........................................................................................................ 747 Frequently accessed objects ............................................................................................. 747 Automatically optimizing data with changing or unknown access patterns ............................. 747 Infrequently accessed objects ........................................................................................... 748 Archiving objects............................................................................................................ 749 Amazon S3 on Outposts .................................................................................................. 750 API Version 2006-03-01 viii Amazon Simple Storage Service User Guide Comparing storage classes............................................................................................... 751 Setting the storage class of an object ............................................................................... 751 Amazon S3 Intelligent-Tiering .................................................................................................. 752 How S3 Intelligent-Tiering works ..................................................................................... 753 Using S3 Intelligent-Tiering ............................................................................................ 754 Managing S3 Intelligent-Tiering ...................................................................................... 757 Managing lifecycle.................................................................................................................. 761 Managing object lifecycle ................................................................................................ 761 Creating a lifecycle configuration ...................................................................................... 761 Transitioning objects....................................................................................................... 762 Expiring objects.............................................................................................................. 767 Setting lifecycle configuration .......................................................................................... 767 Using other bucket configurations .................................................................................... 778 Configuring Lifecycle event notifications ........................................................................... 779 Lifecycle configuration elements ...................................................................................... 781 Examples of S3 Lifecycle configuration .............................................................................. 788 Managing inventory................................................................................................................ 799 Amazon S3 Inventory buckets .......................................................................................... 799 Inventory lists................................................................................................................ 800 Configuring Amazon S3 Inventory .................................................................................... 801 Setting up notifications for inventory completion ............................................................... 805 Locating your inventory .................................................................................................. 806 Querying inventory with Athena ....................................................................................... 809 Converting empty version ID strings to null strings ............................................................. 811 Replicating objects.................................................................................................................. 813 Why use replication ........................................................................................................ 814 When to use Cross-Region Replication .............................................................................. 814 When to use Same-Region Replication .............................................................................. 815 When to use two-way replication (bi-directional replication) ................................................. 815 When to use S3 Batch Replication .................................................................................... 815 Requirements for replication ............................................................................................ 816 What's replicated?........................................................................................................... 816 Setting up replication ..................................................................................................... 818 Replicate existing objects ................................................................................................ 859 Additional configurations................................................................................................. 867 Getting replication status ................................................................................................ 884 Troubleshooting............................................................................................................. 886 Additional considerations................................................................................................. 888 Using object tags ................................................................................................................... 889 API operations related to object tagging ........................................................................... 891 Additional configurations................................................................................................. 892 Access control................................................................................................................ 892 Managing object tags ...................................................................................................... 894 Using cost allocation tags ........................................................................................................ 898 More Info ...................................................................................................................... 899 Billing and usage reporting ...................................................................................................... 899 Billing reports................................................................................................................ 900 Usage report.................................................................................................................. 901 Understanding billing and usage reports ........................................................................... 903 Using Amazon S3 Select .......................................................................................................... 916 Requirements and limits .................................................................................................. 916 Constructing a request .................................................................................................... 917 Errors............................................................................................................................ 917 S3 Select examples ......................................................................................................... 918 SQL Reference............................................................................................................... 920 Using Batch Operations........................................................................................................... 945 Batch Operations basics.................................................................................................. 945 API Version 2006-03-01 ix Amazon Simple Storage Service User Guide S3 Batch Operations tutorial............................................................................................ 946 Granting permissions...................................................................................................... 946 Creating a job ................................................................................................................ 953 Supported operations..................................................................................................... 960 Managing jobs................................................................................................................ 984 Tracking job status and completion reports ....................................................................... 987 Using tags..................................................................................................................... 997 Managing S3 Object Lock .............................................................................................. 1007 S3 Batch Operations tutorial.......................................................................................... 1023 Monitoring Amazon S3.................................................................................................................. 1024 Monitoring tools................................................................................................................... 1024 Automated tools........................................................................................................... 1024 Manual tools................................................................................................................ 1025 Logging options................................................................................................................... 1025 Logging with CloudTrail ......................................................................................................... 1027 Using CloudTrail logs with Amazon S3 server access logs and CloudWatch Logs ..................... 1027 CloudTrail tracking with Amazon S3 SOAP API calls .......................................................... 1028 CloudTrail events.......................................................................................................... 1028 Example log files.......................................................................................................... 1032 Enabling CloudTrail ....................................................................................................... 1036 Identifying S3 requests .................................................................................................. 1038 Logging server access ............................................................................................................ 1044 How do I enable log delivery? ........................................................................................ 1044 Log object key format ................................................................................................... 1045 How are logs delivered?................................................................................................ 1045 Best effort server log delivery ........................................................................................ 1045 Bucket logging status changes take effect over time ......................................................... 1046 Enabling server access logging ....................................................................................... 1046 Log format.................................................................................................................. 1055 Deleting log files.......................................................................................................... 1065 Identifying S3 requests .................................................................................................. 1065 Monitoring metrics with CloudWatch ....................................................................................... 1069 Metrics and dimensions................................................................................................. 1070 Accessing CloudWatch metrics........................................................................................ 1080 CloudWatch metrics configurations ................................................................................. 1081 Amazon S3 Event Notifications ............................................................................................... 1087 Overview..................................................................................................................... 1087 Notification types and destinations................................................................................. 1088 Using SQS, SNS, and Lambda ......................................................................................... 1092 Using EventBridge........................................................................................................ 1111 Using analytics and insights ........................................................................................................... 1118 Storage Class Analysis........................................................................................................... 1118 How to set up storage class analysis ............................................................................... 1118 Storage class analysis.................................................................................................... 1119 How can I export storage class analysis data? ................................................................... 1120 Configuring storage class analysis................................................................................... 1121 S3 Storage Lens................................................................................................................... 1123 S3 Storage Lens metrics and features .............................................................................. 1124 Understanding S3 Storage Lens...................................................................................... 1126 Working with Organizations........................................................................................... 1132 S3 Storage Lens permissions.......................................................................................... 1134 Viewing storage metrics................................................................................................ 1136 Amazon S3 Storage Lens metrics use cases ...................................................................... 1155 Metrics glossary............................................................................................................ 1171 Working with S3 Storage Lens ....................................................................................... 1181 Tracing requests using X-Ray .................................................................................................. 1208 How X-Ray works with Amazon S3 .................................................................................. 1208 API Version 2006-03-01 x

Description:
Table of Contents. Welcome to the Amazon S3 Console User Guide . 98. Grant Amazon S3 Permission to Encrypt Using Your AWS KMS Key .
ebook img

Amazon Simple Storage Service - Console User Guide PDF

124 Pages·2017·3.29 MB·English
by  
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Amazon Simple Storage Service - Console User Guide

Amazon Simple Storage Service User Guide API Version 2006-03-01 Amazon Simple Storage Service User Guide Amazon Simple Storage Service: User Guide Amazon Simple Storage Service User Guide Table of Contents What is Amazon S3? ........................................................................................................................... 1 Features of Amazon S3 ............................................................................................................... 1 Storage classes.................................................................................................................. 1 Storage management......................................................................................................... 1 Access management........................................................................................................... 2 Data processing................................................................................................................. 2 Storage logging and monitoring.......................................................................................... 2 Analytics and insights......................................................................................................... 3 Strong consistency............................................................................................................. 3 How Amazon S3 works ............................................................................................................... 3 Buckets............................................................................................................................. 4 Objects............................................................................................................................. 4 Keys................................................................................................................................. 5 S3 Versioning.................................................................................................................... 5 Version ID ......................................................................................................................... 5 Bucket policy..................................................................................................................... 5 S3 Access Points................................................................................................................ 5 Access control lists (ACLs)................................................................................................... 6 Regions............................................................................................................................. 6 Amazon S3 data consistency model .............................................................................................. 6 Concurrent applications...................................................................................................... 7 Related services......................................................................................................................... 8 Accessing Amazon S3 ................................................................................................................. 9 Amazon Web Services Management Console ......................................................................... 9 Amazon Command Line Interface......................................................................................... 9 Amazon SDKs.................................................................................................................... 9 Amazon S3 REST API.......................................................................................................... 9 Paying for Amazon S3 .............................................................................................................. 10 PCI DSS compliance.................................................................................................................. 10 Getting started ................................................................................................................................ 11 Setting up............................................................................................................................... 11 Sign up for an Amazon Web Services account...................................................................... 12 Secure IAM users .............................................................................................................. 12 Step 1: Create a bucket ............................................................................................................. 12 Step 2: Upload an object .......................................................................................................... 14 Step 3: Download an object...................................................................................................... 15 Using the S3 console ........................................................................................................ 15 Step 4: Copy an object ............................................................................................................. 15 Step 5: Delete the objects and bucket ........................................................................................ 16 Deleting an object ............................................................................................................ 16 Emptying your bucket ....................................................................................................... 17 Deleting your bucket ........................................................................................................ 17 Next steps............................................................................................................................... 17 Understand common use cases .......................................................................................... 18 Control access to your buckets and objects .......................................................................... 18 Manage and monitor your storage ...................................................................................... 19 Develop with Amazon S3 .................................................................................................. 19 Learn from tutorials ......................................................................................................... 20 Explore training and support ............................................................................................. 21 Access control.......................................................................................................................... 22 Creating a new bucket ...................................................................................................... 22 Storing and sharing data................................................................................................... 23 Sharing resources............................................................................................................. 24 Protecting data................................................................................................................ 24 API Version 2006-03-01 iii Amazon Simple Storage Service User Guide Tutorials.......................................................................................................................................... 27 Getting started........................................................................................................................ 20 Optimizing storage costs........................................................................................................... 20 Hosting videos and websites...................................................................................................... 20 Data processing ........................................................................................................................ 20 Protecting data........................................................................................................................ 20 Transforming data with S3 Object Lambda.................................................................................. 28 Prerequisites.................................................................................................................... 29 Step 1: Create an S3 bucket .............................................................................................. 30 Step 2: Upload a file to the S3 bucket ................................................................................ 31 Step 3: Create an S3 access point ....................................................................................... 31 Step 4: Create a Lambda function ...................................................................................... 32 Step 5: Configure an IAM policy for your Lambda function's execution role ............................... 36 Step 6: Create an S3 Object Lambda access point ................................................................. 36 Step 7: View the transformed data..................................................................................... 37 Step 8: Clean up.............................................................................................................. 39 Next steps....................................................................................................................... 41 Detecting and redacting PII data ................................................................................................ 42 Prerequisites: Create an IAM user with permissions ............................................................... 43 Step 1: Create an S3 bucket .............................................................................................. 44 Step 2: Upload a file to the S3 bucket ................................................................................ 45 Step 3: Create an S3 access point ....................................................................................... 45 Step 4: Configure and deploy a prebuilt Lambda function ...................................................... 46 Step 5: Create an S3 Object Lambda access point ................................................................. 47 Step 6: Use the S3 Object Lambda access point to retrieve the redacted file ............................. 48 Step 7: Clean up.............................................................................................................. 48 Next steps....................................................................................................................... 51 Hosting video streaming........................................................................................................... 52 Prerequisites: Register and configure a custom domain with Route 53 ..................................... 52 Step 1: Create an S3 bucket .............................................................................................. 53 Step 2: Upload a video to the S3 bucket ............................................................................. 54 Step 3: Create a CloudFront origin access identity................................................................ 54 Step 4: Create a CloudFront distribution .............................................................................. 54 Step 5: Access the video through the CloudFront distribution ................................................. 56 Step 6: Configure your CloudFront distribution to use your custom domain name ...................... 57 Step 7: Access the S3 video through the CloudFront distribution with the custom domain name .. 60 (Optional) Step 8: View data about requests received by your CloudFront distribution ................ 60 Step 9: Clean up.............................................................................................................. 61 Next steps....................................................................................................................... 64 Batch-transcoding videos.......................................................................................................... 64 Prerequisites.................................................................................................................... 65 Step 1: Create an S3 bucket for the output media files ......................................................... 65 Step 2: Create an IAM role for MediaConvert ....................................................................... 67 Step 3: Create an IAM role for your Lambda function ............................................................ 67 Step 4: Create a Lambda function for video transcoding ........................................................ 69 Step 5: Configure Amazon S3 Inventory for your S3 source bucket .......................................... 81 Step 6: Create an IAM role for S3 Batch Operations .............................................................. 83 Step 7: Create and run an S3 Batch Operations job .............................................................. 86 Step 8: Check the output media files from your S3 destination bucket ..................................... 89 Step 9: Clean up.............................................................................................................. 89 Next steps....................................................................................................................... 91 Configuring a static website...................................................................................................... 92 Step 1: Create a bucket ..................................................................................................... 92 Step 2: Enable static website hosting .................................................................................. 92 Step 3: Edit Block Public Access settings ............................................................................. 93 Step 4: Add a bucket policy that makes your bucket content publicly available .......................... 94 Step 5: Configure an index document ................................................................................. 95 API Version 2006-03-01 iv Amazon Simple Storage Service User Guide Step 6: Configure an error document .................................................................................. 96 Step 7: Test your website endpoint .................................................................................... 97 Step 8: Clean up.............................................................................................................. 97 Configuring a static website using a custom domain ..................................................................... 97 Before you begin .............................................................................................................. 98 Step 1: Register a custom domain with Route 53 .................................................................. 98 Step 2: Create two buckets ................................................................................................ 98 Step 3: Configure root Domain bucket ................................................................................ 99 Step 4: Configure subdomain bucket for redirect ................................................................ 100 Step 5: Configure logging ................................................................................................ 101 Step 6: Upload index and website content ......................................................................... 101 Step 7: Upload an error document .................................................................................... 102 Step 8: Edit Block Public Access ....................................................................................... 103 Step 9: Attach a bucket policy ......................................................................................... 104 Step 10: Test your domain endpoint ................................................................................. 105 Step 11: Add alias records ............................................................................................... 106 Step 12: Test the website ................................................................................................ 109 Speeding up your website with Amazon CloudFront ............................................................ 109 Cleaning up example resources ........................................................................................ 112 Working with buckets ..................................................................................................................... 114 Buckets overview.................................................................................................................... 114 About permissions.......................................................................................................... 115 Managing public access to buckets ................................................................................... 115 Bucket configuration....................................................................................................... 116 Naming rules......................................................................................................................... 118 Example bucket names .................................................................................................... 119 Creating a bucket ................................................................................................................... 119 Default settings for new S3 buckets FAQ ................................................................................... 125 Viewing bucket properties ....................................................................................................... 126 Methods for accessing a bucket ................................................................................................ 127 Virtual-hosted–style access.............................................................................................. 127 Path-style access............................................................................................................ 127 Accessing an S3 bucket over IPv6 ..................................................................................... 128 Accessing a bucket through S3 access points ...................................................................... 128 Accessing a bucket using S3:// ......................................................................................... 128 Emptying a bucket ................................................................................................................. 129 Deleting a bucket ................................................................................................................... 130 Setting default bucket encryption ............................................................................................ 133 Using SSE-KMS encryption for cross-account operations ...................................................... 134 Using default encryption with replication .......................................................................... 134 Using Amazon S3 Bucket Keys with default encryption ........................................................ 135 Enabling default encryption ............................................................................................. 135 Monitoring default encryption .......................................................................................... 137 Configuring Transfer Acceleration ............................................................................................. 138 Why use Transfer Acceleration? ........................................................................................ 138 Requirements for using Transfer Acceleration ..................................................................... 138 Getting Started .............................................................................................................. 139 Enabling Transfer Acceleration ......................................................................................... 140 Speed Comparison tool................................................................................................... 145 Using Requester Pays .............................................................................................................. 145 How Requester Pays charges work .................................................................................... 146 Configuring Requester Pays ............................................................................................. 146 Retrieving the requestPayment configuration ..................................................................... 148 Downloading objects in Requester Pays buckets ................................................................. 148 Restrictions and limitations ...................................................................................................... 149 Working with objects ...................................................................................................................... 151 Objects.................................................................................................................................. 151 API Version 2006-03-01 v Amazon Simple Storage Service User Guide Subresources.................................................................................................................. 152 Creating object keys ............................................................................................................... 152 Object key naming guidelines .......................................................................................... 153 Working with metadata ........................................................................................................... 155 System-defined object metadata ...................................................................................... 156 User-defined object metadata .......................................................................................... 157 Editing object metadata .................................................................................................. 159 Uploading objects................................................................................................................... 160 Using multipart upload ........................................................................................................... 169 Multipart upload process ................................................................................................. 169 Checksums with multipart upload operations ..................................................................... 170 Concurrent multipart upload operations ............................................................................ 171 Multipart upload and pricing ........................................................................................... 171 API support for multipart upload ..................................................................................... 171 Amazon Command Line Interface support for multipart upload ............................................ 172 Amazon SDK support for multipart upload ........................................................................ 172 Multipart upload API and permissions ............................................................................... 172 Configuring a lifecycle policy ........................................................................................... 174 Uploading an object using multipart upload ...................................................................... 176 Uploading a directory ..................................................................................................... 189 Listing multipart uploads ................................................................................................. 191 Tracking a multipart upload ............................................................................................. 193 Aborting a multipart upload ............................................................................................ 195 Copying an object .......................................................................................................... 199 Multipart upload limits .................................................................................................... 204 Copying objects...................................................................................................................... 204 To copy an object ........................................................................................................... 205 Downloading an object ........................................................................................................... 211 Checking object integrity ......................................................................................................... 218 Using supported checksum algorithms .............................................................................. 218 Using Content-MD5 when uploading objects ...................................................................... 224 Using Content-MD5 and the ETag to verify uploaded objects ............................................... 224 Using trailing checksums................................................................................................. 225 Using part-level checksums for multipart uploads ............................................................... 225 Deleting objects..................................................................................................................... 226 Programmatically deleting objects from a version-enabled bucket ........................................ 227 Deleting objects from an MFA-enabled bucket .................................................................... 227 Deleting a single object ................................................................................................... 227 Deleting multiple objects ................................................................................................. 234 Organizing and listing objects .................................................................................................. 246 Using prefixes................................................................................................................ 247 Listing objects................................................................................................................ 248 Using folders................................................................................................................. 258 Viewing an object overview ............................................................................................. 261 Viewing object properties ................................................................................................ 261 Using presigned URLs ............................................................................................................. 262 Limiting presigned URL capabilities ................................................................................... 262 Who can create a presigned URL ...................................................................................... 263 When does Amazon S3 check the expiration date and time of a presigned URL? ...................... 264 Sharing objects.............................................................................................................. 264 Uploading objects........................................................................................................... 268 Deleting an object .......................................................................................................... 282 Transforming objects.............................................................................................................. 284 Creating Object Lambda access points .............................................................................. 286 Using Amazon S3 Object Lambda Access Points ................................................................. 295 Security considerations.................................................................................................... 296 Writing Lambda functions ............................................................................................... 300 API Version 2006-03-01 vi Amazon Simple Storage Service User Guide Using Amazon built functions .......................................................................................... 321 Best practices and guidelines for S3 Object Lambda ........................................................... 322 S3 Object Lambda tutorials ............................................................................................. 324 Debugging S3 Object Lambda .......................................................................................... 324 Working with access points .............................................................................................................. 325 Configuring IAM policies.......................................................................................................... 325 Access point policy examples ........................................................................................... 326 Condition keys............................................................................................................... 328 Delegating access control to access points ......................................................................... 329 Granting permissions for cross-account access points .......................................................... 330 Creating access points ............................................................................................................. 330 Rules for naming Amazon S3 access points ........................................................................ 330 Creating an access point .................................................................................................. 331 Creating access points restricted to a VPC ......................................................................... 332 Managing public access ................................................................................................... 334 Using access points ................................................................................................................. 335 Monitoring and logging................................................................................................... 335 Managing access points ................................................................................................... 337 Using a bucket-style alias for your access point .................................................................. 339 Using access points with Amazon S3 operations ................................................................. 340 Restrictions and limitations ...................................................................................................... 342 Working with Multi-Region Access Points ........................................................................................... 344 Creating Multi-Region Access Points .......................................................................................... 346 Rules for naming Amazon S3 Multi-Region Access Points ..................................................... 347 Rules for choosing buckets for Amazon S3 Multi-Region Access Points ................................... 348 Blocking public access with Amazon S3 Multi-Region Access Points ....................................... 348 Creating Amazon S3 Multi-Region Access Points ................................................................. 349 Configuring Amazon PrivateLink ....................................................................................... 350 Using a Multi-Region Access Point ............................................................................................ 352 Multi-Region Access Point hostnames ................................................................................ 353 Multi-Region Access Points and Amazon S3 Transfer Acceleration ......................................... 354 Multi-Region Access Point permissions .............................................................................. 354 Request routing.............................................................................................................. 358 Failover configuration..................................................................................................... 359 Bucket replication........................................................................................................... 364 Supported operations..................................................................................................... 367 Managing Multi-Region Access Points ........................................................................................ 373 Monitoring and logging........................................................................................................... 373 Monitoring and logging requests made to Multi-Region Access Point management APIs ........... 374 Using CloudTrail............................................................................................................. 375 Restrictions and limitations ...................................................................................................... 375 Security......................................................................................................................................... 378 Data protection...................................................................................................................... 379 Data encryption..................................................................................................................... 379 Server-side encryption.................................................................................................... 380 Using client-side encryption ............................................................................................. 424 Internetwork privacy............................................................................................................... 428 Traffic between service and on-premises clients and applications .......................................... 428 Traffic between Amazon resources in the same Region ........................................................ 429 Amazon PrivateLink for Amazon S3 .......................................................................................... 429 Types of VPC endpoints .................................................................................................. 429 Restrictions and limitations of Amazon PrivateLink for Amazon S3 ........................................ 430 Creating a VPC endpoint ................................................................................................. 430 Accessing Amazon S3 interface endpoints .......................................................................... 430 Accessing buckets and S3 access points from S3 interface endpoints ..................................... 431 Updating an on-premises DNS configuration ...................................................................... 434 Creating a VPC endpoint policy ........................................................................................ 435 API Version 2006-03-01 vii Amazon Simple Storage Service User Guide Identity and access management .............................................................................................. 438 Overview....................................................................................................................... 438 Access policy guidelines ................................................................................................... 444 Request authorization..................................................................................................... 448 Bucket policies and user policies ....................................................................................... 456 Amazon managed policies ............................................................................................... 611 Managing access with ACLs .............................................................................................. 613 Using CORS ................................................................................................................... 632 Blocking public access ..................................................................................................... 643 Reviewing bucket access .................................................................................................. 653 Verifying bucket ownership .............................................................................................. 658 Controlling object ownership ................................................................................................... 661 Object Ownership settings ............................................................................................... 663 Changes introduced by disabling ACLs ............................................................................... 664 Prerequisites for disabling ACLs ........................................................................................ 665 Object Ownership permissions ......................................................................................... 666 Disabling ACLs for all new buckets ................................................................................... 666 Replication and Object Ownership .................................................................................... 667 Setting Object Ownership ................................................................................................ 667 Prerequisites for disabling ACLs ........................................................................................ 668 Creating a bucket ........................................................................................................... 676 Setting Object Ownership ................................................................................................ 679 Viewing Object Ownership settings ................................................................................... 682 Disabling ACLs for all new buckets ................................................................................... 683 Troubleshooting............................................................................................................. 684 Logging and monitoring.......................................................................................................... 687 Compliance Validation............................................................................................................. 688 Resilience.............................................................................................................................. 689 Backup encryption.......................................................................................................... 691 Infrastructure security............................................................................................................. 692 Configuration and vulnerability analysis .................................................................................... 693 Security Best Practices ............................................................................................................ 694 Amazon S3 preventative security best Practices .................................................................. 694 Amazon S3 Monitoring and auditing best practices ............................................................. 697 Managing storage........................................................................................................................... 699 Using S3 Versioning ................................................................................................................ 699 Unversioned, versioning-enabled, and versioning-suspended buckets ..................................... 700 Using S3 Versioning with S3 Lifecycle ............................................................................... 700 S3 Versioning ................................................................................................................. 701 Enabling versioning on buckets ........................................................................................ 704 Configuring MFA delete ................................................................................................... 708 Working with versioning-enabled objects ........................................................................... 709 Working with versioning-suspended objects ....................................................................... 728 Using Amazon Backup for Amazon S3 ....................................................................................... 730 Working with archived objects ................................................................................................. 731 Archive retrieval options .................................................................................................. 732 Restoring an archived object ............................................................................................ 734 Using Object Lock .................................................................................................................. 738 S3 Object Lock ............................................................................................................... 739 Configuring Object Lock on the console ............................................................................ 743 Managing Object Lock .................................................................................................... 744 Managing storage classes........................................................................................................ 747 Frequently accessed objects ............................................................................................. 747 Automatically optimizing data with changing or unknown access patterns ............................. 747 Infrequently accessed objects ........................................................................................... 748 Archiving objects............................................................................................................ 749 Amazon S3 on Outposts .................................................................................................. 750 API Version 2006-03-01 viii Amazon Simple Storage Service User Guide Comparing storage classes............................................................................................... 751 Setting the storage class of an object ............................................................................... 751 Amazon S3 Intelligent-Tiering .................................................................................................. 752 How S3 Intelligent-Tiering works ..................................................................................... 753 Using S3 Intelligent-Tiering ............................................................................................ 754 Managing S3 Intelligent-Tiering ...................................................................................... 757 Managing lifecycle.................................................................................................................. 761 Managing object lifecycle ................................................................................................ 761 Creating a lifecycle configuration ...................................................................................... 761 Transitioning objects....................................................................................................... 762 Expiring objects.............................................................................................................. 767 Setting lifecycle configuration .......................................................................................... 767 Using other bucket configurations .................................................................................... 778 Configuring Lifecycle event notifications ........................................................................... 779 Lifecycle configuration elements ...................................................................................... 781 Examples of S3 Lifecycle configuration .............................................................................. 788 Managing inventory................................................................................................................ 799 Amazon S3 Inventory buckets .......................................................................................... 799 Inventory lists................................................................................................................ 800 Configuring Amazon S3 Inventory .................................................................................... 801 Setting up notifications for inventory completion ............................................................... 805 Locating your inventory .................................................................................................. 806 Querying inventory with Athena ....................................................................................... 809 Converting empty version ID strings to null strings ............................................................. 811 Replicating objects.................................................................................................................. 813 Why use replication ........................................................................................................ 814 When to use Cross-Region Replication .............................................................................. 814 When to use Same-Region Replication .............................................................................. 815 When to use two-way replication (bi-directional replication) ................................................. 815 When to use S3 Batch Replication .................................................................................... 815 Requirements for replication ............................................................................................ 816 What's replicated?........................................................................................................... 816 Setting up replication ..................................................................................................... 818 Replicate existing objects ................................................................................................ 859 Additional configurations................................................................................................. 867 Getting replication status ................................................................................................ 884 Troubleshooting............................................................................................................. 886 Additional considerations................................................................................................. 888 Using object tags ................................................................................................................... 889 API operations related to object tagging ........................................................................... 891 Additional configurations................................................................................................. 892 Access control................................................................................................................ 892 Managing object tags ...................................................................................................... 894 Using cost allocation tags ........................................................................................................ 898 More Info ...................................................................................................................... 899 Billing and usage reporting ...................................................................................................... 899 Billing reports................................................................................................................ 900 Usage report.................................................................................................................. 901 Understanding billing and usage reports ........................................................................... 903 Using Amazon S3 Select .......................................................................................................... 916 Requirements and limits .................................................................................................. 916 Constructing a request .................................................................................................... 917 Errors............................................................................................................................ 917 S3 Select examples ......................................................................................................... 918 SQL Reference............................................................................................................... 920 Using Batch Operations........................................................................................................... 945 Batch Operations basics.................................................................................................. 945 API Version 2006-03-01 ix Amazon Simple Storage Service User Guide S3 Batch Operations tutorial............................................................................................ 946 Granting permissions...................................................................................................... 946 Creating a job ................................................................................................................ 953 Supported operations..................................................................................................... 960 Managing jobs................................................................................................................ 984 Tracking job status and completion reports ....................................................................... 987 Using tags..................................................................................................................... 997 Managing S3 Object Lock .............................................................................................. 1007 S3 Batch Operations tutorial.......................................................................................... 1023 Monitoring Amazon S3.................................................................................................................. 1024 Monitoring tools................................................................................................................... 1024 Automated tools........................................................................................................... 1024 Manual tools................................................................................................................ 1025 Logging options................................................................................................................... 1025 Logging with CloudTrail ......................................................................................................... 1027 Using CloudTrail logs with Amazon S3 server access logs and CloudWatch Logs ..................... 1027 CloudTrail tracking with Amazon S3 SOAP API calls .......................................................... 1028 CloudTrail events.......................................................................................................... 1028 Example log files.......................................................................................................... 1032 Enabling CloudTrail ....................................................................................................... 1036 Identifying S3 requests .................................................................................................. 1038 Logging server access ............................................................................................................ 1044 How do I enable log delivery? ........................................................................................ 1044 Log object key format ................................................................................................... 1045 How are logs delivered?................................................................................................ 1045 Best effort server log delivery ........................................................................................ 1045 Bucket logging status changes take effect over time ......................................................... 1046 Enabling server access logging ....................................................................................... 1046 Log format.................................................................................................................. 1055 Deleting log files.......................................................................................................... 1065 Identifying S3 requests .................................................................................................. 1065 Monitoring metrics with CloudWatch ....................................................................................... 1069 Metrics and dimensions................................................................................................. 1070 Accessing CloudWatch metrics........................................................................................ 1080 CloudWatch metrics configurations ................................................................................. 1081 Amazon S3 Event Notifications ............................................................................................... 1087 Overview..................................................................................................................... 1087 Notification types and destinations................................................................................. 1088 Using SQS, SNS, and Lambda ......................................................................................... 1092 Using EventBridge........................................................................................................ 1111 Using analytics and insights ........................................................................................................... 1118 Storage Class Analysis........................................................................................................... 1118 How to set up storage class analysis ............................................................................... 1118 Storage class analysis.................................................................................................... 1119 How can I export storage class analysis data? ................................................................... 1120 Configuring storage class analysis................................................................................... 1121 S3 Storage Lens................................................................................................................... 1123 S3 Storage Lens metrics and features .............................................................................. 1124 Understanding S3 Storage Lens...................................................................................... 1126 Working with Organizations........................................................................................... 1132 S3 Storage Lens permissions.......................................................................................... 1134 Viewing storage metrics................................................................................................ 1136 Amazon S3 Storage Lens metrics use cases ...................................................................... 1155 Metrics glossary............................................................................................................ 1171 Working with S3 Storage Lens ....................................................................................... 1181 Tracing requests using X-Ray .................................................................................................. 1208 How X-Ray works with Amazon S3 .................................................................................. 1208 API Version 2006-03-01 x

Description:
Table of Contents. Welcome to the Amazon S3 Console User Guide . 98. Grant Amazon S3 Permission to Encrypt Using Your AWS KMS Key .
See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users