Table Of Content

Amazon Simple Storage Service User Guide API Version 2006-03-01 Amazon Simple Storage Service User Guide Amazon Simple Storage Service: User Guide Amazon Simple Storage Service User Guide Table of Contents What is Amazon S3? ........................................................................................................................... 1 Features of Amazon S3 ............................................................................................................... 1 Storage classes.................................................................................................................. 1 Storage management......................................................................................................... 1 Access management........................................................................................................... 2 Data processing................................................................................................................. 2 Storage logging and monitoring.......................................................................................... 2 Analytics and insights......................................................................................................... 3 Strong consistency............................................................................................................. 3 How Amazon S3 works ............................................................................................................... 3 Buckets............................................................................................................................. 4 Objects............................................................................................................................. 4 Keys................................................................................................................................. 5 S3 Versioning.................................................................................................................... 5 Version ID ......................................................................................................................... 5 Bucket policy..................................................................................................................... 5 S3 Access Points................................................................................................................ 5 Access control lists (ACLs)................................................................................................... 6 Regions............................................................................................................................. 6 Amazon S3 data consistency model .............................................................................................. 6 Concurrent applications...................................................................................................... 7 Related services......................................................................................................................... 8 Accessing Amazon S3 ................................................................................................................. 9 Amazon Web Services Management Console ......................................................................... 9 Amazon Command Line Interface......................................................................................... 9 Amazon SDKs.................................................................................................................... 9 Amazon S3 REST API.......................................................................................................... 9 Paying for Amazon S3 .............................................................................................................. 10 PCI DSS compliance.................................................................................................................. 10 Getting started ................................................................................................................................ 11 Setting up............................................................................................................................... 11 Sign up for an Amazon Web Services account...................................................................... 12 Secure IAM users .............................................................................................................. 12 Step 1: Create a bucket ............................................................................................................. 12 Step 2: Upload an object .......................................................................................................... 14 Step 3: Download an object...................................................................................................... 15 Using the S3 console ........................................................................................................ 15 Step 4: Copy an object ............................................................................................................. 15 Step 5: Delete the objects and bucket ........................................................................................ 16 Deleting an object ............................................................................................................ 16 Emptying your bucket ....................................................................................................... 17 Deleting your bucket ........................................................................................................ 17 Next steps............................................................................................................................... 17 Understand common use cases .......................................................................................... 18 Control access to your buckets and objects .......................................................................... 18 Manage and monitor your storage ...................................................................................... 19 Develop with Amazon S3 .................................................................................................. 19 Learn from tutorials ......................................................................................................... 20 Explore training and support ............................................................................................. 21 Access control.......................................................................................................................... 22 Creating a new bucket ...................................................................................................... 22 Storing and sharing data................................................................................................... 23 Sharing resources............................................................................................................. 24 Protecting data................................................................................................................ 24 API Version 2006-03-01 iii Amazon Simple Storage Service User Guide Tutorials.......................................................................................................................................... 27 Getting started........................................................................................................................ 20 Optimizing storage costs........................................................................................................... 20 Hosting videos and websites...................................................................................................... 20 Data processing ........................................................................................................................ 20 Protecting data........................................................................................................................ 20 Transforming data with S3 Object Lambda.................................................................................. 28 Prerequisites.................................................................................................................... 29 Step 1: Create an S3 bucket .............................................................................................. 30 Step 2: Upload a file to the S3 bucket ................................................................................ 31 Step 3: Create an S3 access point ....................................................................................... 31 Step 4: Create a Lambda function ...................................................................................... 32 Step 5: Configure an IAM policy for your Lambda function's execution role ............................... 36 Step 6: Create an S3 Object Lambda access point ................................................................. 36 Step 7: View the transformed data..................................................................................... 37 Step 8: Clean up.............................................................................................................. 39 Next steps....................................................................................................................... 41 Detecting and redacting PII data ................................................................................................ 42 Prerequisites: Create an IAM user with permissions ............................................................... 43 Step 1: Create an S3 bucket .............................................................................................. 44 Step 2: Upload a file to the S3 bucket ................................................................................ 45 Step 3: Create an S3 access point ....................................................................................... 45 Step 4: Configure and deploy a prebuilt Lambda function ...................................................... 46 Step 5: Create an S3 Object Lambda access point ................................................................. 47 Step 6: Use the S3 Object Lambda access point to retrieve the redacted file ............................. 48 Step 7: Clean up.............................................................................................................. 48 Next steps....................................................................................................................... 51 Hosting video streaming........................................................................................................... 52 Prerequisites: Register and configure a custom domain with Route 53 ..................................... 52 Step 1: Create an S3 bucket .............................................................................................. 53 Step 2: Upload a video to the S3 bucket ............................................................................. 54 Step 3: Create a CloudFront origin access identity................................................................ 54 Step 4: Create a CloudFront distribution .............................................................................. 54 Step 5: Access the video through the CloudFront distribution ................................................. 56 Step 6: Configure your CloudFront distribution to use your custom domain name ...................... 57 Step 7: Access the S3 video through the CloudFront distribution with the custom domain name .. 60 (Optional) Step 8: View data about requests received by your CloudFront distribution ................ 60 Step 9: Clean up.............................................................................................................. 61 Next steps....................................................................................................................... 64 Batch-transcoding videos.......................................................................................................... 64 Prerequisites.................................................................................................................... 65 Step 1: Create an S3 bucket for the output media files ......................................................... 65 Step 2: Create an IAM role for MediaConvert ....................................................................... 67 Step 3: Create an IAM role for your Lambda function ............................................................ 67 Step 4: Create a Lambda function for video transcoding ........................................................ 69 Step 5: Configure Amazon S3 Inventory for your S3 source bucket .......................................... 81 Step 6: Create an IAM role for S3 Batch Operations .............................................................. 83 Step 7: Create and run an S3 Batch Operations job .............................................................. 86 Step 8: Check the output media files from your S3 destination bucket ..................................... 89 Step 9: Clean up.............................................................................................................. 89 Next steps....................................................................................................................... 91 Configuring a static website...................................................................................................... 92 Step 1: Create a bucket ..................................................................................................... 92 Step 2: Enable static website hosting .................................................................................. 92 Step 3: Edit Block Public Access settings ............................................................................. 93 Step 4: Add a bucket policy that makes your bucket content publicly available .......................... 94 Step 5: Configure an index document ................................................................................. 95 API Version 2006-03-01 iv Amazon Simple Storage Service User Guide Step 6: Configure an error document .................................................................................. 96 Step 7: Test your website endpoint .................................................................................... 97 Step 8: Clean up.............................................................................................................. 97 Configuring a static website using a custom domain ..................................................................... 97 Before you begin .............................................................................................................. 98 Step 1: Register a custom domain with Route 53 .................................................................. 98 Step 2: Create two buckets ................................................................................................ 98 Step 3: Configure root Domain bucket ................................................................................ 99 Step 4: Configure subdomain bucket for redirect ................................................................ 100 Step 5: Configure logging ................................................................................................ 101 Step 6: Upload index and website content ......................................................................... 101 Step 7: Upload an error document .................................................................................... 102 Step 8: Edit Block Public Access ....................................................................................... 103 Step 9: Attach a bucket policy ......................................................................................... 104 Step 10: Test your domain endpoint ................................................................................. 105 Step 11: Add alias records ............................................................................................... 106 Step 12: Test the website ................................................................................................ 109 Speeding up your website with Amazon CloudFront ............................................................ 109 Cleaning up example resources ........................................................................................ 112 Working with buckets ..................................................................................................................... 114 Buckets overview.................................................................................................................... 114 About permissions.......................................................................................................... 115 Managing public access to buckets ................................................................................... 115 Bucket configuration....................................................................................................... 116 Naming rules......................................................................................................................... 118 Example bucket names .................................................................................................... 119 Creating a bucket ................................................................................................................... 119 Default settings for new S3 buckets FAQ ................................................................................... 125 Viewing bucket properties ....................................................................................................... 126 Methods for accessing a bucket ................................................................................................ 127 Virtual-hosted–style access.............................................................................................. 127 Path-style access............................................................................................................ 127 Accessing an S3 bucket over IPv6 ..................................................................................... 128 Accessing a bucket through S3 access points ...................................................................... 128 Accessing a bucket using S3:// ......................................................................................... 128 Emptying a bucket ................................................................................................................. 129 Deleting a bucket ................................................................................................................... 130 Setting default bucket encryption ............................................................................................ 133 Using SSE-KMS encryption for cross-account operations ...................................................... 134 Using default encryption with replication .......................................................................... 134 Using Amazon S3 Bucket Keys with default encryption ........................................................ 135 Enabling default encryption ............................................................................................. 135 Monitoring default encryption .......................................................................................... 137 Configuring Transfer Acceleration ............................................................................................. 138 Why use Transfer Acceleration? ........................................................................................ 138 Requirements for using Transfer Acceleration ..................................................................... 138 Getting Started .............................................................................................................. 139 Enabling Transfer Acceleration ......................................................................................... 140 Speed Comparison tool................................................................................................... 145 Using Requester Pays .............................................................................................................. 145 How Requester Pays charges work .................................................................................... 146 Configuring Requester Pays ............................................................................................. 146 Retrieving the requestPayment configuration ..................................................................... 148 Downloading objects in Requester Pays buckets ................................................................. 148 Restrictions and limitations ...................................................................................................... 149 Working with objects ...................................................................................................................... 151 Objects.................................................................................................................................. 151 API Version 2006-03-01 v Amazon Simple Storage Service User Guide Subresources.................................................................................................................. 152 Creating object keys ............................................................................................................... 152 Object key naming guidelines .......................................................................................... 153 Working with metadata ........................................................................................................... 155 System-defined object metadata ...................................................................................... 156 User-defined object metadata .......................................................................................... 157 Editing object metadata .................................................................................................. 159 Uploading objects................................................................................................................... 160 Using multipart upload ........................................................................................................... 169 Multipart upload process ................................................................................................. 169 Checksums with multipart upload operations ..................................................................... 170 Concurrent multipart upload operations ............................................................................ 171 Multipart upload and pricing ........................................................................................... 171 API support for multipart upload ..................................................................................... 171 Amazon Command Line Interface support for multipart upload ............................................ 172 Amazon SDK support for multipart upload ........................................................................ 172 Multipart upload API and permissions ............................................................................... 172 Configuring a lifecycle policy ........................................................................................... 174 Uploading an object using multipart upload ...................................................................... 176 Uploading a directory ..................................................................................................... 189 Listing multipart uploads ................................................................................................. 191 Tracking a multipart upload ............................................................................................. 193 Aborting a multipart upload ............................................................................................ 195 Copying an object .......................................................................................................... 199 Multipart upload limits .................................................................................................... 204 Copying objects...................................................................................................................... 204 To copy an object ........................................................................................................... 205 Downloading an object ........................................................................................................... 211 Checking object integrity ......................................................................................................... 218 Using supported checksum algorithms .............................................................................. 218 Using Content-MD5 when uploading objects ...................................................................... 224 Using Content-MD5 and the ETag to verify uploaded objects ............................................... 224 Using trailing checksums................................................................................................. 225 Using part-level checksums for multipart uploads ............................................................... 225 Deleting objects..................................................................................................................... 226 Programmatically deleting objects from a version-enabled bucket ........................................ 227 Deleting objects from an MFA-enabled bucket .................................................................... 227 Deleting a single object ................................................................................................... 227 Deleting multiple objects ................................................................................................. 234 Organizing and listing objects .................................................................................................. 246 Using prefixes................................................................................................................ 247 Listing objects................................................................................................................ 248 Using folders................................................................................................................. 258 Viewing an object overview ............................................................................................. 261 Viewing object properties ................................................................................................ 261 Using presigned URLs ............................................................................................................. 262 Limiting presigned URL capabilities ................................................................................... 262 Who can create a presigned URL ...................................................................................... 263 When does Amazon S3 check the expiration date and time of a presigned URL? ...................... 264 Sharing objects.............................................................................................................. 264 Uploading objects........................................................................................................... 268 Deleting an object .......................................................................................................... 282 Transforming objects.............................................................................................................. 284 Creating Object Lambda access points .............................................................................. 286 Using Amazon S3 Object Lambda Access Points ................................................................. 295 Security considerations.................................................................................................... 296 Writing Lambda functions ............................................................................................... 300 API Version 2006-03-01 vi Amazon Simple Storage Service User Guide Using Amazon built functions .......................................................................................... 321 Best practices and guidelines for S3 Object Lambda ........................................................... 322 S3 Object Lambda tutorials ............................................................................................. 324 Debugging S3 Object Lambda .......................................................................................... 324 Working with access points .............................................................................................................. 325 Configuring IAM policies.......................................................................................................... 325 Access point policy examples ........................................................................................... 326 Condition keys............................................................................................................... 328 Delegating access control to access points ......................................................................... 329 Granting permissions for cross-account access points .......................................................... 330 Creating access points ............................................................................................................. 330 Rules for naming Amazon S3 access points ........................................................................ 330 Creating an access point .................................................................................................. 331 Creating access points restricted to a VPC ......................................................................... 332 Managing public access ................................................................................................... 334 Using access points ................................................................................................................. 335 Monitoring and logging................................................................................................... 335 Managing access points ................................................................................................... 337 Using a bucket-style alias for your access point .................................................................. 339 Using access points with Amazon S3 operations ................................................................. 340 Restrictions and limitations ...................................................................................................... 342 Working with Multi-Region Access Points ........................................................................................... 344 Creating Multi-Region Access Points .......................................................................................... 346 Rules for naming Amazon S3 Multi-Region Access Points ..................................................... 347 Rules for choosing buckets for Amazon S3 Multi-Region Access Points ................................... 348 Blocking public access with Amazon S3 Multi-Region Access Points ....................................... 348 Creating Amazon S3 Multi-Region Access Points ................................................................. 349 Configuring Amazon PrivateLink ....................................................................................... 350 Using a Multi-Region Access Point ............................................................................................ 352 Multi-Region Access Point hostnames ................................................................................ 353 Multi-Region Access Points and Amazon S3 Transfer Acceleration ......................................... 354 Multi-Region Access Point permissions .............................................................................. 354 Request routing.............................................................................................................. 358 Failover configuration..................................................................................................... 359 Bucket replication........................................................................................................... 364 Supported operations..................................................................................................... 367 Managing Multi-Region Access Points ........................................................................................ 373 Monitoring and logging........................................................................................................... 373 Monitoring and logging requests made to Multi-Region Access Point management APIs ........... 374 Using CloudTrail............................................................................................................. 375 Restrictions and limitations ...................................................................................................... 375 Security......................................................................................................................................... 378 Data protection...................................................................................................................... 379 Data encryption..................................................................................................................... 379 Server-side encryption.................................................................................................... 380 Using client-side encryption ............................................................................................. 424 Internetwork privacy............................................................................................................... 428 Traffic between service and on-premises clients and applications .......................................... 428 Traffic between Amazon resources in the same Region ........................................................ 429 Amazon PrivateLink for Amazon S3 .......................................................................................... 429 Types of VPC endpoints .................................................................................................. 429 Restrictions and limitations of Amazon PrivateLink for Amazon S3 ........................................ 430 Creating a VPC endpoint ................................................................................................. 430 Accessing Amazon S3 interface endpoints .......................................................................... 430 Accessing buckets and S3 access points from S3 interface endpoints ..................................... 431 Updating an on-premises DNS configuration ...................................................................... 434 Creating a VPC endpoint policy ........................................................................................ 435 API Version 2006-03-01 vii Amazon Simple Storage Service User Guide Identity and access management .............................................................................................. 438 Overview....................................................................................................................... 438 Access policy guidelines ................................................................................................... 444 Request authorization..................................................................................................... 448 Bucket policies and user policies ....................................................................................... 456 Amazon managed policies ............................................................................................... 611 Managing access with ACLs .............................................................................................. 613 Using CORS ................................................................................................................... 632 Blocking public access ..................................................................................................... 643 Reviewing bucket access .................................................................................................. 653 Verifying bucket ownership .............................................................................................. 658 Controlling object ownership ................................................................................................... 661 Object Ownership settings ............................................................................................... 663 Changes introduced by disabling ACLs ............................................................................... 664 Prerequisites for disabling ACLs ........................................................................................ 665 Object Ownership permissions ......................................................................................... 666 Disabling ACLs for all new buckets ................................................................................... 666 Replication and Object Ownership .................................................................................... 667 Setting Object Ownership ................................................................................................ 667 Prerequisites for disabling ACLs ........................................................................................ 668 Creating a bucket ........................................................................................................... 676 Setting Object Ownership ................................................................................................ 679 Viewing Object Ownership settings ................................................................................... 681 Disabling ACLs for all new buckets ................................................................................... 682 Troubleshooting............................................................................................................. 684 Logging and monitoring.......................................................................................................... 686 Compliance Validation............................................................................................................. 687 Resilience.............................................................................................................................. 688 Backup encryption.......................................................................................................... 690 Infrastructure security............................................................................................................. 691 Configuration and vulnerability analysis .................................................................................... 692 Security Best Practices ............................................................................................................ 693 Amazon S3 preventative security best Practices .................................................................. 693 Amazon S3 Monitoring and auditing best practices ............................................................. 696 Managing storage........................................................................................................................... 698 Using S3 Versioning ................................................................................................................ 698 Unversioned, versioning-enabled, and versioning-suspended buckets ..................................... 699 Using S3 Versioning with S3 Lifecycle ............................................................................... 699 S3 Versioning ................................................................................................................. 700 Enabling versioning on buckets ........................................................................................ 703 Configuring MFA delete ................................................................................................... 707 Working with versioning-enabled objects ........................................................................... 708 Working with versioning-suspended objects ....................................................................... 727 Using Amazon Backup for Amazon S3 ....................................................................................... 729 Working with archived objects ................................................................................................. 730 Archive retrieval options .................................................................................................. 731 Restoring an archived object ............................................................................................ 733 Using Object Lock .................................................................................................................. 737 S3 Object Lock ............................................................................................................... 738 Configuring Object Lock on the console ............................................................................ 742 Managing Object Lock .................................................................................................... 743 Managing storage classes........................................................................................................ 746 Frequently accessed objects ............................................................................................. 746 Automatically optimizing data with changing or unknown access patterns ............................. 746 Infrequently accessed objects ........................................................................................... 747 Archiving objects............................................................................................................ 748 Amazon S3 on Outposts .................................................................................................. 749 API Version 2006-03-01 viii Amazon Simple Storage Service User Guide Comparing storage classes............................................................................................... 750 Setting the storage class of an object ............................................................................... 750 Amazon S3 Intelligent-Tiering .................................................................................................. 751 How S3 Intelligent-Tiering works ..................................................................................... 752 Using S3 Intelligent-Tiering ............................................................................................ 753 Managing S3 Intelligent-Tiering ...................................................................................... 756 Managing lifecycle.................................................................................................................. 760 Managing object lifecycle ................................................................................................ 760 Creating a lifecycle configuration ...................................................................................... 760 Transitioning objects....................................................................................................... 761 Expiring objects.............................................................................................................. 766 Setting lifecycle configuration .......................................................................................... 766 Using other bucket configurations .................................................................................... 777 Configuring Lifecycle event notifications ........................................................................... 778 Lifecycle configuration elements ...................................................................................... 780 Examples of S3 Lifecycle configuration .............................................................................. 787 Managing inventory................................................................................................................ 798 Amazon S3 Inventory buckets .......................................................................................... 798 Inventory lists................................................................................................................ 799 Configuring Amazon S3 Inventory .................................................................................... 800 Setting up notifications for inventory completion ............................................................... 804 Locating your inventory .................................................................................................. 805 Querying inventory with Athena ....................................................................................... 808 Converting empty version ID strings to null strings ............................................................. 810 Replicating objects.................................................................................................................. 812 Why use replication ........................................................................................................ 813 When to use Cross-Region Replication .............................................................................. 813 When to use Same-Region Replication .............................................................................. 814 When to use two-way replication (bi-directional replication) ................................................. 814 When to use S3 Batch Replication .................................................................................... 814 Requirements for replication ............................................................................................ 815 What's replicated?........................................................................................................... 815 Setting up replication ..................................................................................................... 817 Replicate existing objects ................................................................................................ 858 Additional configurations................................................................................................. 866 Getting replication status ................................................................................................ 883 Troubleshooting............................................................................................................. 885 Additional considerations................................................................................................. 887 Using object tags ................................................................................................................... 888 API operations related to object tagging ........................................................................... 890 Additional configurations................................................................................................. 891 Access control................................................................................................................ 891 Managing object tags ...................................................................................................... 893 Using cost allocation tags ........................................................................................................ 897 More Info ...................................................................................................................... 898 Billing and usage reporting ...................................................................................................... 898 Billing reports................................................................................................................ 899 Usage report.................................................................................................................. 900 Understanding billing and usage reports ........................................................................... 902 Using Amazon S3 Select .......................................................................................................... 915 Requirements and limits .................................................................................................. 915 Constructing a request .................................................................................................... 916 Errors............................................................................................................................ 916 S3 Select examples ......................................................................................................... 917 SQL Reference............................................................................................................... 919 Using Batch Operations........................................................................................................... 944 Batch Operations basics.................................................................................................. 944 API Version 2006-03-01 ix Amazon Simple Storage Service User Guide S3 Batch Operations tutorial............................................................................................ 945 Granting permissions...................................................................................................... 945 Creating a job ................................................................................................................ 952 Supported operations..................................................................................................... 959 Managing jobs................................................................................................................ 983 Tracking job status and completion reports ....................................................................... 986 Using tags..................................................................................................................... 996 Managing S3 Object Lock .............................................................................................. 1006 S3 Batch Operations tutorial.......................................................................................... 1022 Monitoring Amazon S3.................................................................................................................. 1023 Monitoring tools................................................................................................................... 1023 Automated tools........................................................................................................... 1023 Manual tools................................................................................................................ 1024 Logging options................................................................................................................... 1024 Logging with CloudTrail ......................................................................................................... 1026 Using CloudTrail logs with Amazon S3 server access logs and CloudWatch Logs ..................... 1026 CloudTrail tracking with Amazon S3 SOAP API calls .......................................................... 1027 CloudTrail events.......................................................................................................... 1027 Example log files.......................................................................................................... 1031 Enabling CloudTrail ....................................................................................................... 1035 Identifying S3 requests .................................................................................................. 1037 Logging server access ............................................................................................................ 1043 How do I enable log delivery? ........................................................................................ 1043 Log object key format ................................................................................................... 1044 How are logs delivered?................................................................................................ 1044 Best effort server log delivery ........................................................................................ 1044 Bucket logging status changes take effect over time ......................................................... 1045 Enabling server access logging ....................................................................................... 1045 Log format.................................................................................................................. 1054 Deleting log files.......................................................................................................... 1064 Identifying S3 requests .................................................................................................. 1064 Monitoring metrics with CloudWatch ....................................................................................... 1068 Metrics and dimensions................................................................................................. 1069 Accessing CloudWatch metrics........................................................................................ 1079 CloudWatch metrics configurations ................................................................................. 1080 Amazon S3 Event Notifications ............................................................................................... 1086 Overview..................................................................................................................... 1086 Notification types and destinations................................................................................. 1087 Using SQS, SNS, and Lambda ......................................................................................... 1091 Using EventBridge........................................................................................................ 1110 Using analytics and insights ........................................................................................................... 1117 Storage Class Analysis........................................................................................................... 1117 How to set up storage class analysis ............................................................................... 1117 Storage class analysis.................................................................................................... 1118 How can I export storage class analysis data? ................................................................... 1119 Configuring storage class analysis................................................................................... 1120 S3 Storage Lens................................................................................................................... 1122 S3 Storage Lens metrics and features .............................................................................. 1123 Understanding S3 Storage Lens...................................................................................... 1125 Working with Organizations........................................................................................... 1131 S3 Storage Lens permissions.......................................................................................... 1133 Viewing storage metrics................................................................................................ 1135 Amazon S3 Storage Lens metrics use cases ...................................................................... 1154 Metrics glossary............................................................................................................ 1170 Working with S3 Storage Lens ....................................................................................... 1180 Tracing requests using X-Ray .................................................................................................. 1207 How X-Ray works with Amazon S3 .................................................................................. 1207 API Version 2006-03-01 x

Description:
API Operations Related to Object Tagging . The following C# example lists object keys in the specified bucket. For illustration, the example obtains require 'aws-sdk-s3' s3 = Aws::S3::Resource.new(region:'us-west-2').
ebook img

Amazon Simple Storage Service PDF

758 Pages·2017·7.39 MB·English
by  
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Amazon Simple Storage Service

Amazon Simple Storage Service User Guide API Version 2006-03-01 Amazon Simple Storage Service User Guide Amazon Simple Storage Service: User Guide Amazon Simple Storage Service User Guide Table of Contents What is Amazon S3? ........................................................................................................................... 1 Features of Amazon S3 ............................................................................................................... 1 Storage classes.................................................................................................................. 1 Storage management......................................................................................................... 1 Access management........................................................................................................... 2 Data processing................................................................................................................. 2 Storage logging and monitoring.......................................................................................... 2 Analytics and insights......................................................................................................... 3 Strong consistency............................................................................................................. 3 How Amazon S3 works ............................................................................................................... 3 Buckets............................................................................................................................. 4 Objects............................................................................................................................. 4 Keys................................................................................................................................. 5 S3 Versioning.................................................................................................................... 5 Version ID ......................................................................................................................... 5 Bucket policy..................................................................................................................... 5 S3 Access Points................................................................................................................ 5 Access control lists (ACLs)................................................................................................... 6 Regions............................................................................................................................. 6 Amazon S3 data consistency model .............................................................................................. 6 Concurrent applications...................................................................................................... 7 Related services......................................................................................................................... 8 Accessing Amazon S3 ................................................................................................................. 9 Amazon Web Services Management Console ......................................................................... 9 Amazon Command Line Interface......................................................................................... 9 Amazon SDKs.................................................................................................................... 9 Amazon S3 REST API.......................................................................................................... 9 Paying for Amazon S3 .............................................................................................................. 10 PCI DSS compliance.................................................................................................................. 10 Getting started ................................................................................................................................ 11 Setting up............................................................................................................................... 11 Sign up for an Amazon Web Services account...................................................................... 12 Secure IAM users .............................................................................................................. 12 Step 1: Create a bucket ............................................................................................................. 12 Step 2: Upload an object .......................................................................................................... 14 Step 3: Download an object...................................................................................................... 15 Using the S3 console ........................................................................................................ 15 Step 4: Copy an object ............................................................................................................. 15 Step 5: Delete the objects and bucket ........................................................................................ 16 Deleting an object ............................................................................................................ 16 Emptying your bucket ....................................................................................................... 17 Deleting your bucket ........................................................................................................ 17 Next steps............................................................................................................................... 17 Understand common use cases .......................................................................................... 18 Control access to your buckets and objects .......................................................................... 18 Manage and monitor your storage ...................................................................................... 19 Develop with Amazon S3 .................................................................................................. 19 Learn from tutorials ......................................................................................................... 20 Explore training and support ............................................................................................. 21 Access control.......................................................................................................................... 22 Creating a new bucket ...................................................................................................... 22 Storing and sharing data................................................................................................... 23 Sharing resources............................................................................................................. 24 Protecting data................................................................................................................ 24 API Version 2006-03-01 iii Amazon Simple Storage Service User Guide Tutorials.......................................................................................................................................... 27 Getting started........................................................................................................................ 20 Optimizing storage costs........................................................................................................... 20 Hosting videos and websites...................................................................................................... 20 Data processing ........................................................................................................................ 20 Protecting data........................................................................................................................ 20 Transforming data with S3 Object Lambda.................................................................................. 28 Prerequisites.................................................................................................................... 29 Step 1: Create an S3 bucket .............................................................................................. 30 Step 2: Upload a file to the S3 bucket ................................................................................ 31 Step 3: Create an S3 access point ....................................................................................... 31 Step 4: Create a Lambda function ...................................................................................... 32 Step 5: Configure an IAM policy for your Lambda function's execution role ............................... 36 Step 6: Create an S3 Object Lambda access point ................................................................. 36 Step 7: View the transformed data..................................................................................... 37 Step 8: Clean up.............................................................................................................. 39 Next steps....................................................................................................................... 41 Detecting and redacting PII data ................................................................................................ 42 Prerequisites: Create an IAM user with permissions ............................................................... 43 Step 1: Create an S3 bucket .............................................................................................. 44 Step 2: Upload a file to the S3 bucket ................................................................................ 45 Step 3: Create an S3 access point ....................................................................................... 45 Step 4: Configure and deploy a prebuilt Lambda function ...................................................... 46 Step 5: Create an S3 Object Lambda access point ................................................................. 47 Step 6: Use the S3 Object Lambda access point to retrieve the redacted file ............................. 48 Step 7: Clean up.............................................................................................................. 48 Next steps....................................................................................................................... 51 Hosting video streaming........................................................................................................... 52 Prerequisites: Register and configure a custom domain with Route 53 ..................................... 52 Step 1: Create an S3 bucket .............................................................................................. 53 Step 2: Upload a video to the S3 bucket ............................................................................. 54 Step 3: Create a CloudFront origin access identity................................................................ 54 Step 4: Create a CloudFront distribution .............................................................................. 54 Step 5: Access the video through the CloudFront distribution ................................................. 56 Step 6: Configure your CloudFront distribution to use your custom domain name ...................... 57 Step 7: Access the S3 video through the CloudFront distribution with the custom domain name .. 60 (Optional) Step 8: View data about requests received by your CloudFront distribution ................ 60 Step 9: Clean up.............................................................................................................. 61 Next steps....................................................................................................................... 64 Batch-transcoding videos.......................................................................................................... 64 Prerequisites.................................................................................................................... 65 Step 1: Create an S3 bucket for the output media files ......................................................... 65 Step 2: Create an IAM role for MediaConvert ....................................................................... 67 Step 3: Create an IAM role for your Lambda function ............................................................ 67 Step 4: Create a Lambda function for video transcoding ........................................................ 69 Step 5: Configure Amazon S3 Inventory for your S3 source bucket .......................................... 81 Step 6: Create an IAM role for S3 Batch Operations .............................................................. 83 Step 7: Create and run an S3 Batch Operations job .............................................................. 86 Step 8: Check the output media files from your S3 destination bucket ..................................... 89 Step 9: Clean up.............................................................................................................. 89 Next steps....................................................................................................................... 91 Configuring a static website...................................................................................................... 92 Step 1: Create a bucket ..................................................................................................... 92 Step 2: Enable static website hosting .................................................................................. 92 Step 3: Edit Block Public Access settings ............................................................................. 93 Step 4: Add a bucket policy that makes your bucket content publicly available .......................... 94 Step 5: Configure an index document ................................................................................. 95 API Version 2006-03-01 iv Amazon Simple Storage Service User Guide Step 6: Configure an error document .................................................................................. 96 Step 7: Test your website endpoint .................................................................................... 97 Step 8: Clean up.............................................................................................................. 97 Configuring a static website using a custom domain ..................................................................... 97 Before you begin .............................................................................................................. 98 Step 1: Register a custom domain with Route 53 .................................................................. 98 Step 2: Create two buckets ................................................................................................ 98 Step 3: Configure root Domain bucket ................................................................................ 99 Step 4: Configure subdomain bucket for redirect ................................................................ 100 Step 5: Configure logging ................................................................................................ 101 Step 6: Upload index and website content ......................................................................... 101 Step 7: Upload an error document .................................................................................... 102 Step 8: Edit Block Public Access ....................................................................................... 103 Step 9: Attach a bucket policy ......................................................................................... 104 Step 10: Test your domain endpoint ................................................................................. 105 Step 11: Add alias records ............................................................................................... 106 Step 12: Test the website ................................................................................................ 109 Speeding up your website with Amazon CloudFront ............................................................ 109 Cleaning up example resources ........................................................................................ 112 Working with buckets ..................................................................................................................... 114 Buckets overview.................................................................................................................... 114 About permissions.......................................................................................................... 115 Managing public access to buckets ................................................................................... 115 Bucket configuration....................................................................................................... 116 Naming rules......................................................................................................................... 118 Example bucket names .................................................................................................... 119 Creating a bucket ................................................................................................................... 119 Default settings for new S3 buckets FAQ ................................................................................... 125 Viewing bucket properties ....................................................................................................... 126 Methods for accessing a bucket ................................................................................................ 127 Virtual-hosted–style access.............................................................................................. 127 Path-style access............................................................................................................ 127 Accessing an S3 bucket over IPv6 ..................................................................................... 128 Accessing a bucket through S3 access points ...................................................................... 128 Accessing a bucket using S3:// ......................................................................................... 128 Emptying a bucket ................................................................................................................. 129 Deleting a bucket ................................................................................................................... 130 Setting default bucket encryption ............................................................................................ 133 Using SSE-KMS encryption for cross-account operations ...................................................... 134 Using default encryption with replication .......................................................................... 134 Using Amazon S3 Bucket Keys with default encryption ........................................................ 135 Enabling default encryption ............................................................................................. 135 Monitoring default encryption .......................................................................................... 137 Configuring Transfer Acceleration ............................................................................................. 138 Why use Transfer Acceleration? ........................................................................................ 138 Requirements for using Transfer Acceleration ..................................................................... 138 Getting Started .............................................................................................................. 139 Enabling Transfer Acceleration ......................................................................................... 140 Speed Comparison tool................................................................................................... 145 Using Requester Pays .............................................................................................................. 145 How Requester Pays charges work .................................................................................... 146 Configuring Requester Pays ............................................................................................. 146 Retrieving the requestPayment configuration ..................................................................... 148 Downloading objects in Requester Pays buckets ................................................................. 148 Restrictions and limitations ...................................................................................................... 149 Working with objects ...................................................................................................................... 151 Objects.................................................................................................................................. 151 API Version 2006-03-01 v Amazon Simple Storage Service User Guide Subresources.................................................................................................................. 152 Creating object keys ............................................................................................................... 152 Object key naming guidelines .......................................................................................... 153 Working with metadata ........................................................................................................... 155 System-defined object metadata ...................................................................................... 156 User-defined object metadata .......................................................................................... 157 Editing object metadata .................................................................................................. 159 Uploading objects................................................................................................................... 160 Using multipart upload ........................................................................................................... 169 Multipart upload process ................................................................................................. 169 Checksums with multipart upload operations ..................................................................... 170 Concurrent multipart upload operations ............................................................................ 171 Multipart upload and pricing ........................................................................................... 171 API support for multipart upload ..................................................................................... 171 Amazon Command Line Interface support for multipart upload ............................................ 172 Amazon SDK support for multipart upload ........................................................................ 172 Multipart upload API and permissions ............................................................................... 172 Configuring a lifecycle policy ........................................................................................... 174 Uploading an object using multipart upload ...................................................................... 176 Uploading a directory ..................................................................................................... 189 Listing multipart uploads ................................................................................................. 191 Tracking a multipart upload ............................................................................................. 193 Aborting a multipart upload ............................................................................................ 195 Copying an object .......................................................................................................... 199 Multipart upload limits .................................................................................................... 204 Copying objects...................................................................................................................... 204 To copy an object ........................................................................................................... 205 Downloading an object ........................................................................................................... 211 Checking object integrity ......................................................................................................... 218 Using supported checksum algorithms .............................................................................. 218 Using Content-MD5 when uploading objects ...................................................................... 224 Using Content-MD5 and the ETag to verify uploaded objects ............................................... 224 Using trailing checksums................................................................................................. 225 Using part-level checksums for multipart uploads ............................................................... 225 Deleting objects..................................................................................................................... 226 Programmatically deleting objects from a version-enabled bucket ........................................ 227 Deleting objects from an MFA-enabled bucket .................................................................... 227 Deleting a single object ................................................................................................... 227 Deleting multiple objects ................................................................................................. 234 Organizing and listing objects .................................................................................................. 246 Using prefixes................................................................................................................ 247 Listing objects................................................................................................................ 248 Using folders................................................................................................................. 258 Viewing an object overview ............................................................................................. 261 Viewing object properties ................................................................................................ 261 Using presigned URLs ............................................................................................................. 262 Limiting presigned URL capabilities ................................................................................... 262 Who can create a presigned URL ...................................................................................... 263 When does Amazon S3 check the expiration date and time of a presigned URL? ...................... 264 Sharing objects.............................................................................................................. 264 Uploading objects........................................................................................................... 268 Deleting an object .......................................................................................................... 282 Transforming objects.............................................................................................................. 284 Creating Object Lambda access points .............................................................................. 286 Using Amazon S3 Object Lambda Access Points ................................................................. 295 Security considerations.................................................................................................... 296 Writing Lambda functions ............................................................................................... 300 API Version 2006-03-01 vi Amazon Simple Storage Service User Guide Using Amazon built functions .......................................................................................... 321 Best practices and guidelines for S3 Object Lambda ........................................................... 322 S3 Object Lambda tutorials ............................................................................................. 324 Debugging S3 Object Lambda .......................................................................................... 324 Working with access points .............................................................................................................. 325 Configuring IAM policies.......................................................................................................... 325 Access point policy examples ........................................................................................... 326 Condition keys............................................................................................................... 328 Delegating access control to access points ......................................................................... 329 Granting permissions for cross-account access points .......................................................... 330 Creating access points ............................................................................................................. 330 Rules for naming Amazon S3 access points ........................................................................ 330 Creating an access point .................................................................................................. 331 Creating access points restricted to a VPC ......................................................................... 332 Managing public access ................................................................................................... 334 Using access points ................................................................................................................. 335 Monitoring and logging................................................................................................... 335 Managing access points ................................................................................................... 337 Using a bucket-style alias for your access point .................................................................. 339 Using access points with Amazon S3 operations ................................................................. 340 Restrictions and limitations ...................................................................................................... 342 Working with Multi-Region Access Points ........................................................................................... 344 Creating Multi-Region Access Points .......................................................................................... 346 Rules for naming Amazon S3 Multi-Region Access Points ..................................................... 347 Rules for choosing buckets for Amazon S3 Multi-Region Access Points ................................... 348 Blocking public access with Amazon S3 Multi-Region Access Points ....................................... 348 Creating Amazon S3 Multi-Region Access Points ................................................................. 349 Configuring Amazon PrivateLink ....................................................................................... 350 Using a Multi-Region Access Point ............................................................................................ 352 Multi-Region Access Point hostnames ................................................................................ 353 Multi-Region Access Points and Amazon S3 Transfer Acceleration ......................................... 354 Multi-Region Access Point permissions .............................................................................. 354 Request routing.............................................................................................................. 358 Failover configuration..................................................................................................... 359 Bucket replication........................................................................................................... 364 Supported operations..................................................................................................... 367 Managing Multi-Region Access Points ........................................................................................ 373 Monitoring and logging........................................................................................................... 373 Monitoring and logging requests made to Multi-Region Access Point management APIs ........... 374 Using CloudTrail............................................................................................................. 375 Restrictions and limitations ...................................................................................................... 375 Security......................................................................................................................................... 378 Data protection...................................................................................................................... 379 Data encryption..................................................................................................................... 379 Server-side encryption.................................................................................................... 380 Using client-side encryption ............................................................................................. 424 Internetwork privacy............................................................................................................... 428 Traffic between service and on-premises clients and applications .......................................... 428 Traffic between Amazon resources in the same Region ........................................................ 429 Amazon PrivateLink for Amazon S3 .......................................................................................... 429 Types of VPC endpoints .................................................................................................. 429 Restrictions and limitations of Amazon PrivateLink for Amazon S3 ........................................ 430 Creating a VPC endpoint ................................................................................................. 430 Accessing Amazon S3 interface endpoints .......................................................................... 430 Accessing buckets and S3 access points from S3 interface endpoints ..................................... 431 Updating an on-premises DNS configuration ...................................................................... 434 Creating a VPC endpoint policy ........................................................................................ 435 API Version 2006-03-01 vii Amazon Simple Storage Service User Guide Identity and access management .............................................................................................. 438 Overview....................................................................................................................... 438 Access policy guidelines ................................................................................................... 444 Request authorization..................................................................................................... 448 Bucket policies and user policies ....................................................................................... 456 Amazon managed policies ............................................................................................... 611 Managing access with ACLs .............................................................................................. 613 Using CORS ................................................................................................................... 632 Blocking public access ..................................................................................................... 643 Reviewing bucket access .................................................................................................. 653 Verifying bucket ownership .............................................................................................. 658 Controlling object ownership ................................................................................................... 661 Object Ownership settings ............................................................................................... 663 Changes introduced by disabling ACLs ............................................................................... 664 Prerequisites for disabling ACLs ........................................................................................ 665 Object Ownership permissions ......................................................................................... 666 Disabling ACLs for all new buckets ................................................................................... 666 Replication and Object Ownership .................................................................................... 667 Setting Object Ownership ................................................................................................ 667 Prerequisites for disabling ACLs ........................................................................................ 668 Creating a bucket ........................................................................................................... 676 Setting Object Ownership ................................................................................................ 679 Viewing Object Ownership settings ................................................................................... 681 Disabling ACLs for all new buckets ................................................................................... 682 Troubleshooting............................................................................................................. 684 Logging and monitoring.......................................................................................................... 686 Compliance Validation............................................................................................................. 687 Resilience.............................................................................................................................. 688 Backup encryption.......................................................................................................... 690 Infrastructure security............................................................................................................. 691 Configuration and vulnerability analysis .................................................................................... 692 Security Best Practices ............................................................................................................ 693 Amazon S3 preventative security best Practices .................................................................. 693 Amazon S3 Monitoring and auditing best practices ............................................................. 696 Managing storage........................................................................................................................... 698 Using S3 Versioning ................................................................................................................ 698 Unversioned, versioning-enabled, and versioning-suspended buckets ..................................... 699 Using S3 Versioning with S3 Lifecycle ............................................................................... 699 S3 Versioning ................................................................................................................. 700 Enabling versioning on buckets ........................................................................................ 703 Configuring MFA delete ................................................................................................... 707 Working with versioning-enabled objects ........................................................................... 708 Working with versioning-suspended objects ....................................................................... 727 Using Amazon Backup for Amazon S3 ....................................................................................... 729 Working with archived objects ................................................................................................. 730 Archive retrieval options .................................................................................................. 731 Restoring an archived object ............................................................................................ 733 Using Object Lock .................................................................................................................. 737 S3 Object Lock ............................................................................................................... 738 Configuring Object Lock on the console ............................................................................ 742 Managing Object Lock .................................................................................................... 743 Managing storage classes........................................................................................................ 746 Frequently accessed objects ............................................................................................. 746 Automatically optimizing data with changing or unknown access patterns ............................. 746 Infrequently accessed objects ........................................................................................... 747 Archiving objects............................................................................................................ 748 Amazon S3 on Outposts .................................................................................................. 749 API Version 2006-03-01 viii Amazon Simple Storage Service User Guide Comparing storage classes............................................................................................... 750 Setting the storage class of an object ............................................................................... 750 Amazon S3 Intelligent-Tiering .................................................................................................. 751 How S3 Intelligent-Tiering works ..................................................................................... 752 Using S3 Intelligent-Tiering ............................................................................................ 753 Managing S3 Intelligent-Tiering ...................................................................................... 756 Managing lifecycle.................................................................................................................. 760 Managing object lifecycle ................................................................................................ 760 Creating a lifecycle configuration ...................................................................................... 760 Transitioning objects....................................................................................................... 761 Expiring objects.............................................................................................................. 766 Setting lifecycle configuration .......................................................................................... 766 Using other bucket configurations .................................................................................... 777 Configuring Lifecycle event notifications ........................................................................... 778 Lifecycle configuration elements ...................................................................................... 780 Examples of S3 Lifecycle configuration .............................................................................. 787 Managing inventory................................................................................................................ 798 Amazon S3 Inventory buckets .......................................................................................... 798 Inventory lists................................................................................................................ 799 Configuring Amazon S3 Inventory .................................................................................... 800 Setting up notifications for inventory completion ............................................................... 804 Locating your inventory .................................................................................................. 805 Querying inventory with Athena ....................................................................................... 808 Converting empty version ID strings to null strings ............................................................. 810 Replicating objects.................................................................................................................. 812 Why use replication ........................................................................................................ 813 When to use Cross-Region Replication .............................................................................. 813 When to use Same-Region Replication .............................................................................. 814 When to use two-way replication (bi-directional replication) ................................................. 814 When to use S3 Batch Replication .................................................................................... 814 Requirements for replication ............................................................................................ 815 What's replicated?........................................................................................................... 815 Setting up replication ..................................................................................................... 817 Replicate existing objects ................................................................................................ 858 Additional configurations................................................................................................. 866 Getting replication status ................................................................................................ 883 Troubleshooting............................................................................................................. 885 Additional considerations................................................................................................. 887 Using object tags ................................................................................................................... 888 API operations related to object tagging ........................................................................... 890 Additional configurations................................................................................................. 891 Access control................................................................................................................ 891 Managing object tags ...................................................................................................... 893 Using cost allocation tags ........................................................................................................ 897 More Info ...................................................................................................................... 898 Billing and usage reporting ...................................................................................................... 898 Billing reports................................................................................................................ 899 Usage report.................................................................................................................. 900 Understanding billing and usage reports ........................................................................... 902 Using Amazon S3 Select .......................................................................................................... 915 Requirements and limits .................................................................................................. 915 Constructing a request .................................................................................................... 916 Errors............................................................................................................................ 916 S3 Select examples ......................................................................................................... 917 SQL Reference............................................................................................................... 919 Using Batch Operations........................................................................................................... 944 Batch Operations basics.................................................................................................. 944 API Version 2006-03-01 ix Amazon Simple Storage Service User Guide S3 Batch Operations tutorial............................................................................................ 945 Granting permissions...................................................................................................... 945 Creating a job ................................................................................................................ 952 Supported operations..................................................................................................... 959 Managing jobs................................................................................................................ 983 Tracking job status and completion reports ....................................................................... 986 Using tags..................................................................................................................... 996 Managing S3 Object Lock .............................................................................................. 1006 S3 Batch Operations tutorial.......................................................................................... 1022 Monitoring Amazon S3.................................................................................................................. 1023 Monitoring tools................................................................................................................... 1023 Automated tools........................................................................................................... 1023 Manual tools................................................................................................................ 1024 Logging options................................................................................................................... 1024 Logging with CloudTrail ......................................................................................................... 1026 Using CloudTrail logs with Amazon S3 server access logs and CloudWatch Logs ..................... 1026 CloudTrail tracking with Amazon S3 SOAP API calls .......................................................... 1027 CloudTrail events.......................................................................................................... 1027 Example log files.......................................................................................................... 1031 Enabling CloudTrail ....................................................................................................... 1035 Identifying S3 requests .................................................................................................. 1037 Logging server access ............................................................................................................ 1043 How do I enable log delivery? ........................................................................................ 1043 Log object key format ................................................................................................... 1044 How are logs delivered?................................................................................................ 1044 Best effort server log delivery ........................................................................................ 1044 Bucket logging status changes take effect over time ......................................................... 1045 Enabling server access logging ....................................................................................... 1045 Log format.................................................................................................................. 1054 Deleting log files.......................................................................................................... 1064 Identifying S3 requests .................................................................................................. 1064 Monitoring metrics with CloudWatch ....................................................................................... 1068 Metrics and dimensions................................................................................................. 1069 Accessing CloudWatch metrics........................................................................................ 1079 CloudWatch metrics configurations ................................................................................. 1080 Amazon S3 Event Notifications ............................................................................................... 1086 Overview..................................................................................................................... 1086 Notification types and destinations................................................................................. 1087 Using SQS, SNS, and Lambda ......................................................................................... 1091 Using EventBridge........................................................................................................ 1110 Using analytics and insights ........................................................................................................... 1117 Storage Class Analysis........................................................................................................... 1117 How to set up storage class analysis ............................................................................... 1117 Storage class analysis.................................................................................................... 1118 How can I export storage class analysis data? ................................................................... 1119 Configuring storage class analysis................................................................................... 1120 S3 Storage Lens................................................................................................................... 1122 S3 Storage Lens metrics and features .............................................................................. 1123 Understanding S3 Storage Lens...................................................................................... 1125 Working with Organizations........................................................................................... 1131 S3 Storage Lens permissions.......................................................................................... 1133 Viewing storage metrics................................................................................................ 1135 Amazon S3 Storage Lens metrics use cases ...................................................................... 1154 Metrics glossary............................................................................................................ 1170 Working with S3 Storage Lens ....................................................................................... 1180 Tracing requests using X-Ray .................................................................................................. 1207 How X-Ray works with Amazon S3 .................................................................................. 1207 API Version 2006-03-01 x

Description:
API Operations Related to Object Tagging . The following C# example lists object keys in the specified bucket. For illustration, the example obtains require 'aws-sdk-s3' s3 = Aws::S3::Resource.new(region:'us-west-2').
See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users