Amazon Redshift Database Developer Guide API Version 2012-12-01 Amazon Redshift Database Developer Guide Amazon Redshift: Database Developer Guide Copyright © 2018 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Redshift Database Developer Guide Table of Contents Welcome........................................................................................................................................... 1 Are You a First-Time Amazon Redshift User? ................................................................................. 1 Are You a Database Developer? ................................................................................................... 2 Prerequisites.............................................................................................................................. 3 Amazon Redshift System Overview ...................................................................................................... 4 Data Warehouse System Architecture ........................................................................................... 4 Performance.............................................................................................................................. 6 Massively Parallel Processing .............................................................................................. 6 Columnar Data Storage..................................................................................................... 7 Data Compression............................................................................................................. 7 Query Optimizer................................................................................................................ 7 Result Caching.................................................................................................................. 7 Compiled Code.................................................................................................................. 8 Columnar Storage...................................................................................................................... 8 Internal Architecture and System Operation ................................................................................ 10 Workload Management............................................................................................................. 11 Using Amazon Redshift with Other Services ................................................................................ 11 Moving Data Between Amazon Redshift and Amazon S3 ....................................................... 11 Using Amazon Redshift with Amazon DynamoDB ................................................................. 11 Importing Data from Remote Hosts over SSH ...................................................................... 11 Automating Data Loads Using AWS Data Pipeline ................................................................. 12 Migrating Data Using AWS Database Migration Service (AWS DMS) ......................................... 12 Getting Started Using Databases ........................................................................................................ 13 Step 1: Create a Database......................................................................................................... 13 Step 2: Create a Database User .................................................................................................. 14 Delete a Database User..................................................................................................... 14 Step 3: Create a Database Table ................................................................................................. 14 Insert Data Rows into a Table ............................................................................................ 15 Select Data from a Table ................................................................................................... 15 Step 4: Load Sample Data......................................................................................................... 15 Step 5: Query the System Tables ............................................................................................... 16 View a List of Table Names ............................................................................................... 16 View Database Users........................................................................................................ 17 View Recent Queries ......................................................................................................... 17 Determine the Process ID of a Running Query..................................................................... 18 Step 6: Cancel a Query ............................................................................................................. 18 Cancel a Query from Another Session ................................................................................. 19 Cancel a Query Using the Superuser Queue ......................................................................... 19 Step 7: Clean Up Your Resources ................................................................................................ 20 Amazon Redshift Best Practices......................................................................................................... 21 Best Practices for Designing Tables ............................................................................................. 21 Take the Tuning Table Design Tutorial ................................................................................ 22 Choose the Best Sort Key .................................................................................................. 22 Choose the Best Distribution Style ..................................................................................... 22 Use Automatic Compression .............................................................................................. 23 Define Constraints............................................................................................................ 23 Use the Smallest Possible Column Size ............................................................................... 23 Using Date/Time Data Types for Date Columns .................................................................... 24 Best Practices for Loading Data................................................................................................. 24 Take the Loading Data Tutorial .......................................................................................... 24 Take the Tuning Table Design Tutorial ................................................................................ 24 Use a COPY Command to Load Data .................................................................................. 25 Use a Single COPY Command ............................................................................................ 25 Split Your Load Data into Multiple Files .............................................................................. 25 API Version 2012-12-01 iii Amazon Redshift Database Developer Guide Compress Your Data Files .................................................................................................. 25 Use a Manifest File........................................................................................................... 25 Verify Data Files Before and After a Load ............................................................................ 25 Use a Multi-Row Insert ..................................................................................................... 26 Use a Bulk Insert .............................................................................................................. 26 Load Data in Sort Key Order .............................................................................................. 26 Load Data in Sequential Blocks.......................................................................................... 26 Use Time-Series Tables ..................................................................................................... 27 Use a Staging Table to Perform a Merge............................................................................. 27 Schedule Around Maintenance Windows ............................................................................. 27 Best Practices for Designing Queries ........................................................................................... 27 Tutorial: Tuning Table Design............................................................................................................. 30 Prerequisites............................................................................................................................ 30 Steps...................................................................................................................................... 30 Step 1: Create a Test Data Set ................................................................................................... 30 To Create a Test Data Set .................................................................................................. 31 Next Step........................................................................................................................ 34 Step 2: Establish a Baseline....................................................................................................... 34 To Test System Performance to Establish a Baseline ............................................................. 35 Next Step........................................................................................................................ 37 Step 3: Select Sort Keys ............................................................................................................ 37 To Select Sort Keys .......................................................................................................... 38 Next Step........................................................................................................................ 38 Step 4: Select Distribution Styles ............................................................................................... 38 Distribution Styles............................................................................................................ 39 To Select Distribution Styles .............................................................................................. 39 Next Step........................................................................................................................ 42 Step 5: Review Compression Encodings....................................................................................... 42 To Review Compression Encodings ..................................................................................... 42 Next Step........................................................................................................................ 44 Step 6: Recreate the Test Data Set ............................................................................................. 44 To Recreate the Test Data Set ............................................................................................ 45 Next Step........................................................................................................................ 47 Step 7: Retest System Performance After Tuning ......................................................................... 47 To Retest System Performance After Tuning ........................................................................ 47 Next Step........................................................................................................................ 51 Step 8: Evaluate the Results...................................................................................................... 51 Next Step........................................................................................................................ 53 Step 9: Clean Up Your Resources ................................................................................................ 53 Next Step........................................................................................................................ 53 Summary................................................................................................................................ 53 Next Step........................................................................................................................ 54 Tutorial: Loading Data from Amazon S3.............................................................................................. 55 Prerequisites............................................................................................................................ 55 Overview................................................................................................................................. 55 Steps...................................................................................................................................... 56 Step 1: Launch a Cluster........................................................................................................... 56 Next Step........................................................................................................................ 57 Step 2: Download the Data Files ................................................................................................ 57 Next Step........................................................................................................................ 57 Step 3: Upload the Files to an Amazon S3 Bucket ........................................................................ 57 ...................................................................................................................................... 58 Next Step........................................................................................................................ 58 Step 4: Create the Sample Tables ............................................................................................... 59 Next Step........................................................................................................................ 61 Step 5: Run the COPY Commands.............................................................................................. 61 COPY Command Syntax .................................................................................................... 61 API Version 2012-12-01 iv Amazon Redshift Database Developer Guide Loading the SSB Tables ..................................................................................................... 62 Step 6: Vacuum and Analyze the Database .................................................................................. 72 Next Step........................................................................................................................ 73 Step 7: Clean Up Your Resources ................................................................................................ 73 Next............................................................................................................................... 73 Summary................................................................................................................................ 73 Next Step........................................................................................................................ 74 Tutorial: Configuring WLM Queues to Improve Query Processing ............................................................ 75 Overview................................................................................................................................. 75 Prerequisites.................................................................................................................... 75 Sections.......................................................................................................................... 75 Section 1: Understanding the Default Queue Processing Behavior ................................................... 75 Step 1: Create the WLM_QUEUE_STATE_VW View ................................................................ 76 Step 2: Create the WLM_QUERY_STATE_VW View ................................................................. 77 Step 3: Run Test Queries ................................................................................................... 78 Section 2: Modifying the WLM Query Queue Configuration ............................................................ 79 Step 1: Create a Parameter Group ...................................................................................... 79 Step 2: Configure WLM ..................................................................................................... 80 Step 3: Associate the Parameter Group with Your Cluster ...................................................... 81 Section 3: Routing Queries to Queues Based on User Groups and Query Groups ................................ 83 Step 1: View Query Queue Configuration in the Database ...................................................... 83 Step 2: Run a Query Using the Query Group Queue .............................................................. 84 Step 3: Create a Database User and Group.......................................................................... 85 Step 4: Run a Query Using the User Group Queue ................................................................ 85 Section 4: Using wlm_query_slot_count to Temporarily Override Concurrency Level in a Queue ........... 86 Step 1: Override the Concurrency Level Using wlm_query_slot_count ...................................... 87 Step 2: Run Queries from Different Sessions ........................................................................ 88 Section 5: Cleaning Up Your Resources ....................................................................................... 88 Managing Database Security .............................................................................................................. 89 Amazon Redshift Security Overview ........................................................................................... 89 Default Database User Privileges................................................................................................ 90 Superusers............................................................................................................................... 90 Users...................................................................................................................................... 91 Creating, Altering, and Deleting Users ................................................................................. 91 Groups.................................................................................................................................... 91 Creating, Altering, and Deleting Groups .............................................................................. 92 Schemas.................................................................................................................................. 92 Creating, Altering, and Deleting Schemas ............................................................................ 92 Search Path..................................................................................................................... 93 Schema-Based Privileges................................................................................................... 93 Example for Controlling User and Group Access ........................................................................... 93 Designing Tables.............................................................................................................................. 95 Choosing a Column Compression Type ........................................................................................ 95 Compression Encodings..................................................................................................... 96 Testing Compression Encodings ........................................................................................ 102 Example: Choosing Compression Encodings for the CUSTOMER Table .................................... 104 Choosing a Data Distribution Style ........................................................................................... 106 Data Distribution Concepts .............................................................................................. 106 Distribution Styles.......................................................................................................... 107 Viewing Distribution Styles .............................................................................................. 108 Evaluating Query Patterns ............................................................................................... 108 Designating Distribution Styles ......................................................................................... 109 Evaluating the Query Plan ............................................................................................... 109 Query Plan Example ....................................................................................................... 111 Distribution Examples..................................................................................................... 115 Choosing Sort Keys ................................................................................................................. 117 Compound Sort Key ........................................................................................................ 117 API Version 2012-12-01 v Amazon Redshift Database Developer Guide Interleaved Sort Key ....................................................................................................... 118 Comparing Sort Styles .................................................................................................... 119 Defining Constraints............................................................................................................... 122 Analyzing Table Design ........................................................................................................... 122 Using Amazon Redshift Spectrum to Query External Data ................................................................... 125 Amazon Redshift Spectrum Overview ....................................................................................... 125 Amazon Redshift Spectrum Regions .................................................................................. 126 Amazon Redshift Spectrum Considerations ........................................................................ 126 Getting Started With Amazon Redshift Spectrum ....................................................................... 127 Prerequisites.................................................................................................................. 127 Steps............................................................................................................................ 127 Step 1. Create an IAM Role .............................................................................................. 127 Step 2: Associate the IAM Role with Your Cluster ................................................................ 128 Step 3: Create an External Schema and an External Table .................................................... 129 Step 4: Query Your Data in Amazon S3 ............................................................................. 129 IAM Policies for Amazon Redshift Spectrum ............................................................................... 131 Creating Data Files for Queries in Amazon Redshift Spectrum ...................................................... 133 Creating External Schemas ...................................................................................................... 134 Working wth External Catalogs ........................................................................................ 136 Creating External Tables .......................................................................................................... 141 Pseudocolumns.............................................................................................................. 142 Partitioning Redshift Spectrum External Tables .................................................................. 143 Improving Amazon Redshift Spectrum Query Performance .......................................................... 146 Monitoring Metrics.................................................................................................................. 148 Troubleshooting Queries.......................................................................................................... 148 Retries Exceeded............................................................................................................ 149 No Rows Returned for a Partitioned Table ......................................................................... 149 Not Authorized Error ....................................................................................................... 149 Incompatible Data Formats .............................................................................................. 149 Syntax Error When Using Hive DDL in Amazon Redshift ....................................................... 150 Permission to Create Temporary Tables ............................................................................. 150 Loading Data................................................................................................................................. 151 Using COPY to Load Data ........................................................................................................ 151 Credentials and Access Permissions ................................................................................... 152 Preparing Your Input Data ............................................................................................... 153 Loading Data from Amazon S3 ........................................................................................ 154 Loading Data from Amazon EMR ...................................................................................... 163 Loading Data from Remote Hosts ..................................................................................... 167 Loading from Amazon DynamoDB .................................................................................... 173 Verifying That the Data Was Loaded Correctly ................................................................... 175 Validating Input Data ...................................................................................................... 175 Automatic Compression................................................................................................... 176 Optimizing for Narrow Tables .......................................................................................... 178 Default Values................................................................................................................ 178 Troubleshooting............................................................................................................. 178 Updating with DML................................................................................................................ 183 Updating and Inserting ........................................................................................................... 183 Merge Method 1: Replacing Existing Rows ......................................................................... 183 Merge Method 2: Specifying a Column List ........................................................................ 184 Creating a Temporary Staging Table ................................................................................. 184 Performing a Merge Operation by Replacing Existing Rows .................................................. 184 Performing a Merge Operation by Specifying a Column List ................................................. 185 Merge Examples............................................................................................................. 186 Performing a Deep Copy ......................................................................................................... 188 Analyzing Tables.................................................................................................................... 190 ANALYZE Command History ............................................................................................. 193 Automatic Analysis......................................................................................................... 194 API Version 2012-12-01 vi Amazon Redshift Database Developer Guide Vacuuming Tables................................................................................................................... 194 VACUUM Frequency........................................................................................................ 195 Sort Stage and Merge Stage ............................................................................................ 196 Vacuum Threshold .......................................................................................................... 196 Vacuum Types................................................................................................................ 196 Managing Vacuum Times ................................................................................................. 197 Vacuum Column Limit Exceeded Error ............................................................................... 203 Managing Concurrent Write Operations ..................................................................................... 204 Serializable Isolation....................................................................................................... 205 Write and Read-Write Operations ..................................................................................... 206 Concurrent Write Examples .............................................................................................. 207 Unloading Data.............................................................................................................................. 209 Unloading Data to Amazon S3 ................................................................................................. 209 Unloading Encrypted Data Files ................................................................................................ 212 Unloading Data in Delimited or Fixed-Width Format ................................................................... 213 Reloading Unloaded Data ........................................................................................................ 214 Creating User-Defined Functions ...................................................................................................... 215 UDF Security and Privileges ..................................................................................................... 215 Creating a Scalar SQL UDF ...................................................................................................... 215 Scalar SQL Function Example ........................................................................................... 216 Creating a Scalar Python UDF .................................................................................................. 216 Scalar Python UDF Example ............................................................................................. 217 Python UDF Data Types .................................................................................................. 217 ANYELEMENT Data Type ................................................................................................. 218 Python Language Support ............................................................................................... 218 UDF Constraints............................................................................................................. 221 Naming UDFs......................................................................................................................... 221 Overloading Function Names ........................................................................................... 222 Preventing UDF Naming Conflicts ..................................................................................... 222 Logging Errors and Warnings ................................................................................................... 222 Tuning Query Performance .............................................................................................................. 224 Query Processing.................................................................................................................... 224 Query Planning And Execution Workflow ........................................................................... 224 Reviewing Query Plan Steps ............................................................................................ 226 Query Plan.................................................................................................................... 227 Factors Affecting Query Performance ................................................................................ 233 Analyzing and Improving Queries ............................................................................................. 234 Query Analysis Workflow ................................................................................................. 234 Reviewing Query Alerts ................................................................................................... 235 Analyzing the Query Plan ................................................................................................ 236 Analyzing the Query Summary ......................................................................................... 237 Improving Query Performance ......................................................................................... 242 Diagnostic Queries for Query Tuning ................................................................................. 244 Troubleshooting Queries.......................................................................................................... 247 Connection Fails............................................................................................................. 248 Query Hangs.................................................................................................................. 248 Query Takes Too Long .................................................................................................... 249 Load Fails...................................................................................................................... 250 Load Takes Too Long ...................................................................................................... 250 Load Data Is Incorrect ..................................................................................................... 250 Setting the JDBC Fetch Size Parameter ............................................................................. 251 Implementing Workload Management............................................................................................... 252 Defining Query Queues ........................................................................................................... 252 Concurrency Level.......................................................................................................... 253 User Groups................................................................................................................... 254 Query Groups................................................................................................................ 254 Wildcards....................................................................................................................... 254 API Version 2012-12-01 vii Amazon Redshift Database Developer Guide WLM Memory Percent to Use ........................................................................................... 255 WLM Timeout................................................................................................................ 255 Query Monitoring Rules .................................................................................................. 255 WLM Query Queue Hopping .................................................................................................... 255 WLM Timeout Queue Hopping......................................................................................... 256 WLM Timeout Reassigned and Restarted Queries ................................................................ 256 QMR Hop Action Queue Hopping ..................................................................................... 256 QMR Hop Action Reassigned and Restarted Queries ............................................................ 257 WLM Query Queue Hopping Summary .............................................................................. 257 Short Query Acceleration ........................................................................................................ 258 Maximum SQA Run Time ................................................................................................. 259 Monitoring SQA.............................................................................................................. 259 Modifying the WLM Configuration ............................................................................................ 260 WLM Queue Assignment Rules................................................................................................. 260 Queue Assignments Example........................................................................................... 262 Assigning Queries to Queues................................................................................................... 263 Assigning Queries to Queues Based on User Groups ............................................................ 263 Assigning a Query to a Query Group ................................................................................. 263 Assigning Queries to the Superuser Queue........................................................................ 264 Dynamic and Static Properties ................................................................................................. 264 WLM Dynamic Memory Allocation .................................................................................... 265 Dynamic WLM Example................................................................................................... 265 Query Monitoring Rules .......................................................................................................... 266 Defining a Query Monitor Rule ......................................................................................... 267 Query Monitoring Metrics ................................................................................................ 268 Query Monitoring Rules Templates ................................................................................... 269 System Tables and Views for Query Monitoring Rules ......................................................... 270 WLM System Tables and Views ................................................................................................ 271 SQL Reference............................................................................................................................... 273 Amazon Redshift SQL ............................................................................................................. 273 SQL Functions Supported on the Leader Node ................................................................... 273 Amazon Redshift and PostgreSQL .................................................................................... 274 Using SQL............................................................................................................................. 279 SQL Reference Conventions ............................................................................................. 279 Basic Elements............................................................................................................... 280 Expressions.................................................................................................................... 303 Conditions..................................................................................................................... 307 SQL Commands...................................................................................................................... 323 ABORT.......................................................................................................................... 325 ALTER DATABASE........................................................................................................... 326 ALTER DEFAULT PRIVILEGES ............................................................................................ 327 ALTER GROUP................................................................................................................ 329 ALTER SCHEMA.............................................................................................................. 330 ALTER TABLE................................................................................................................. 331 ALTER TABLE APPEND ..................................................................................................... 339 ALTER USER................................................................................................................... 342 ANALYZE....................................................................................................................... 345 ANALYZE COMPRESSION................................................................................................. 347 BEGIN........................................................................................................................... 349 CANCEL......................................................................................................................... 350 CLOSE........................................................................................................................... 352 COMMENT..................................................................................................................... 353 COMMIT........................................................................................................................ 354 COPY............................................................................................................................ 355 CREATE DATABASE.......................................................................................................... 410 CREATE EXTERNAL SCHEMA ............................................................................................ 411 CREATE EXTERNAL TABLE ................................................................................................ 414 API Version 2012-12-01 viii Amazon Redshift Database Developer Guide CREATE FUNCTION......................................................................................................... 423 CREATE GROUP.............................................................................................................. 427 CREATE LIBRARY............................................................................................................ 428 CREATE SCHEMA............................................................................................................ 430 CREATE TABLE............................................................................................................... 431 CREATE TABLE AS ........................................................................................................... 443 CREATE USER................................................................................................................. 450 CREATE VIEW................................................................................................................. 454 DEALLOCATE.................................................................................................................. 456 DECLARE....................................................................................................................... 456 DELETE......................................................................................................................... 459 DROP DATABASE............................................................................................................ 461 DROP FUNCTION............................................................................................................ 461 DROP GROUP................................................................................................................ 462 DROP LIBRARY............................................................................................................... 463 DROP SCHEMA............................................................................................................... 463 DROP TABLE.................................................................................................................. 464 DROP USER................................................................................................................... 466 DROP VIEW................................................................................................................... 468 END.............................................................................................................................. 469 EXECUTE....................................................................................................................... 470 EXPLAIN........................................................................................................................ 471 FETCH........................................................................................................................... 475 GRANT.......................................................................................................................... 476 INSERT.......................................................................................................................... 480 LOCK............................................................................................................................ 485 PREPARE....................................................................................................................... 485 RESET........................................................................................................................... 487 REVOKE......................................................................................................................... 487 ROLLBACK..................................................................................................................... 491 SELECT.......................................................................................................................... 492 SELECT INTO.................................................................................................................. 520 SET............................................................................................................................... 521 SET SESSION AUTHORIZATION ........................................................................................ 524 SET SESSION CHARACTERISTICS ....................................................................................... 524 SHOW........................................................................................................................... 524 START TRANSACTION...................................................................................................... 525 TRUNCATE..................................................................................................................... 525 UNLOAD........................................................................................................................ 526 UPDATE......................................................................................................................... 538 VACUUM........................................................................................................................ 543 SQL Functions Reference ......................................................................................................... 546 Leader Node–Only Functions ........................................................................................... 546 Aggregate Functions....................................................................................................... 547 Bit-Wise Aggregate Functions .......................................................................................... 563 Window Functions.......................................................................................................... 568 Conditional Expressions................................................................................................... 612 Date and Time Functions ................................................................................................. 621 Math Functions.............................................................................................................. 657 String Functions............................................................................................................. 681 JSON Functions.............................................................................................................. 717 Data Type Formatting Functions ....................................................................................... 723 System Administration Functions ...................................................................................... 733 System Information Functions .......................................................................................... 736 Reserved Words...................................................................................................................... 749 System Tables Reference ................................................................................................................. 753 System Tables and Views ......................................................................................................... 753 API Version 2012-12-01 ix Amazon Redshift Database Developer Guide Types of System Tables and Views ............................................................................................ 753 Visibility of Data in System Tables and Views ............................................................................. 754 Filtering System-Generated Queries .................................................................................. 754 STL Tables for Logging ........................................................................................................... 754 STL_AGGR..................................................................................................................... 756 STL_ALERT_EVENT_LOG.................................................................................................. 757 STL_ANALYZE................................................................................................................. 759 STL_BCAST.................................................................................................................... 760 STL_COMMIT_STATS ....................................................................................................... 762 STL_CONNECTION_LOG................................................................................................... 763 STL_DDLTEXT................................................................................................................. 764 STL_DELETE................................................................................................................... 765 STL_DIST....................................................................................................................... 767 STL_ERROR.................................................................................................................... 769 STL_EXPLAIN................................................................................................................. 770 STL_FILE_SCAN.............................................................................................................. 771 STL_HASH..................................................................................................................... 772 STL_HASHJOIN............................................................................................................... 774 STL_INSERT................................................................................................................... 775 STL_LIMIT...................................................................................................................... 776 STL_LOAD_COMMITS...................................................................................................... 778 STL_LOAD_ERRORS......................................................................................................... 780 STL_LOADERROR_DETAIL................................................................................................ 782 STL_MERGE................................................................................................................... 784 STL_MERGEJOIN............................................................................................................. 785 STL_NESTLOOP.............................................................................................................. 786 STL_PARSE.................................................................................................................... 787 STL_PLAN_INFO............................................................................................................. 788 STL_PROJECT................................................................................................................. 790 STL_QUERY.................................................................................................................... 791 STL_QUERY_METRICS ...................................................................................................... 793 STL_QUERYTEXT............................................................................................................ 795 STL_REPLACEMENTS....................................................................................................... 797 STL_RESTARTED_SESSIONS............................................................................................. 798 STL_RETURN.................................................................................................................. 798 STL_S3CLIENT................................................................................................................ 799 STL_S3CLIENT_ERROR..................................................................................................... 801 STL_SAVE...................................................................................................................... 802 STL_SCAN...................................................................................................................... 803 STL_SESSIONS............................................................................................................... 805 STL_SORT...................................................................................................................... 806 STL_SSHCLIENT_ERROR................................................................................................... 808 STL_STREAM_SEGS......................................................................................................... 808 STL_TR_CONFLICT.......................................................................................................... 809 STL_UNDONE................................................................................................................. 810 STL_UNIQUE.................................................................................................................. 811 STL_UNLOAD_LOG.......................................................................................................... 812 STL_USERLOG................................................................................................................ 813 STL_UTILITYTEXT........................................................................................................... 814 STL_VACUUM................................................................................................................. 816 STL_WINDOW................................................................................................................ 818 STL_WLM_ERROR........................................................................................................... 819 STL_WLM_RULE_ACTION................................................................................................. 820 STL_WLM_QUERY........................................................................................................... 821 STV Tables for Snapshot Data .................................................................................................. 822 STV_ACTIVE_CURSORS.................................................................................................... 823 STV_BLOCKLIST............................................................................................................. 824 API Version 2012-12-01 x
Description: