Table Of Content

Amazon Neptune User Guide Amazon Neptune User Guide Amazon Neptune: User Guide Copyright © 2022 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Neptune User Guide Table of Contents What is Neptune?............................................................................................................................... 1 Latest Updates.................................................................................................................................. 3 Getting started ................................................................................................................................ 29 What's a graph database? .......................................................................................................... 29 Why use graphs?...................................................................................................................... 30 Graph database applications...................................................................................................... 31 Graph query languages ............................................................................................................. 33 Query examples............................................................................................................... 33 Online Neptune course ............................................................................................................. 35 Digging deeper........................................................................................................................ 35 Use graph notebooks................................................................................................................ 36 Using Neptune workbench ................................................................................................. 36 Enabling CloudWatch logs ................................................................................................. 39 Local hosting................................................................................................................... 39 Workbench magics................................................................................................................... 40 Variable injection............................................................................................................. 41 Common query arguments ................................................................................................ 42 %seed............................................................................................................................. 43 %load............................................................................................................................. 43 %load_ids....................................................................................................................... 43 %load_status................................................................................................................... 43 %cancel_load................................................................................................................... 43 %status........................................................................................................................... 44 %gremlin_status.............................................................................................................. 44 %opencypher_status, or %oc_status ................................................................................... 44 %sparql_status................................................................................................................. 44 %stream_viewer............................................................................................................ 44 %graph_notebook_config.................................................................................................. 44 %graph_notebook_host.................................................................................................... 45 %graph_notebook_version................................................................................................ 45 %graph_notebook_vis_options........................................................................................... 45 %%graph_notebook_config............................................................................................... 45 %%sparql ........................................................................................................................ 46 %%gremlin..................................................................................................................... 46 %%opencypher, or %%oc ................................................................................................. 47 %%graph_notebook_vis_options........................................................................................ 47 %neptune_ml.................................................................................................................. 48 %%neptune_ml............................................................................................................... 50 Graph visualization................................................................................................................... 52 Graph interface................................................................................................................ 52 Gremlin visualization........................................................................................................ 54 SPARQL visualization........................................................................................................ 55 Visualization tutorials....................................................................................................... 56 Neptune setup................................................................................................................................. 57 DB instance types..................................................................................................................... 57 Instance resource allocation ............................................................................................... 57 t3 and t4g..................................................................................................................... 58 r4 instances.................................................................................................................... 58 r5 instances.................................................................................................................... 58 r5d instances.................................................................................................................. 59 r6g instances.................................................................................................................. 59 x2g instances.................................................................................................................. 59 serverless instances...................................................................................................... 59 Create a DB cluster ................................................................................................................... 60 iii Amazon Neptune User Guide Prerequisites.................................................................................................................... 61 Create the Cluster ............................................................................................................ 64 Configure the VPC .................................................................................................................... 65 Add subnets.................................................................................................................... 66 Create a subnet group ...................................................................................................... 66 Create a security group ..................................................................................................... 67 DNS in Your VPC .............................................................................................................. 68 Connect to your graph .............................................................................................................. 68 Set up curl or awscurl.................................................................................................. 68 Ways to connect .............................................................................................................. 68 From within the VPC ........................................................................................................ 69 From a different VPC ........................................................................................................ 70 From a private network ..................................................................................................... 70 Neptune Security...................................................................................................................... 71 IAM policies..................................................................................................................... 71 VPC security groups .......................................................................................................... 71 IAM authentication........................................................................................................... 71 Access the graph...................................................................................................................... 72 Setting up curl ............................................................................................................... 68 Query languages.............................................................................................................. 73 Use Gremlin..................................................................................................................... 73 Use openCypher............................................................................................................... 77 Use RDF/SPARQL.............................................................................................................. 77 Loading Data........................................................................................................................... 78 Monitoring Neptune................................................................................................................. 78 Troubleshooting and Best Practices ............................................................................................ 78 Global databases.............................................................................................................................. 79 Overview................................................................................................................................. 79 Advantages.............................................................................................................................. 80 Limitations.............................................................................................................................. 80 Setup...................................................................................................................................... 81 Configuration requirements............................................................................................... 81 Creating a global database ................................................................................................ 82 Use an existing DB cluster as primary ................................................................................. 83 Adding a secondary region ................................................................................................ 83 Connecting ...................................................................................................................... 84 Managing a Neptune global database......................................................................................... 84 Removing a cluster ........................................................................................................... 85 Deleting a global database................................................................................................ 85 Modifying a global database .............................................................................................. 85 Using failover.......................................................................................................................... 86 Detach and promote ......................................................................................................... 86 Managed planned failovers ................................................................................................ 87 Monitoring Neptune global databases......................................................................................... 88 Neptune overview............................................................................................................................ 90 Standards Compliance............................................................................................................... 91 Gremlin standards compliance ........................................................................................... 91 SPARQL standards compliance ......................................................................................... 100 openCypher standards compliance .................................................................................... 104 Graph Data Model.................................................................................................................. 113 Indexing Strategy........................................................................................................... 113 Gremlin data model ........................................................................................................ 115 Lookup cache......................................................................................................................... 116 Use cases for the lookup cache........................................................................................ 116 Using the cache.............................................................................................................. 117 Transaction Semantics............................................................................................................. 118 Isolation Levels.............................................................................................................. 118 iv Amazon Neptune User Guide Neptune Isolation Levels ................................................................................................. 119 Transaction examples...................................................................................................... 123 Exceptions and Retries .................................................................................................... 125 Clusters and Instances ............................................................................................................. 127 The primary DB instance ................................................................................................. 127 Read-replica instances..................................................................................................... 127 Sizing instances.............................................................................................................. 128 Monitoring instances....................................................................................................... 128 Storage, reliability and availability ............................................................................................ 129 Allocation...................................................................................................................... 129 Storage billing............................................................................................................... 129 Storage best practices ..................................................................................................... 130 Reliability and high availability ......................................................................................... 130 Endpoint Connections............................................................................................................. 132 Endpoint considerations.................................................................................................. 134 Custom endpoints.......................................................................................................... 134 Create custom endpoints ................................................................................................. 135 View custom endpoints................................................................................................... 135 Edit custom endpoints .................................................................................................... 136 Delete custom endpoints................................................................................................. 136 Custom queryId...................................................................................................................... 137 Using the HTTP Header ................................................................................................... 137 Using a SPARQL Query Hint ............................................................................................. 137 Using queryId to Check Status ......................................................................................... 137 Lab Mode.............................................................................................................................. 139 Using Lab Mode............................................................................................................. 139 OSGP index................................................................................................................... 140 Transaction Semantics..................................................................................................... 140 Neptune DFE engine............................................................................................................... 141 Controlling DFE use ........................................................................................................ 141 Queries executed by the DFE ........................................................................................... 142 DFE statistics................................................................................................................. 142 JDBC connectivity................................................................................................................... 149 Getting started............................................................................................................... 149 Using Tableau ................................................................................................................ 150 Troubleshooting............................................................................................................. 152 Neptune engine updates......................................................................................................... 153 Security......................................................................................................................................... 154 Data Protection...................................................................................................................... 154 Amazon VPC protection .................................................................................................. 155 Encryption in Transit ....................................................................................................... 155 Encryption at Rest .......................................................................................................... 156 IAM overview......................................................................................................................... 159 Different roles................................................................................................................ 159 Using Identities.............................................................................................................. 160 Enabling IAM......................................................................................................................... 162 Connecting and Signing .......................................................................................................... 162 EC2 Prerequisites............................................................................................................ 163 Gremlin Console............................................................................................................. 164 Gremlin Java.................................................................................................................. 167 SPARQL Java (RDF4J and Jena) ........................................................................................ 168 SPARQL with Node.js ...................................................................................................... 170 Python Example............................................................................................................. 171 Using IAM Policies .................................................................................................................. 178 Identity-Based Policies.................................................................................................... 179 Service Control Policies (SCP) ........................................................................................... 179 Neptune Console Access .................................................................................................. 179 v Amazon Neptune User Guide Attaching a Policy .......................................................................................................... 179 Types of IAM policies ...................................................................................................... 180 Using condition Keys ....................................................................................................... 180 IAM feature support ........................................................................................................ 181 IAM Policy Limitations ..................................................................................................... 181 Managed policies............................................................................................................ 181 Condition Keys............................................................................................................... 189 Administrative policy statements ...................................................................................... 190 Data-access policy statements .......................................................................................... 207 Neptune Service-Linked Roles .................................................................................................. 219 Role Permissions............................................................................................................ 220 Creating a Service-Linked Role ......................................................................................... 221 Editing a Service-Linked Role ........................................................................................... 221 Deleting a Service-Linked Role ......................................................................................... 221 Temporary Credentials............................................................................................................ 223 Get Credentials with the AWS CLI ..................................................................................... 223 Setting Up Lambda ......................................................................................................... 225 Set Up Amazon EC2 ........................................................................................................ 226 Logging and Monitoring.......................................................................................................... 228 Compliance Validation............................................................................................................. 228 Resilience.............................................................................................................................. 229 Migrating to Neptune..................................................................................................................... 230 Migrating from Neo4j ............................................................................................................. 231 General information........................................................................................................ 231 Preparing for migration ................................................................................................... 233 Provisioning infrastructure............................................................................................... 237 Data migration............................................................................................................... 239 Application migration...................................................................................................... 243 Neptune compatibility..................................................................................................... 246 Cypher rewrites.............................................................................................................. 249 Migration resources......................................................................................................... 254 Migrating from TinkerPop ........................................................................................................ 255 Migrating from RDF ................................................................................................................ 256 Using AWS DMS to migrate ..................................................................................................... 257 Migrating from Blazegraph ...................................................................................................... 258 Neptune compatibility..................................................................................................... 258 Provisioning infrastructure............................................................................................... 258 Exporting data............................................................................................................... 259 Create an Amazon S3 bucket ........................................................................................... 261 Import the data ............................................................................................................. 261 Loading data.................................................................................................................................. 263 Neptune Bulk Loader.............................................................................................................. 263 IAM Role and Amazon S3 Access ...................................................................................... 264 Data Formats................................................................................................................. 267 Loading Example............................................................................................................ 277 Optimizing a bulk load.................................................................................................... 280 Loader Reference............................................................................................................ 281 Load data using DMS.............................................................................................................. 300 GraphMappingConfig...................................................................................................... 300 Replicating to Neptune ................................................................................................... 303 Querying ....................................................................................................................................... 307 Query queuing....................................................................................................................... 307 Finding how many queries are in the queue ....................................................................... 307 Query timeouts.............................................................................................................. 308 Gremlin................................................................................................................................. 308 Installing the Gremlin console .......................................................................................... 309 HTTPS REST.................................................................................................................. 313 vi Amazon Neptune User Guide Java.............................................................................................................................. 314 Python.......................................................................................................................... 322 .NET.............................................................................................................................. 324 Node.js.......................................................................................................................... 325 Query hints................................................................................................................... 327 Query status.................................................................................................................. 333 Query cancellation.......................................................................................................... 334 Gremlin script-based sessions ........................................................................................... 334 Gremlin transactions....................................................................................................... 335 Using the Gremlin API ..................................................................................................... 337 Caching query results ...................................................................................................... 337 Efficient upserts............................................................................................................. 342 Gremlin explain........................................................................................................... 351 Gremlin and DFE ............................................................................................................ 387 openCypher........................................................................................................................... 388 Gremlin vs. openCypher .................................................................................................. 388 Using openCypher.......................................................................................................... 389 Status endpoint.............................................................................................................. 390 HTTPS endpoint............................................................................................................. 392 Using the Bolt protocol ................................................................................................... 395 Parameterized examples.................................................................................................. 407 Data model.................................................................................................................... 408 openCypher explain..................................................................................................... 409 Transactions................................................................................................................... 420 Restrictions.................................................................................................................... 426 Exceptions..................................................................................................................... 426 SPARQL................................................................................................................................. 428 RDF4J Console............................................................................................................... 429 RDF4J Workbench.......................................................................................................... 431 Java.............................................................................................................................. 432 HTTP API....................................................................................................................... 434 Query hints................................................................................................................... 444 Query status.................................................................................................................. 455 Query cancellation.......................................................................................................... 457 Graph store protocol ....................................................................................................... 458 SPARQL explain ........................................................................................................... 459 SPARQL SERVICE Extension............................................................................................ 481 Visualization tools.......................................................................................................................... 483 Tom Sawyer Software ............................................................................................................. 483 Cambridge Intelligence............................................................................................................ 485 Graphistry.............................................................................................................................. 487 metaphacts............................................................................................................................ 488 G.V( )..................................................................................................................................... 491 Exporting data............................................................................................................................... 492 neptune-export...................................................................................................................... 493 Neptune-Export service........................................................................................................... 494 Install the service ........................................................................................................... 494 Enable access to Neptune ................................................................................................ 496 Enable access to Neptune-Export ..................................................................................... 496 Run an export job .......................................................................................................... 496 Monitor the job.............................................................................................................. 497 Cancel a job ................................................................................................................... 498 neptune-export utility............................................................................................................. 499 Prerequisites.................................................................................................................. 499 Running neptune-export................................................................................................. 500 Example commands........................................................................................................ 500 Exported files......................................................................................................................... 502 vii Amazon Neptune User Guide Export parameters.................................................................................................................. 503 command ...................................................................................................................... 504 outputS3Path................................................................................................................. 504 jobSize.......................................................................................................................... 504 params.......................................................................................................................... 505 additionalParams............................................................................................................ 505 params.......................................................................................................................... 506 Filtering examples.......................................................................................................... 514 Troubleshooting ..................................................................................................................... 518 Managing Neptune......................................................................................................................... 521 IAM user permissions.............................................................................................................. 522 Service-linked role policy ................................................................................................. 522 Create a new IAM user .................................................................................................... 522 Parameter groups................................................................................................................... 524 Editing a Parameter Group .............................................................................................. 525 Creating a parameter group ............................................................................................. 525 Parameters............................................................................................................................ 526 neptune_enable_audit_log............................................................................................... 526 neptune_lab_mode......................................................................................................... 526 neptune_query_timeout.................................................................................................. 526 neptune_streams............................................................................................................ 527 neptune_streams_expiry_days.......................................................................................... 527 neptune_lookup_cache.................................................................................................... 527 neptune_autoscaling_config ............................................................................................. 527 neptune_ml_iam_role...................................................................................................... 528 neptune_ml_endpoint..................................................................................................... 528 neptune_dfe_query_engine.............................................................................................. 528 neptune_query_timeout.................................................................................................. 528 neptune_result_cache...................................................................................................... 529 neptune_enforce_ssl ....................................................................................................... 529 Launch using the console ........................................................................................................ 530 Stopping and starting a cluster ................................................................................................ 534 Stopping and starting overview ........................................................................................ 534 Stopping a cluster.......................................................................................................... 534 Starting a DB cluster ....................................................................................................... 535 Fast reset API ......................................................................................................................... 537 Using IAM-Auth.............................................................................................................. 539 The %db_reset magic ..................................................................................................... 539 Common errors.............................................................................................................. 540 Creating a Replica .................................................................................................................. 541 Modifying a DB Cluster ........................................................................................................... 542 Modify an Instance ......................................................................................................... 542 Adding Neptune Replicas ......................................................................................................... 544 Performance and Scaling ......................................................................................................... 546 Storage Scaling.............................................................................................................. 546 Instance Scaling;............................................................................................................. 546 Read Scaling .................................................................................................................. 546 Auto-scaling........................................................................................................................... 547 Enabling auto-scaling...................................................................................................... 548 Remove auto-scaling ....................................................................................................... 550 Maintaining a DB Cluster......................................................................................................... 551 Maintenance Window...................................................................................................... 551 Neptune Updates................................................................................................................... 553 Manually Upgrading........................................................................................................ 553 Automatic Upgrading...................................................................................................... 553 Cloning a DB Cluster............................................................................................................... 555 Limitations..................................................................................................................... 556 viii Amazon Neptune User Guide Copy-on-Write Protocol ................................................................................................... 556 Deleting a Source Database ............................................................................................. 558 Managing Instances................................................................................................................ 559 T3 Burstable Instances .................................................................................................... 560 Modifying an Instance ..................................................................................................... 562 Renaming a Neptune DB Instance ..................................................................................... 565 Rebooting a DB instance ................................................................................................. 566 Deleting a DB Instance .................................................................................................... 567 Serverless...................................................................................................................................... 569 Serverless use cases ................................................................................................................ 569 Constraints............................................................................................................................ 569 Capacity scaling ..................................................................................................................... 570 Setting the minimum ...................................................................................................... 571 Setting the maximum ..................................................................................................... 571 Estimate capacity settings ............................................................................................... 572 Additional configuration.......................................................................................................... 573 Mixed configuration........................................................................................................ 573 Setting promotion tiers ................................................................................................... 573 Reader-writer alignment.................................................................................................. 574 Avoid very large timeout values ....................................................................................... 574 Optimizing your configuration .......................................................................................... 574 Using Serverless..................................................................................................................... 575 Creating a serverless cluster ............................................................................................. 575 Converting to Serverless .................................................................................................. 575 Modifying the capacity range ........................................................................................... 576 Converting to provisioned ............................................................................................... 576 Monitoring..................................................................................................................... 576 Neptune streams............................................................................................................................ 578 Using Streams........................................................................................................................ 579 Enabling Streams........................................................................................................... 579 Disabling Streams........................................................................................................... 580 Calling the Streams API ................................................................................................... 580 Streams Response........................................................................................................... 581 Streams Exceptions......................................................................................................... 583 Streams Record Formats .......................................................................................................... 583 PG_JSON....................................................................................................................... 584 RDF-NQUADS ................................................................................................................. 586 Streams Examples.................................................................................................................. 586 AT_SEQUENCE_NUMBER Examples................................................................................... 586 AFTER_SEQUENCE_NUMBER Example............................................................................... 587 TRIM_HORIZON Example................................................................................................. 588 LATEST Example............................................................................................................. 588 Compression Example..................................................................................................... 589 Neptune-to-Neptune Replication Setup ..................................................................................... 590 Choose an AWS CloudFormation template ......................................................................... 590 Add stack details ............................................................................................................ 591 Run the Template ........................................................................................................... 594 Updating the stream poller .............................................................................................. 594 Streams for disaster recovery ................................................................................................... 594 Replication setup............................................................................................................ 595 Other considerations....................................................................................................... 598 Neptune full text search .................................................................................................................. 599 Using.................................................................................................................................... 599 Use of Lucene syntax .............................................................................................................. 600 Full-text search setup ............................................................................................................. 600 Set up Full-Text Search in Your Region .............................................................................. 601 Add stack details ............................................................................................................ 602 ix Amazon Neptune User Guide Run the Template ........................................................................................................... 604 Updating the stream poller .............................................................................................. 605 Two approaches to enabling full text search ...................................................................... 605 Pausing the stream poller ................................................................................................ 606 Custom OpenSearch fields ............................................................................................... 606 Neptune Full-text search data model ........................................................................................ 606 SPARQL sample document ............................................................................................... 607 Gremlin sample document ............................................................................................... 608 Full-text search parameters ..................................................................................................... 609 Non-string indexing................................................................................................................ 612 Updating an existing stack............................................................................................... 613 Excluding fields.............................................................................................................. 614 Datatype mappings......................................................................................................... 616 Datatype validation........................................................................................................ 617 Sample queries.............................................................................................................. 621 Full-text-search query execution ............................................................................................... 623 Sample SPARQL full-text search queries .................................................................................... 624 match Query.................................................................................................................. 624 prefix............................................................................................................................ 624 fuzzy............................................................................................................................. 624 term............................................................................................................................. 625 query_string ................................................................................................................... 625 simple_query_string ........................................................................................................ 625 sort by string field .......................................................................................................... 625 sort by non-string field ................................................................................................... 625 sort by ID ...................................................................................................................... 626 sort by label .................................................................................................................. 626 sort by doc_type ............................................................................................................ 627 Lucene syntax................................................................................................................ 627 Sample Gremlin full-text search queries .................................................................................... 627 Basic match................................................................................................................... 628 match........................................................................................................................... 628 fuzzy........................................................................................................................... 628 query_string fuzzy..................................................................................................... 628 query_string regex..................................................................................................... 629 Hybrid query.................................................................................................................. 629 Full-text search example ................................................................................................. 629 query_string, '+' and '-' ............................................................................................... 630 query_string, AND and OR ............................................................................................ 631 term ............................................................................................................................. 631 prefix ......................................................................................................................... 631 Lucene syntax................................................................................................................ 631 Modern TinkerPop graph ................................................................................................. 632 Sort by string field ......................................................................................................... 633 Sort by non-string field ................................................................................................... 633 Sort by ID field .............................................................................................................. 633 Sort by label field .......................................................................................................... 633 Sort by document_type field......................................................................................... 633 Troubleshooting and metrics .................................................................................................... 634 Reads troubleshooting.................................................................................................... 634 Writes troubleshooting .................................................................................................... 634 Continuous ExpiredStreamException errors.................................................................. 635 AWS Lambda functions................................................................................................................... 636 Gremlin WebSocket connections ............................................................................................... 636 Gremlin Lambda recommendations ........................................................................................... 637 Write-request recommendations....................................................................................... 637 Read-request recommendations....................................................................................... 638 x

Description:
With Amazon Neptune, you can use relationships to process financial and purchase transactions in near real time to easily detect fraud patterns. Neptune provides a fully managed service to execute fast graph queries to detect that a potential purchaser is using the same email address and credit car
ebook img

Amazon Neptune - User Guide PDF

371 Pages·2017·2.37 MB·English
by  
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Amazon Neptune - User Guide

Amazon Neptune User Guide Amazon Neptune User Guide Amazon Neptune: User Guide Copyright © 2022 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Neptune User Guide Table of Contents What is Neptune?............................................................................................................................... 1 Latest Updates.................................................................................................................................. 3 Getting started ................................................................................................................................ 29 What's a graph database? .......................................................................................................... 29 Why use graphs?...................................................................................................................... 30 Graph database applications...................................................................................................... 31 Graph query languages ............................................................................................................. 33 Query examples............................................................................................................... 33 Online Neptune course ............................................................................................................. 35 Digging deeper........................................................................................................................ 35 Use graph notebooks................................................................................................................ 36 Using Neptune workbench ................................................................................................. 36 Enabling CloudWatch logs ................................................................................................. 39 Local hosting................................................................................................................... 39 Workbench magics................................................................................................................... 40 Variable injection............................................................................................................. 41 Common query arguments ................................................................................................ 42 %seed............................................................................................................................. 43 %load............................................................................................................................. 43 %load_ids....................................................................................................................... 43 %load_status................................................................................................................... 43 %cancel_load................................................................................................................... 43 %status........................................................................................................................... 44 %gremlin_status.............................................................................................................. 44 %opencypher_status, or %oc_status ................................................................................... 44 %sparql_status................................................................................................................. 44 %stream_viewer............................................................................................................ 44 %graph_notebook_config.................................................................................................. 44 %graph_notebook_host.................................................................................................... 45 %graph_notebook_version................................................................................................ 45 %graph_notebook_vis_options........................................................................................... 45 %%graph_notebook_config............................................................................................... 45 %%sparql ........................................................................................................................ 46 %%gremlin..................................................................................................................... 46 %%opencypher, or %%oc ................................................................................................. 47 %%graph_notebook_vis_options........................................................................................ 47 %neptune_ml.................................................................................................................. 48 %%neptune_ml............................................................................................................... 50 Graph visualization................................................................................................................... 52 Graph interface................................................................................................................ 52 Gremlin visualization........................................................................................................ 54 SPARQL visualization........................................................................................................ 55 Visualization tutorials....................................................................................................... 56 Neptune setup................................................................................................................................. 57 DB instance types..................................................................................................................... 57 Instance resource allocation ............................................................................................... 57 t3 and t4g..................................................................................................................... 58 r4 instances.................................................................................................................... 58 r5 instances.................................................................................................................... 58 r5d instances.................................................................................................................. 59 r6g instances.................................................................................................................. 59 x2g instances.................................................................................................................. 59 serverless instances...................................................................................................... 59 Create a DB cluster ................................................................................................................... 60 iii Amazon Neptune User Guide Prerequisites.................................................................................................................... 61 Create the Cluster ............................................................................................................ 64 Configure the VPC .................................................................................................................... 65 Add subnets.................................................................................................................... 66 Create a subnet group ...................................................................................................... 66 Create a security group ..................................................................................................... 67 DNS in Your VPC .............................................................................................................. 68 Connect to your graph .............................................................................................................. 68 Set up curl or awscurl.................................................................................................. 68 Ways to connect .............................................................................................................. 68 From within the VPC ........................................................................................................ 69 From a different VPC ........................................................................................................ 70 From a private network ..................................................................................................... 70 Neptune Security...................................................................................................................... 71 IAM policies..................................................................................................................... 71 VPC security groups .......................................................................................................... 71 IAM authentication........................................................................................................... 71 Access the graph...................................................................................................................... 72 Setting up curl ............................................................................................................... 68 Query languages.............................................................................................................. 73 Use Gremlin..................................................................................................................... 73 Use openCypher............................................................................................................... 77 Use RDF/SPARQL.............................................................................................................. 77 Loading Data........................................................................................................................... 78 Monitoring Neptune................................................................................................................. 78 Troubleshooting and Best Practices ............................................................................................ 78 Global databases.............................................................................................................................. 79 Overview................................................................................................................................. 79 Advantages.............................................................................................................................. 80 Limitations.............................................................................................................................. 80 Setup...................................................................................................................................... 81 Configuration requirements............................................................................................... 81 Creating a global database ................................................................................................ 82 Use an existing DB cluster as primary ................................................................................. 83 Adding a secondary region ................................................................................................ 83 Connecting ...................................................................................................................... 84 Managing a Neptune global database......................................................................................... 84 Removing a cluster ........................................................................................................... 85 Deleting a global database................................................................................................ 85 Modifying a global database .............................................................................................. 85 Using failover.......................................................................................................................... 86 Detach and promote ......................................................................................................... 86 Managed planned failovers ................................................................................................ 87 Monitoring Neptune global databases......................................................................................... 88 Neptune overview............................................................................................................................ 90 Standards Compliance............................................................................................................... 91 Gremlin standards compliance ........................................................................................... 91 SPARQL standards compliance ......................................................................................... 100 openCypher standards compliance .................................................................................... 104 Graph Data Model.................................................................................................................. 113 Indexing Strategy........................................................................................................... 113 Gremlin data model ........................................................................................................ 115 Lookup cache......................................................................................................................... 116 Use cases for the lookup cache........................................................................................ 116 Using the cache.............................................................................................................. 117 Transaction Semantics............................................................................................................. 118 Isolation Levels.............................................................................................................. 118 iv Amazon Neptune User Guide Neptune Isolation Levels ................................................................................................. 119 Transaction examples...................................................................................................... 123 Exceptions and Retries .................................................................................................... 125 Clusters and Instances ............................................................................................................. 127 The primary DB instance ................................................................................................. 127 Read-replica instances..................................................................................................... 127 Sizing instances.............................................................................................................. 128 Monitoring instances....................................................................................................... 128 Storage, reliability and availability ............................................................................................ 129 Allocation...................................................................................................................... 129 Storage billing............................................................................................................... 129 Storage best practices ..................................................................................................... 130 Reliability and high availability ......................................................................................... 130 Endpoint Connections............................................................................................................. 132 Endpoint considerations.................................................................................................. 134 Custom endpoints.......................................................................................................... 134 Create custom endpoints ................................................................................................. 135 View custom endpoints................................................................................................... 135 Edit custom endpoints .................................................................................................... 136 Delete custom endpoints................................................................................................. 136 Custom queryId...................................................................................................................... 137 Using the HTTP Header ................................................................................................... 137 Using a SPARQL Query Hint ............................................................................................. 137 Using queryId to Check Status ......................................................................................... 137 Lab Mode.............................................................................................................................. 139 Using Lab Mode............................................................................................................. 139 OSGP index................................................................................................................... 140 Transaction Semantics..................................................................................................... 140 Neptune DFE engine............................................................................................................... 141 Controlling DFE use ........................................................................................................ 141 Queries executed by the DFE ........................................................................................... 142 DFE statistics................................................................................................................. 142 JDBC connectivity................................................................................................................... 149 Getting started............................................................................................................... 149 Using Tableau ................................................................................................................ 150 Troubleshooting............................................................................................................. 152 Neptune engine updates......................................................................................................... 153 Security......................................................................................................................................... 154 Data Protection...................................................................................................................... 154 Amazon VPC protection .................................................................................................. 155 Encryption in Transit ....................................................................................................... 155 Encryption at Rest .......................................................................................................... 156 IAM overview......................................................................................................................... 159 Different roles................................................................................................................ 159 Using Identities.............................................................................................................. 160 Enabling IAM......................................................................................................................... 162 Connecting and Signing .......................................................................................................... 162 EC2 Prerequisites............................................................................................................ 163 Gremlin Console............................................................................................................. 164 Gremlin Java.................................................................................................................. 167 SPARQL Java (RDF4J and Jena) ........................................................................................ 168 SPARQL with Node.js ...................................................................................................... 170 Python Example............................................................................................................. 171 Using IAM Policies .................................................................................................................. 178 Identity-Based Policies.................................................................................................... 179 Service Control Policies (SCP) ........................................................................................... 179 Neptune Console Access .................................................................................................. 179 v Amazon Neptune User Guide Attaching a Policy .......................................................................................................... 179 Types of IAM policies ...................................................................................................... 180 Using condition Keys ....................................................................................................... 180 IAM feature support ........................................................................................................ 181 IAM Policy Limitations ..................................................................................................... 181 Managed policies............................................................................................................ 181 Condition Keys............................................................................................................... 189 Administrative policy statements ...................................................................................... 190 Data-access policy statements .......................................................................................... 207 Neptune Service-Linked Roles .................................................................................................. 219 Role Permissions............................................................................................................ 220 Creating a Service-Linked Role ......................................................................................... 221 Editing a Service-Linked Role ........................................................................................... 221 Deleting a Service-Linked Role ......................................................................................... 221 Temporary Credentials............................................................................................................ 223 Get Credentials with the AWS CLI ..................................................................................... 223 Setting Up Lambda ......................................................................................................... 225 Set Up Amazon EC2 ........................................................................................................ 226 Logging and Monitoring.......................................................................................................... 228 Compliance Validation............................................................................................................. 228 Resilience.............................................................................................................................. 229 Migrating to Neptune..................................................................................................................... 230 Migrating from Neo4j ............................................................................................................. 231 General information........................................................................................................ 231 Preparing for migration ................................................................................................... 233 Provisioning infrastructure............................................................................................... 237 Data migration............................................................................................................... 239 Application migration...................................................................................................... 243 Neptune compatibility..................................................................................................... 246 Cypher rewrites.............................................................................................................. 249 Migration resources......................................................................................................... 254 Migrating from TinkerPop ........................................................................................................ 255 Migrating from RDF ................................................................................................................ 256 Using AWS DMS to migrate ..................................................................................................... 257 Migrating from Blazegraph ...................................................................................................... 258 Neptune compatibility..................................................................................................... 258 Provisioning infrastructure............................................................................................... 258 Exporting data............................................................................................................... 259 Create an Amazon S3 bucket ........................................................................................... 261 Import the data ............................................................................................................. 261 Loading data.................................................................................................................................. 263 Neptune Bulk Loader.............................................................................................................. 263 IAM Role and Amazon S3 Access ...................................................................................... 264 Data Formats................................................................................................................. 267 Loading Example............................................................................................................ 277 Optimizing a bulk load.................................................................................................... 280 Loader Reference............................................................................................................ 281 Load data using DMS.............................................................................................................. 300 GraphMappingConfig...................................................................................................... 300 Replicating to Neptune ................................................................................................... 303 Querying ....................................................................................................................................... 307 Query queuing....................................................................................................................... 307 Finding how many queries are in the queue ....................................................................... 307 Query timeouts.............................................................................................................. 308 Gremlin................................................................................................................................. 308 Installing the Gremlin console .......................................................................................... 309 HTTPS REST.................................................................................................................. 313 vi Amazon Neptune User Guide Java.............................................................................................................................. 314 Python.......................................................................................................................... 322 .NET.............................................................................................................................. 324 Node.js.......................................................................................................................... 325 Query hints................................................................................................................... 327 Query status.................................................................................................................. 333 Query cancellation.......................................................................................................... 334 Gremlin script-based sessions ........................................................................................... 334 Gremlin transactions....................................................................................................... 335 Using the Gremlin API ..................................................................................................... 337 Caching query results ...................................................................................................... 337 Efficient upserts............................................................................................................. 342 Gremlin explain........................................................................................................... 351 Gremlin and DFE ............................................................................................................ 387 openCypher........................................................................................................................... 388 Gremlin vs. openCypher .................................................................................................. 388 Using openCypher.......................................................................................................... 389 Status endpoint.............................................................................................................. 390 HTTPS endpoint............................................................................................................. 392 Using the Bolt protocol ................................................................................................... 395 Parameterized examples.................................................................................................. 407 Data model.................................................................................................................... 408 openCypher explain..................................................................................................... 409 Transactions................................................................................................................... 420 Restrictions.................................................................................................................... 426 Exceptions..................................................................................................................... 426 SPARQL................................................................................................................................. 428 RDF4J Console............................................................................................................... 429 RDF4J Workbench.......................................................................................................... 431 Java.............................................................................................................................. 432 HTTP API....................................................................................................................... 434 Query hints................................................................................................................... 444 Query status.................................................................................................................. 455 Query cancellation.......................................................................................................... 457 Graph store protocol ....................................................................................................... 458 SPARQL explain ........................................................................................................... 459 SPARQL SERVICE Extension............................................................................................ 481 Visualization tools.......................................................................................................................... 483 Tom Sawyer Software ............................................................................................................. 483 Cambridge Intelligence............................................................................................................ 485 Graphistry.............................................................................................................................. 487 metaphacts............................................................................................................................ 488 G.V( )..................................................................................................................................... 491 Exporting data............................................................................................................................... 492 neptune-export...................................................................................................................... 493 Neptune-Export service........................................................................................................... 494 Install the service ........................................................................................................... 494 Enable access to Neptune ................................................................................................ 496 Enable access to Neptune-Export ..................................................................................... 496 Run an export job .......................................................................................................... 496 Monitor the job.............................................................................................................. 497 Cancel a job ................................................................................................................... 498 neptune-export utility............................................................................................................. 499 Prerequisites.................................................................................................................. 499 Running neptune-export................................................................................................. 500 Example commands........................................................................................................ 500 Exported files......................................................................................................................... 502 vii Amazon Neptune User Guide Export parameters.................................................................................................................. 503 command ...................................................................................................................... 504 outputS3Path................................................................................................................. 504 jobSize.......................................................................................................................... 504 params.......................................................................................................................... 505 additionalParams............................................................................................................ 505 params.......................................................................................................................... 506 Filtering examples.......................................................................................................... 514 Troubleshooting ..................................................................................................................... 518 Managing Neptune......................................................................................................................... 521 IAM user permissions.............................................................................................................. 522 Service-linked role policy ................................................................................................. 522 Create a new IAM user .................................................................................................... 522 Parameter groups................................................................................................................... 524 Editing a Parameter Group .............................................................................................. 525 Creating a parameter group ............................................................................................. 525 Parameters............................................................................................................................ 526 neptune_enable_audit_log............................................................................................... 526 neptune_lab_mode......................................................................................................... 526 neptune_query_timeout.................................................................................................. 526 neptune_streams............................................................................................................ 527 neptune_streams_expiry_days.......................................................................................... 527 neptune_lookup_cache.................................................................................................... 527 neptune_autoscaling_config ............................................................................................. 527 neptune_ml_iam_role...................................................................................................... 528 neptune_ml_endpoint..................................................................................................... 528 neptune_dfe_query_engine.............................................................................................. 528 neptune_query_timeout.................................................................................................. 528 neptune_result_cache...................................................................................................... 529 neptune_enforce_ssl ....................................................................................................... 529 Launch using the console ........................................................................................................ 530 Stopping and starting a cluster ................................................................................................ 534 Stopping and starting overview ........................................................................................ 534 Stopping a cluster.......................................................................................................... 534 Starting a DB cluster ....................................................................................................... 535 Fast reset API ......................................................................................................................... 537 Using IAM-Auth.............................................................................................................. 539 The %db_reset magic ..................................................................................................... 539 Common errors.............................................................................................................. 540 Creating a Replica .................................................................................................................. 541 Modifying a DB Cluster ........................................................................................................... 542 Modify an Instance ......................................................................................................... 542 Adding Neptune Replicas ......................................................................................................... 544 Performance and Scaling ......................................................................................................... 546 Storage Scaling.............................................................................................................. 546 Instance Scaling;............................................................................................................. 546 Read Scaling .................................................................................................................. 546 Auto-scaling........................................................................................................................... 547 Enabling auto-scaling...................................................................................................... 548 Remove auto-scaling ....................................................................................................... 550 Maintaining a DB Cluster......................................................................................................... 551 Maintenance Window...................................................................................................... 551 Neptune Updates................................................................................................................... 553 Manually Upgrading........................................................................................................ 553 Automatic Upgrading...................................................................................................... 553 Cloning a DB Cluster............................................................................................................... 555 Limitations..................................................................................................................... 556 viii Amazon Neptune User Guide Copy-on-Write Protocol ................................................................................................... 556 Deleting a Source Database ............................................................................................. 558 Managing Instances................................................................................................................ 559 T3 Burstable Instances .................................................................................................... 560 Modifying an Instance ..................................................................................................... 562 Renaming a Neptune DB Instance ..................................................................................... 565 Rebooting a DB instance ................................................................................................. 566 Deleting a DB Instance .................................................................................................... 567 Serverless...................................................................................................................................... 569 Serverless use cases ................................................................................................................ 569 Constraints............................................................................................................................ 569 Capacity scaling ..................................................................................................................... 570 Setting the minimum ...................................................................................................... 571 Setting the maximum ..................................................................................................... 571 Estimate capacity settings ............................................................................................... 572 Additional configuration.......................................................................................................... 573 Mixed configuration........................................................................................................ 573 Setting promotion tiers ................................................................................................... 573 Reader-writer alignment.................................................................................................. 574 Avoid very large timeout values ....................................................................................... 574 Optimizing your configuration .......................................................................................... 574 Using Serverless..................................................................................................................... 575 Creating a serverless cluster ............................................................................................. 575 Converting to Serverless .................................................................................................. 575 Modifying the capacity range ........................................................................................... 576 Converting to provisioned ............................................................................................... 576 Monitoring..................................................................................................................... 576 Neptune streams............................................................................................................................ 578 Using Streams........................................................................................................................ 579 Enabling Streams........................................................................................................... 579 Disabling Streams........................................................................................................... 580 Calling the Streams API ................................................................................................... 580 Streams Response........................................................................................................... 581 Streams Exceptions......................................................................................................... 583 Streams Record Formats .......................................................................................................... 583 PG_JSON....................................................................................................................... 584 RDF-NQUADS ................................................................................................................. 586 Streams Examples.................................................................................................................. 586 AT_SEQUENCE_NUMBER Examples................................................................................... 586 AFTER_SEQUENCE_NUMBER Example............................................................................... 587 TRIM_HORIZON Example................................................................................................. 588 LATEST Example............................................................................................................. 588 Compression Example..................................................................................................... 589 Neptune-to-Neptune Replication Setup ..................................................................................... 590 Choose an AWS CloudFormation template ......................................................................... 590 Add stack details ............................................................................................................ 591 Run the Template ........................................................................................................... 594 Updating the stream poller .............................................................................................. 594 Streams for disaster recovery ................................................................................................... 594 Replication setup............................................................................................................ 595 Other considerations....................................................................................................... 598 Neptune full text search .................................................................................................................. 599 Using.................................................................................................................................... 599 Use of Lucene syntax .............................................................................................................. 600 Full-text search setup ............................................................................................................. 600 Set up Full-Text Search in Your Region .............................................................................. 601 Add stack details ............................................................................................................ 602 ix Amazon Neptune User Guide Run the Template ........................................................................................................... 604 Updating the stream poller .............................................................................................. 605 Two approaches to enabling full text search ...................................................................... 605 Pausing the stream poller ................................................................................................ 606 Custom OpenSearch fields ............................................................................................... 606 Neptune Full-text search data model ........................................................................................ 606 SPARQL sample document ............................................................................................... 607 Gremlin sample document ............................................................................................... 608 Full-text search parameters ..................................................................................................... 609 Non-string indexing................................................................................................................ 612 Updating an existing stack............................................................................................... 613 Excluding fields.............................................................................................................. 614 Datatype mappings......................................................................................................... 616 Datatype validation........................................................................................................ 617 Sample queries.............................................................................................................. 621 Full-text-search query execution ............................................................................................... 623 Sample SPARQL full-text search queries .................................................................................... 624 match Query.................................................................................................................. 624 prefix............................................................................................................................ 624 fuzzy............................................................................................................................. 624 term............................................................................................................................. 625 query_string ................................................................................................................... 625 simple_query_string ........................................................................................................ 625 sort by string field .......................................................................................................... 625 sort by non-string field ................................................................................................... 625 sort by ID ...................................................................................................................... 626 sort by label .................................................................................................................. 626 sort by doc_type ............................................................................................................ 627 Lucene syntax................................................................................................................ 627 Sample Gremlin full-text search queries .................................................................................... 627 Basic match................................................................................................................... 628 match........................................................................................................................... 628 fuzzy........................................................................................................................... 628 query_string fuzzy..................................................................................................... 628 query_string regex..................................................................................................... 629 Hybrid query.................................................................................................................. 629 Full-text search example ................................................................................................. 629 query_string, '+' and '-' ............................................................................................... 630 query_string, AND and OR ............................................................................................ 631 term ............................................................................................................................. 631 prefix ......................................................................................................................... 631 Lucene syntax................................................................................................................ 631 Modern TinkerPop graph ................................................................................................. 632 Sort by string field ......................................................................................................... 633 Sort by non-string field ................................................................................................... 633 Sort by ID field .............................................................................................................. 633 Sort by label field .......................................................................................................... 633 Sort by document_type field......................................................................................... 633 Troubleshooting and metrics .................................................................................................... 634 Reads troubleshooting.................................................................................................... 634 Writes troubleshooting .................................................................................................... 634 Continuous ExpiredStreamException errors.................................................................. 635 AWS Lambda functions................................................................................................................... 636 Gremlin WebSocket connections ............................................................................................... 636 Gremlin Lambda recommendations ........................................................................................... 637 Write-request recommendations....................................................................................... 637 Read-request recommendations....................................................................................... 638 x

Description:
With Amazon Neptune, you can use relationships to process financial and purchase transactions in near real time to easily detect fraud patterns. Neptune provides a fully managed service to execute fast graph queries to detect that a potential purchaser is using the same email address and credit car
See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users