ISB Cancer Genomics Cloud Documentation

234 Pages·2017·5.11 MB·English

ISB Cancer Genomics Cloud
Documentation
Release 1.0.0
the ISB-CGC team
May 29, 2020
GETTING STARTED
1 About the ISB-CGC Platform 3
2 ISB-CGC Data Overview 5
3 Quick-Start Guide 7
4 Getting Started with Analysis 9
5 How to Request Cloud Credits 13
6 Best Practices 15
7 Benefits of Using The Cloud 17
8 Mitelman Database 19
9 Programs and Data Sets 21
10 Data Access and Security Overview 47
11 Accessing Controlled Data 49
12 ISB-CGC BigQuery Tables 63
13 ISB-CGC BigQuery Table Search 85
14 ISB-CGC Data Explorer 93
15 ISB-CGC Web Interface (Web App) 95
16 ISB-CGC APIs 145
17 ISB-CGC Notebooks 149
18 Statistical Notebooks 151
19 Tutorials and How-To Guides 153
20 Release Notes 323
i
21 Quick Links 377
22 Frequently Asked Questions (FAQ) 381
23 Contact Us 389
ii
ISB Cancer Genomics Cloud Documentation, Release 1.0.0
Democratizing access to cancer data in the cloud
Contained within this documentation are descriptions of ISB-CGC features along with guides and tips for exploring
data sets hosted on the Google Cloud Platform.
The ISB-CGC aims to serve the needs of a broad range of cancer researchers ranging from scientists or clinicians
who prefer to use an interactive web-based application to access and explore the rich TCGA, TARGET, CCLE, and
COSMIC datasets, to computational scientists who want to write their own custom scripts using languages such as
R or Python, accessing the data through APIs, and to algorithm developers who wish to spin up thousands of virtual
machines to analyze hundreds of terabytes of sequence data.
– the ISB-CGC team
GETTING STARTED 1
ISB Cancer Genomics Cloud Documentation, Release 1.0.0
2 GETTING STARTED
CHAPTER 1
About the ISB-CGC Platform
The ISB Cancer Genomics Cloud (ISB-CGC) is one of three National Cancer Institute (NCI) Cloud Resources tasked
with bringing cancer data and computation power together through cloud platforms. It is a collaboration between
the Institute for Systems Biology (ISB) and General Dynamics Information Technology Inc. (GDIT). Since starting in
2014 as part of NCI’s Cloud Pilot Resource initiative, ISB-CGC has provided access to increasing amounts of cancer
data in the cloud.
1.1 Exploring Cancer Data
The ISB-CGC Platform enables a wide range of users to bring their analysis tools to the data in the cloud, eliminating
the need to download and store large data sets. Built with the Google Cloud Platform, it provides several entry points
for exploring and analyzing cancer data:
• The ISB-CGC Web Application allows users to interactively create and explore cohorts of interest.
• The ISB-CGC API gives users the ability to programmatically work with data such as cases, samples, cohorts,
files and cloud projects.
• The ISB-CGC BigQuery Table Search is a discovery tool that allows the user to explore and search for ISB-
CGC Google BiqQuery tables.
• On the Google Cloud Platform BigQuery Console, ISB-CGC tables can be viewed and queried directly.
• Python and R can interface with the ISB-CGC tables, retrieving and analyzing data.
• Using Google Compute Engines and VMs, workflows can be run to perform data analysis.
Please see the USER GUIDE section to learn more about each of these tools and the MORE INFORMATION section
to see examples, tutorials, Jupyter and R Notebooks, Frequently Asked Questions and more.
3
ISB Cancer Genomics Cloud Documentation, Release 1.0.0
Have feedback or corrections? Please email us at
CHAPTER 2
ISB-CGC Data Overview
ISB-CGC provides access to data from several research programs, such as The Cancer Genome Atlas (TCGA),
Therapeutically Applicable Research to Generate Effective Treatments (TARGET), Cancer Cell Line Encyclo-
pedia (CCLE) and Catalogue of Somatic Mutations in Cancer (COSMIC). The full list is available here.
The majority of the data made available through ISB-CGC originates from NCI Genomic Data Commons (GDC).
Users can access GDC data on the cloud through ISB-CGC. Users have access to both raw and processed data from
cancer patients.
In general, almost all raw data is controlled-access and is accessible through Google Cloud Storage buckets; only those
users with proper authorization can access them. The GDC has established bioinformatics workflows/pipelines exe-
cuted on the raw data to generate processed data. In this way, users can directly access the processed data without hav-
ing to run compute-intensive workflows themselves. However, users who wish to run their own workflows/pipelines
still have access to the raw data as well.
GDC processed data, however, are generally open-access. ISB-CGC allows users to utilize this processed data in two
ways on the platform:
• Google Cloud Storage: All individual processed data files are accessible through GDC Google Cloud Storage
buckets; ISB-CGC provides pointers to these files.
• Google BigQuery: Processed data are consolidated by datatype (ex. Clinical, DNA Methylation, RNAseq,
Somatic Mutation, etc.) and transformed into ISB-CGC Google BigQuery tables for ease of access and analysis.
This novel approach allows our users to quickly analyze information from thousands of patients in our curated
BigQuery tables.
5
ISB Cancer Genomics Cloud Documentation, Release 1.0.0
2.1 Google Cloud Storage
Google Cloud Storage (GCS) is a cloud-based object-store that is used to store many types of (usually binary) data,
typically processed by custom software pipelines. The data hosted by GDC is contained within Google Cloud Storage.
Metadata stored within ISB-CGC BigQuery tables contains pointers to file locations in this GDC data.
2.2 Google BigQuery
Google BigQuery (BQ) is a columnar database ideal for storing tabular data. Its query speed is automatically scaled
by multiprocessing. Data is accessed using a powerful SQL language interface.
ISB-CGC stores high-level clinical, biospecimen, and molecular data from the main NCI programs in the BigQuery
project isb-cgc. It also stores a large amount of metadata about files that are stored in the GDC Google Cloud Storage,
as well as genome reference sources (e.g. GENCODE, miRBase, etc.). All of these data sets and tables are completely
open access and available to the research community.
Have feedback or corrections? Please email us at