High-Performance In-Memory Genome Data Analysis: How In-Memory Database Technology Accelerates Personalized Medicine PDF
Preview High-Performance In-Memory Genome Data Analysis: How In-Memory Database Technology Accelerates Personalized Medicine
In-Memory Data Management Research Series Editor Prof. Dr. Dr. h.c. Hasso Plattner Hasso Plattner Institute Potsdam, Germany For further volumes: http://www.springer.com/series/11642 This book series presents selected research results in the context of In-Memory Data Management. The volumes in this series describe research results in in-mem- ory database technology, logical and physical data management, software architec- tures, real-time analysis of enterprise data, innovative new business applications, and infl uenced business processes. In addition, programming models and software engineering techniques, tools, and benchmarks are elaborated on and discussed. All books are introduced by a member of the editorial board, who outlines the popular context and the social relevance of each work. Globally, companies generate a steadily increasing amount of data, day after day. This data is obtained to optimize logistics, create knowledge, explore business rela- tionships, and to improve management decisions. The trend towards acquiring more and more data, also known as “big data,” requires fundamental support in data anal- ysis. In-Memory Technology – the common theme in all volumes of this series – has become a de facto standard for fulfi lling new requirements that are stated towards enterprise applications. Hasso Plattner • Matthieu-P. Schapranow Editors High-Performance In-Memory Genome Data Analysis How In-Memory Database Technology Accelerates Personalized Medicine 123 Editors Hasso Plattner Hasso Plattner Institute Enterprise Platform and Integration Concepts Potsdam, Germany Matthieu-P. Schapranow Hasso Plattner Institute Enterprise Platform and Integration Concepts Potsdam, Germany ISBN 978-3-319-03034-0 ISBN 978-3-319-03035-7 (eBook) DOI 10.1007/978-3-319-03035-7 Springer Cham Heidelberg New York Dordrecht London Libarary of Congress Control Number: 2013954438 c Springer International Publishing Switzerland 2014 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) Quotes An increased utility of sequencing data should follow from the ability to process hundreds of gigabytes of raw sequence data informatically prior to subsequent downstream analy- sis. Plattner and Schapranow share concrete details on how to accelerate data processing “ with in-memory database technology, and also highlight how to accelerate the analysis of sequencing data by leveraging relevant information. With their work they eliminate time-consuming enquiries for relevant data (from disk storage) and enable instant inter- pretation of fndings. This innovative approach should be of great value for applications ranging from research through to precision medicine. Scott Kahn, Illumina, CIO ” It will be essential to improve our understanding of the core functions of the human genome in order to develop stratifed treatments for complex diseases and to provide a foundation for treatments to prevent or delay onset of diseases. By applying advanced in- “ memory technology to concrete problems of personalized medicine, Plattner and Schapra- now demonstrate how interdisciplinary teams can develop innovative and appropriate so- lutions. Collaborative approaches of computational, scientifc, and clinical teams have an enormous potential to improve the way we provide medical treatments in the future. Fi- nally, the authors describe novel methods for fexible real-time analysis of medically rele- vant data that provide a powerful basis for timely decision making in personalized medical contexts. Prof. Dr. Peter N. Robinson, Charité, Head of the Computational Biology Group ” v vi Quotes At Cytolon, we provide IT services to identify the most appropriate cord blood sample to limit patient’s immune response. For this service, we need to analyze thousands of sam- ples and combine them with a variety of heterogeneous patient properties. Plattner and “ Schapranow present that in-memory technology provides a meaningful way to integrate heterogeneous data. In addition, they show that real-time analyses of patient data is a paradigm shift in today’s medicine. Thus, we believe this technology can help us to speed up the performance of our matching service. Thomas Klein, Cytolon AG, Founder and CEO ” At LGC Genomics, we build on our long-standing experience in providing DNA sequenc- ing and analysis services to our customers. Latest sequencing machines have sped up extraction of DNA reads, but analysis is still time-intensive due to the sheer amount of “ generated data. Plattner and Schapranow apply the innovative in-memory technology to challenging analyses with impressive results. Long-running analysis processing, e.g. co- hort analysis is reduced from taking up hours reduce to just a few seconds. We believe that this technology helps us to speed up our day-to-day business, allowing us to faster report back to our customer. Dr. Wolfgang Zimmermann, LGC Genomics, Business Unit Manager ” I am proud and thankful that HPI provides an environment that fosters teaching, research, and innovation in IT. Building on their former research results in database technology, Hasso Plattner and Matthieu Schapranow share insights of their high-performance in- “ memory genome platform that combines among others structured and unstructured med- ical data from various heterogeneous data sources to enable its real-time analyses in a sin- gle system. The platform is the outcome of a dedicated cooperation with various experts from biology, medicine, and computer science. As a result, it proves that interdisciplinary teams with actual knowledge from IT are able to considerably contribute in implementing the vision of great personalized medicine. Prof. Dr. Christoph Meinel, Hasso Plattner Institute, CEO ” Preface The human genome project was ofcially launched in 1990 equipped with a re- search funding ofmore than three billion USD. However, it tookmore than a decade and thousands of worldwide research institutes to discover and decode the full hu- man genome sequence. Nowadays, so-called next-generation sequencing devices process whole DNA and RNA within hours at moderate costs. Latest devices generate raw DNA reads with more than 30-times coverage in less than two days. However, interpretation and analysis of these raw data is still a time-consuming process potentially taking weeks. Next-generation sequencing devices are increasingly used in research and clinical environments to support treatment of specifc diseases, such as cancer. This example highlights how fast the technological developments currently afect our daily lives. Next-generation sequencing is also named to be the foundation for individual treatment decision, optimized therapies in course of personalized medicine and systems biology. Personalized medicine aims at treating patients specifcally based on individual dispositions, such as genetic or environmental factors. However, the increasing amount of gathered diagnostic data requires specifc software tools to identify relevant portions of data, process them at high-throughput, and provide ways to analyze them interactively. We wrote this book to provide details about innovative approaches to process, combine, and analyze data required in the course of personalized treatment. It con- tains latest research results of applying in-memory database technology to process and analyze big genomic data. Furthermore, we share how to design and develop specifc research tools that require real-time analysis of scientifc data. With this book, we contribute by bridging the gap between medical experts, such as physician, clinicians, and biological researchers, and technology experts, such as software developers, database specialists, and statisticians. As a result, we designed a specifc structure of the book to support the individual audiences. The book is structured as follows. vii viii Preface • Part I addresses the data acquisition, the modeling of processing and analysis pipelines, and how to accelerate preprocessing of data. This part is designed for bioinformaticians and researchers, who want to understand how to optimize the data preparation for their experiments. • Part II gives examples how to design and implement specifc applications en- abling real-time analysis of scientifc data. Furthermore, it provides guidelines to operate and to exchange huge data at fast pace. This part is intended for re- searchers and medical experts, who require to work with big data on a daily basis. It also provides guidelines for IT experts how to operate on these data from a software engineering perspective. Potsdam, Oct 20, 2013 Hasso Plattner and Matthieu-P. Schapranow Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 1 Innovations for Personalized Medicine . . . . . . . . . . . . . . . . . . . . . . . . . 1 Hasso Plattner, Matthieu-P. Schapranow and Franziska Häger 1.1 Requirements for Personalized Medicine . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Researchers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 Clinicians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.3 Patients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2 Interdisciplinary Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3 Trends in Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4 In-memory Technology Building Blocks . . . . . . . . . . . . . . . . . . . . . . . 13 1.4.1 Combined Column and Row Store . . . . . . . . . . . . . . . . . . . . 14 1.4.2 Complete History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.4.3 Lightweight Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.4.4 Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4.5 Multi-core and Parallelization . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4.6 Active and Passive Data Store . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4.7 Reduction of Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.5 High-performance In-memory Genome Platform. . . . . . . . . . . . . . . 17 1.5.1 Application Layer with Micro Applications . . . . . . . . . . . . 19 1.5.2 Platform Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.5.3 Data Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.6 Structure of the Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 ix
The list of books you might like
