Project

General

Profile

Introduction to Arvados » History » Version 5

Ward Vandewege, 04/08/2013 10:41 AM

1 1 Anonymous
h1. Introduction to Orvos
2
3
h2. Overview
4 2 Anonymous
5 4 Ward Vandewege
Orvos is a platform for storing, organizing, processing, and sharing genomic and other biomedical big data. The platform is designed to make it easier for bioinformaticians to develop analysis, developers to create web applications with genomic data and IT administers to manage large-scale compute and storage resources for genomic data. The platform is designed to run on top of "cloud operating systems" such as Amazon Web Services and OpenStack. Currently, there are implementations that work on AWS and Xen+Debian/Ubuntu.
6 1 Anonymous
7
h2. Project
8 2 Anonymous
9 3 Anonymous
The core technology has been under development at Harvard Medical School for many years. We are now in the process of refactoring original code, refactoring the APIs, and developing significant new capabilities. (See the [[Orvos Project History]] and the Backlog for more information on the roadmap.) 
10 1 Anonymous
11
h2. Why Orvos 
12 2 Anonymous
13 1 Anonymous
A set of relatively low-level compute and data management functions are consistent across a wide range of analysis pipelines and applications that are being built for genomic data. Unfortunately, every organization working with these data have been forced to build their own custom systems for these low level functions. At the same time, there are proprietary platforms emerging that seek to solve these same problems. Orvos was created to provide and common solution that could be used across a wide range of applications and would be free and open source. 
14
15
h2. Benefits
16 2 Anonymous
17 1 Anonymous
The Orvos platform seeks to solve a set of common problems that face informaticians and IT Organizations: 
18
19
Benefits to informaticians: 
20
* Make authoring analyses and constructing pipelines in any language as efficient as possible
21
* Provide an environment that can run open source and commercial tools (e.g. Galaxy, GATK, etc.) 
22
* Enable deep provenance and reproducability across all pipelines 
23
* Provide a way to flexibly organize data and ensure data integrity 
24
* Make queries of variant and other compact genome data very high-performance
25
* Create a simple way to run distributed batch processing jobs 
26
* Enable the secure sharing of data sets from small to very large 
27
* Provide a set of common APIs that enable application and pipeline portability across systems
28
* Offer a reference environment for implementation of standards 
29
* Standardize file format translation
30
31
Benefits to IT organizations: 
32
* Low total cost of ownership 
33
* Eliminate unnecessary data duplication 
34
* Ability to create private, on-premise clouds
35
* Self-service provision of resources
36
* Ability to utilize low-cost off the shelf hardware
37
* Easy-to-manage horizontally scaling architecture
38
* Straight-forward browser-based administration
39
* Provide facilities for hybrid (public and private) clouds 
40
* Ensure full compliance with security and regulatory standards
41
* Support data sets from 10s of TB to exabytes
42
43
h2. Functional Capabilities 
44 2 Anonymous
45 5 Ward Vandewege
Functionally, Orvos has two major sets of capabilities: (a) data management and (b) compute management.
46 1 Anonymous
47
h3. Data Management
48 2 Anonymous
49 1 Anonymous
The data manage services are designed to handle all of the challenges associated with storing and organizing large omic data sets. The heart of theses services is the Data Manager, which brokers data storage. The data management system is designed to handle the following needs: 
50
51 5 Ward Vandewege
* Store files (e.g. BAM, FASTQ, VCF, etc.) in a reliable way
52 1 Anonymous
* Store metadata about files for a wide variety of organizational schema
53
* Create collections (sets of files) that can be used in analyses 
54
* Ensure files are not unnecessary duplicated
55
* Maintain provenance on files (be able to identify their origin) 
56
* Secure access to files 
57
* Translate files between formats 
58
* Make it easy to access files 
59
* Leverage large arrays of distributed commodity drives for reliable storage
60
61
h3. Compute Management 
62 2 Anonymous
63 1 Anonymous
The compute management services are designed to handle the challenges associated with creating and running pipelines as large scale distributed processing jobs. 
64
65
* Enable a common way to represent pipelines (JSON) 
66
* Support the use of any pipeline creation tool 
67
* Keep all pipeline code in a GIT repository 
68
* Run pipelines as distributed computations using MapReduce
69
* Easily retrieve and store data from pipelines in the data management system 
70
* Store a record of every pipeline that is run 
71
* Eliminate the need to re-run pipeline jobs that have already been run
72
* Make it possible to easily and reliably re-run and verify any past pipeline 
73
* Create a straightforward way to author web applications that use underlying data and pipelines 
74
* Enable easy sharing of pipelines and applications between systems 
75
* Be able to run distributed computations across clusters in different data centers to access very large data sets 
76
77
The compute management system also includes a sub-component for doing tertiary analysis. This component has not been built yet, but we envision that it will provide an in-memory database for very high-performance queries of a compact representation of a genome that includes variants and other relevant data needed for tertiary analysis. (Learn more about the [[compact genome]] here.) 
78 3 Anonymous
79
h2. Virtual Machines 
80
81
Orvos envisions and environment where informaticians receive access to virtual machines (VMs) on a private or public cloud. Using this approach eliminates the need for managing separate physical servers for different projects significantly increasing the utilization of underlying hardware resources. It also gives informaticians a great deal of freedom in choosing everything from their operating system to their tools. With virtual machines, each informatician or project team can have full isolation, security, and privacy for their work. 
82
83
The Orvos platform provides shared common services that can be used from within a virtual machine. All of the Orvos services are accessible through APIs. 
84 1 Anonymous
85
h2. APIs and SDKs 
86 2 Anonymous
87 1 Anonymous
Orvos is designed so all of the data management and compute management services can be accessed through a set of a consistent APIs and interfaces. Most of the functionality is represented in a set of REST APIs. Some components use native interfaces (e.g. GIT). Since most informaticians are unfamiliar with the REST syntax, their is a roadmap to develop SDKs for specific languages (Python, Perl, Ruby, R, and Java). There is also a command line interface to the system. 
88
89
h2. Dashboard 
90 2 Anonymous
91 1 Anonymous
In addition to the APIs we are building a browser-based UI for the representing key information in the system and to provide visual tools for common activities that are well represented through tools. 
92
93
h2. Related Articles 
94 2 Anonymous
95 1 Anonymous
[[Technical Architecture]]
96 4 Ward Vandewege
[[Key Components]]