Technical Architecture » History » Version 54

Nancy Ouyang, 06/02/2015 11:38 PM

1 1 Anonymous
h1. Technical Architecture
2 1 Anonymous
3 49 Nancy Ouyang
h2. High-Level Overview
4 49 Nancy Ouyang
5 49 Nancy Ouyang
For more information, see the whitepaper at https://curoverse.com/capabilities/it-teams. 
6 49 Nancy Ouyang
7 54 Nancy Ouyang
  !https://arvados.org/attachments/download/626/technical%20architecture.jpg!
8 49 Nancy Ouyang
9 49 Nancy Ouyang
h2. Details
10 49 Nancy Ouyang
11 39 Ward Vandewege
!ArvadosTechnicalDiagramV16_website.png!
12 3 Anonymous
13 12 Ward Vandewege
The technical diagram above represents the basic architecture of Arvados. 
14 3 Anonymous
15 40 Ward Vandewege
At the base layer is a "cloud operating system." Arvados does not depend on any particular stack - it runs in the cloud and on bare metal.
16 40 Ward Vandewege
17 41 Ward Vandewege
The platform is currently deployed on Amazon Web Services (AWS) and on Google Compute Platform (GCP), as well as on bare metal. 
18 40 Ward Vandewege
19 43 Ward Vandewege
Stack-specific integration - for example for AWS or GCP - is kept to the bare minimum: specifically, the [[Node Manager]] component.
20 3 Anonymous
21 3 Anonymous
h2. Key Components
22 3 Anonymous
23 29 Anonymous
*[[Keep|Content Addressable Distributed File System ("Keep")]]* - Arvados stores files in Keep. Keep is a distributed file system that has been optimized for biomedical data files and write once read many (WORM) scenarios. Keep chunks files into 64MB data blocks and distributes them across physical drives or virtual volumes. Keep writes the data blocks to an underlying file system (e.g. Linux ext4). Keep is also a content addressable store (CAS). When a file is stored, a content address is created for each data block using a cryptographic digest of the contents of block. Then a manifest is created that identifies all of the blocks that make up the file. Each manifest has its own unique content address. This ensures every file can be accurately verified every time it is retrieved from the system. Keep also supports the creation of collections, which include multiple files, as a flexible way to define data sets without re-organizing data on disk. 
24 3 Anonymous
25 47 Ward Vandewege
*[[Computation and Pipeline Processing|Container Management Engine ("Crunch") & Pipeline Management]]* - Crunch manages distributed processing tasks across cores. Tasks are executed inside "Docker":http://docker.io containers. Crunch is designed to maintain data provenance and pipeline reproducibility. The system supports a flexible mechanism for defining and invoking pipelines that use common components such as GATK or custom components. It automatically tracks the data inputs and outputs through Keep, the code used for each job through the git repository, the execution environment through the Docker container that runs the job, and the job parameters through the metadata database. 
26 3 Anonymous
27 44 Ward Vandewege
*"In-Memory Compact Genome Database 'Lightning'":https://github.com/curoverse/lightning* - Lightning uses a scale-out, open source in-memory database to store genomic data in a compact genome format. VCF files are not suitable for efficient look-ups so we are developing a format to represent variants and other key data for tertiary analysis. Putting this in in a scale-out, in-memory database will make it possible to do very fast queries of these data. This part of the project is in the early development stage.
28 3 Anonymous
29 30 Anonymous
*[[REST API Server]]* - This component provides OAuth2-authenticated REST APIs to Arvados subsystems (metadata database, jobs, etc.) with the notable exception of Keep (which requires direct access to avoid network performance bottlenecks) and VMs and git (which use the SSH protocol and public key authentication).
30 19 Anonymous
31 3 Anonymous
*[[Workbench]]* - Workbench is a set of visual tools for using the underlying Arvados services from a web browser. This is especially helpful for querying and browsing data, visualizing provenance, and monitoring jobs and pipelines. Workbench has a modular architecture designed for seamless integration with other Arvados applications.
32 1 Anonymous
33 1 Anonymous
*[[SDKs|Command Line Interface]]* - The CLI tools provide convenient access to the Arvados API and services in the Arvados platform from the command line. 
34 1 Anonymous
35 29 Anonymous
*[[SDKs]]* - Arvados provides native language SDKs for Python, Perl, Ruby, R, and Java to make it easier to work with the REST APIs in common development environments. The SDKs also support the development of clients for Keep, Crunch and Lightning. (Some SDKs have not yet been implemented.)
36 35 Ward Vandewege
37 45 Ward Vandewege
*[[Data Manager]]* - Data Manager helps to orchestrate interactions with data storage. This includes managing rules about permissions, replication, archiving, etc.
38 41 Ward Vandewege
39 46 Ward Vandewege
*[[Node Manager]]* - Node manager manages compute resources in a cloud environment. It starts and stops compute nodes on demand. For a bare metal installation, the number of compute nodes tends to be static, which means Node Manager is not required. Node Manager currently supports AWS and GCP.
40 3 Anonymous
41 48 Ward Vandewege
*[[Keep Proxy]]* - Keep Proxy provides remote (non-LAN) authenticated access to Keep (read and write). It allows remote Keep clients to upload one copy of a block, and takes care of storing the desired number of replicated copies in Keep. It is API compatible with Keep.
42 48 Ward Vandewege
43 38 Ward Vandewege
*"Documentation":http://doc.arvados.org* - This is the official documentation, which is also included in the Arvados source tree. There is also documentation developer information on the [[Documentation project]] wiki page.