Cloud Operating Systems and Virtualization¶

Arvados is designed to run on a cloud OS. This is a new category of systems that is also sometimes called a cloud management platform. The cloud OS layer provides a number of key services that Arvados uses:

• Virtualization - Allows administrators to create and provision virtual machines from a pool of hardware resources.

• Networking - Connectivity among compute nodes, and between compute and storage resources.

• Self-service Provisioning - Allows end users to provision resources for themselves.

• Administration - Provides tools for monitoring, managing, and administering clusters.

• Block Storage - In some cloud OS, such as Amazon Web Services, Arvados uses ext4 filesystems on elastic block storage (EBS) volumes instead of physical disks as the backing store for the Keep storage system.

• User Management - Management of user accounts and permissions.

Public, Private, and Hybrid Clouds¶

When people hear the word "cloud," they usually think of public clouds such as Amazon Web Services (AWS) and Google Cloud Platform. However, there is an emerging trend, especially in the biomedical industry, to implement private clouds. A private cloud uses a similar architecture to a public cloud, but it runs on systems owned by an organization in a data center they control. In this sense, cloud computing should be thought of as an architecture rather than simply a hosted service.

A number of features characterize cloud architectures:

• Horizontally scaling hardware, usually with uniform nodes that combine compute and storage.

• Commodity hardware that significantly lowers costs for large scale computing projects.

• Virtualization, which makes it possible to dynamically allocate computing resources and to isolate applications and users.

• Distributed computing technologies that scale horizontally across equipment and are elastic, such as block storage, object storage, distributed file systems, and MapReduce.

• Low-latency access to storage, which is achieved by putting storage and compute on the same node and by distributing computations to cores near the storage. This capability is not achieved in all cloud architectures, but it is ideal for I/O-intensive tasks like alignment and variant calling.

A cloud architecture allows IT leaders to take clusters of commodity computing equipment and use them in a wide variety of ways.

For most informatics teams, the cloud architecture is a departure from the high-performance computing (HPC) model that they are most familiar with. It's a different configuration than the traditional combination of network attached storage (NAS) systems, storage area networks (SAN), and compute clusters with job queueing systems such as Sun Grid Engine.

A cloud architecture provides a variety of advantages for informatics data:

• Lower total cost of ownership

• Significantly more flexible use of computing resources

• Self-service provisioning of compute and storage resources

• Easier overall system administration

• Faster and more efficient scaling

A private cloud can be used in conjunction with other computing resources. For example, a private cloud can be used with a public cloud for burstable compute capacity and archival storage. A private cloud could also be integrated with existing NAS storage, using the NAS as a slower storage tier or the other storage systems for archiving. Hybrid approaches work well with the Keep content addressing architecture. No matter where data are stored, client software automatically verifies that it is the correct file by checking a cryptographic digest. A private cloud could also leverage a traditional HPC compute cluster for some jobs. (Currently this is not a high-priority use case for Arvados; development will depend on community demand.)

DIY Cloud OS¶

At the Personal Genome Project (PGP), where we are currently running two Arvados clusters, we have implemented our own cloud OS with a combination of Xen, Ubuntu, and a number of other open source components. (We will be documenting this configuration more thoroughly for organizations that want to replicate it.)

Cloud OS for Private Clouds¶

A number of cloud OS solutions have emerged for private clouds. The leading proprietary solution is VMware's vCloud Suite, but there is a lot of momentum around open source efforts as well. The leading open source efforts include OpenStack, Eucalyptus, and CloudStack.

We plan to integrate OpenStack as a cloud OS to more easily run Arvados in a private cloud environment. Some of the core OpenStack components, such as the Swift object file store, are not needed by Arvados: these subsystems are replaced by Arvados services that are optimized for biomedical data. However, Arvados will take advantage of the security, virtualization, and overall cluster management features provided by OpenStack.

Public Cloud OS¶

Public clouds like Amazon Web Services and Google Cloud Platform are currently much more widely adopted than private cloud OS solutions. These services offer similar capabilities and services under different names. Although public clouds are typically less appropriate for working with biomedical big data, they offer the important benefits of fast provisioning and dynamic scaling with no up-front commitment.

One of the key goals of Arvados is to provide a smooth transition of data and applications between heterogeneous public and private clouds, so informaticians can easily and safely make use of cloud platforms suitable for the type of work being done.