Glossary of Arvados Terms¶

collection - snapshot of a set of data files (file names and contents). Analogous to a directory tree, with the additional property that it reliably identifies the data itself, not just symbolic names.

content address - a cryptographic digest of a data blob that can be used to determine where the data is stored and to verify the file.

data manager - a component that assists Keep servers in enforcing site policies and monitors the state of the storage facility as a whole; we expect that the Data Manager will also broker backup and archival services and interfaces to other external data stores.

job dispatcher - invokes jobs from the queue, as compute nodes become available.

job script - a program written and organized to be implemented using MapReduce for distributed computation.

job task - a portion of a Job that can be computed asynchronously and independently of other Job Tasks; could also be called a “MapReduce task” or “MapReduce step”.

job - an execution of a MapReduce program (Job Script). Has specific inputs, outputs, start time, finish time, etc.

Keep - a content addressable distributed file system.

manifest - a plain-text encoding of a collection. Lists filenames, sizes, and data block locators (hashes). Reference: http://en.wikipedia.org/wiki/Manifest_file.

MapReduce Engine - supervises a job from start to finish: sets up execution environment, queues and executes tasks, monitors task/node failures, reports statistics.

pipeline component - a portion of a pipeline template (or instance) that describes criteria for selecting or submitting a job; can also refer to a specific Job (e.g., in a completed pipeline instance).

pipeline instance - the act or record of applying a pipeline template to a specific set of inputs; generally, a pipeline instance refers to the UUIDs of jobs that have been run to satisfy the pipeline components.

pipeline manager - looks up (and submits, as needed) jobs to satisfy pipeline components, monitors the jobs as they run, and handles dependencies (e.g., wait for job A to complete, and use its output as job B’s input).

pipeline template - a pattern that describes the relationships among the component Jobs: for example, the template specifies that job A's output is job B's input; a pipeline template is analogous to a Makefile.

provenance - the origin of data and the details of how a data set was computed (including processes and source data).

Workbench - the browser-based visual tools for users to interface with Arvados capabilities; Workbench mirrors many of the same functions accessible through the command line interface (CLI).