Glossary of Arvados Terms » History » Version 1

Anonymous, 04/17/2013 02:45 PM

1 1 Anonymous
h1. Glossary of Arvados Terms 
2 1 Anonymous
3 1 Anonymous
*collection* - snapshot of a set of data files (file names and contents). Analogous to a directory tree, with the additional property that it reliably identifies the data itself, not just symbolic names.
4 1 Anonymous
 
5 1 Anonymous
*content address* - a cryptographic digest of a data blob that can be used to determine where the data is stored and to verify the file.
6 1 Anonymous
7 1 Anonymous
*data manager* - a component that assists Keep servers in enforcing site policies and monitors the state of the storage facility as a whole; we expect that the Data Manager will also broker backup and archival services and interfaces to other external data stores.
8 1 Anonymous
 
9 1 Anonymous
*job dispatcher* - invokes jobs from the queue, as compute nodes become available.
10 1 Anonymous
 
11 1 Anonymous
*job script* - a program written and organized to be implemented using MapReduce for distributed computation.
12 1 Anonymous
 
13 1 Anonymous
*job task* - a portion of a Job that can be computed asynchronously and independently of other Job Tasks; could also be called a “MapReduce task” or “MapReduce step”.
14 1 Anonymous
 
15 1 Anonymous
*job* - an execution of a MapReduce program (Job Script). Has specific inputs, outputs, start time, finish time, etc.
16 1 Anonymous
 
17 1 Anonymous
*Keep* - a content addressable distributed file system.
18 1 Anonymous
 
19 1 Anonymous
*manifest* - a plain-text encoding of a collection. Lists filenames, sizes, and data block locators (hashes). Reference: http://en.wikipedia.org/wiki/Manifest_file.
20 1 Anonymous
 
21 1 Anonymous
*MapReduce Engine* - supervises a job from start to finish: sets up execution environment, queues and executes tasks, monitors task/node failures, reports statistics.
22 1 Anonymous
 
23 1 Anonymous
*pipeline component* - a portion of a pipeline template (or instance) that describes criteria for selecting or submitting a job; can also refer to a specific Job (e.g., in a completed pipeline instance).
24 1 Anonymous
 
25 1 Anonymous
*pipeline instance* - the act or record of applying a pipeline template to a specific set of inputs; generally, a pipeline instance refers to the UUIDs of jobs that have been run to satisfy the pipeline components.
26 1 Anonymous
 
27 1 Anonymous
*pipeline manager* - looks up (and submits, as needed) jobs to satisfy pipeline components, monitors the jobs as they run, and handles dependencies (e.g., wait for job A to complete, and use its output as job B’s input).
28 1 Anonymous
 
29 1 Anonymous
*pipeline template* - a pattern that describes the relationships among the component Jobs: for example, the template specifies that job A's output is job B's input; a pipeline template is analogous to a Makefile.
30 1 Anonymous
 
31 1 Anonymous
*provenance* - the origin of data and the details of how a data set was computed (including processes and source data).
32 1 Anonymous
 
33 1 Anonymous
*Workbench* - the browser-based visual tools for users to interface with Arvados capabilities; Workbench mirrors many of the same functions accessible through the command line interface (CLI).