Arvados

Wiki » Proposals »

Containers API » History » Revision 8

Revision 7 (Tom Clegg, 06/03/2015 02:04 PM) → Revision 8/64 (Tom Clegg, 06/04/2015 05:58 PM)

h1. Jobs API (DRAFT) 

 Clients control JobRequests. The system controls Jobs, and assigns them to JobRequests. When the system has assigned a Job to a JobRequest, anyone with permission to read the JobRequest also has permission to read the Job. 

 A JobRequest describes job _constraints_ which can have different interpretations over time. For example, a JobRequest with @{"git_revision":"abc123..master"}@ might be satisfiable by any of several different source trees, and this set of satisfying source trees changes whenever the repository's "master" branch is updated. 

 A Job is an unambiguously specified process. Git revisions, data collections, docker images, etc. are specified using content addresses. A Job serves as a statement of exactly _what computation will be attempted_ and, later, a record of _what computation was done_. 

 h2. JobRequest/Job life cycle 

 Illustrating job re-use and preview facility: 
 # Client CA creates a JobRequest JRA with priority=0. 
 # Server creates job JX and assigns JX to JRA, but does not try to run JX yet because max(priority)=0. 
 # Client CA presents JX to the user. "We haven't computed this result yet, so we'll have to run a new job. Is this OK?" 
 # Client CB creates a JobRequest JRB with priority=1. 
 # Server assigns JX to JRB and puts JX in the execution queue with priority=1. 
 # Client CA updates JRA with priority=2. 
 # Server updates JX with priority=2. 
 # Job JX starts. 
 # Client CA updates JRA with priority=0. (This is the "cancel" operation.) 
 # Server updates JX with priority=1. (JRB still wants this job to complete.) 
 # Job JX finishes. 
 # Clients CA and CB have permission to read JX (ever since JX was assigned to their respective JobRequests) as well as its progress indicators, output, and log. 

 h2. "JobRequest" schema 

 |Attribute|Type|Description|Discussion|Examples| 
 |uuid, owner_uuid, modified_by_client_uuid,    modified_by_user_uuid|string|Usual Arvados model attributes||| 
 | 
 |created_at, modified_at|datetime|Usual Arvados model attributes||| 
 | 
 |name|string|Unparsed||| 
 | 
 |description|text|Unparsed||| 
 | 
 |job_uuid|uuid|The job that satisfies this job request.| 
 Can be null if a suitable job has not yet been found or queued. 
 Assigned by the system: cannot be modified directly by clients. 
 If null, it can be changed by the system at any time. 
 If not null, it can be reset to null by a client _if priority is zero_.|| 
 | 
 |mounts|hash|Objects |input|hash|Hash of arbitrary keys and values.|Any collection UUID appearing here (as an array element or hash value) will be resolved to attach a PDH in order to the container's filesystem and stdin/stdout. 
 Keys starting with find or create a forward slash indicate objects mounted in the container's filesystem. Job record. 
 Other keys are given special meanings here.| 
 We use "stdin" instead of "/dev/stdin" because literally replacing /dev/stdin with It is an error to refer to a file would have a confusing effect on many unix programs. The stdin feature only affects collection here (by UUID or PDH) unless it exists and is readable by the standard input of the first process started in the container; after that, the usual rules apply.| 
 <pre>{ submitting user.|<pre>{ 
  "/input/foo":{ 
   "kind":"collection", 
   "portable_data_hash":"d41d8cd98f00b204e9800998ecf8427e+0" "foo":"d41d8cd98f00b204e9800998ecf8427e+0", 
  }, 
  "stdin":{ 
   "kind":"file", 
   "uuid":"zzzzz-4zz18-yyyyyyyyyyyyyyy", 
   "path":"/foo.txt" 
  }, 
  "stdout":{ 
   "kind":"output:object" 
  } "bar":123 
 }</pre>| 
 | 
 |runtime_permissions|hash|Restrict the job's access to the outside world (apart from its explicitly stated inputs and output). 
 Each key is the name of a capability, like "internet" or "API" or "clock". The corresponding value is @true@ (the capability must be available in the job's runtime environment) or @false@ (must not). If a key is omitted, availability of the corresponding capability is acceptable but not necessary.|This is a generalized version of "enforce purity restrictions": it is not a claim that the job will be pure. However, knowing which restrictions were in place can be helpful when reasoning about whether a given job was pure. 
 In the most basic implementation, no capabilities are defined, and the only acceptable value of this attribute is the empty hash. 
 (TC)This name isn't great, and conflicts with the "readable/writable" kind of permissions. Perhaps something along the lines of capabilities or interfaces?| 
 <pre>{}</pre>| 
 | 
 |git_trees|hash|Source code (or other) repositories and versions to check out before the job starts. 
 In the example, the "myrepo" tree will be checked out at @/tmp/foobar@ and the "arvados" tree will be checked out at the default location @{basedir}/tree/arvados@.|refs #3820 
 (TC)Generalize to something like "repository_checkouts"? 
 (TC)"name" vs "url" vs "git-url"? 
 (TC)Taken as a revision range, "master" includes its parents, which is counterintuitive. Should we require git's own syntax ("master^!" for a single commit, or use "revision_range" or "revisions" to specify ranges, or...?| 
 <pre> 
 { 
  "arvados":{ 
   "url":"https://github.com/curoverse/arvados.git", 
   "revision":"master^!" 
  }, 
  "/tmp/foobar":{ 
   "name":"myusername/myrepo", 
   "revision":"abcd123..master" 
  } 
 } 
 </pre>| 
 | 
 |docker_image|string|Docker image repository and tag, docker image hash, collection UUID, or collection PDH.||| 
 | 
 |temp_dir, output_dir, keep_dir|string|Desired paths *inside the docker container* where temporary directory, output directories and keep mount should go.| 
 (TC)What are the defaults? This flexibility seems useful for a job that submits other jobs (like a workflow/pipeline runner) but would be cumbersome to specify every time ("remind me, where does workflow runner X expect its keep mount to be?). 
 (TC)What is the significance of output_dir? [How] does Crunch merge the content of the @output_dir@ and the value of the @output@ attribute to arrive at the final output?|| 
 | 
 |stdin|string|A file in Keep that should sent to standard input. 
 Given as an absolute path (relative to the container filesystem root). 
 The process must not rely on stdin being a regular file (the system is not required to set up stdin so that it's seekable.) 
 This cannot be used to make additional inputs available to the process beyond those listed in the input hash.| 
 (TC)If given as a relative path, relative to where? 
 (TC)How does stdin refer to one of the inputs in the input hash? 
 (TC)If the job does not finish reading it, is that an error, like @set -o pipefail@ in bash?| 
 @/data/foo.txt@| 
 | 
 |stdout|string|A filename in the output directory to which standard output should be directed.|(TC)If this is not given, is stdout sent to stderr/logs as it is now? 
 (TC)Relationship between stdout and output is unclear. If I specify a "stdout" but the job process sets its output by itself, is Crunch expected to clobber that output with the collection resulting from the "stdout" mechanism?|| 
 | 
 |environment|hash|environment variables and values that should be set in the container environment (docker run --env). This augments and (when conflicts exists) overrides environment --env)| 
 (TC)If this contains variables given in already used by Crunch (TASK_KEEPMOUNT), which has precedence?|| 
 | 
 |initial_collection|uuid|A collection describing the image's Dockerfile.||| starting contents of the output directory.| 
 (TC)Not a fan of this attribute name. 
 (TC)Is it an error if this collection is not one of the inputs? Or do all provenance queries need to treat this separately? 
 (TC)Perhaps better if each @input@ item were available at @{job_workdir}/input/{inputkey}@ and the "preload" behavior could be achieved by setting @output_dir@ to @input/foo@?|| 
 | 
 |cwd|string|initial working directory, given as an absolute path (in the container) or a path relative to the WORKDIR given in the image's Dockerfile. The default is @"."@.||<pre>"/tmp"</pre>| {job_workdir}. Default "output".||/tmp 
 output 
 input/foo| 
 | 
 |command|array of strings|Command strings|parameters to execute in the container. Default is the CMD given in the image's Dockerfile.| actual executable command line.| 
 (TC)Possible to specify a pipe, like "echo foo &#124; tr f b"? Any shell variables supported? Or do you just use @["sh","-c","echo $PATH &#124; wc"]@ if you want a shell?|| 
 | 
 |runtime_debugging|boolean|Enable debug logging for the infrastructure (such as arv-mount) (this might get logged privately away from the end user)| 
 (TC)This doesn't sound like it should be a job attribute. Infrastructure debugging shouldn't require touching users' job records. An analogous user feature would be useful, but perhaps it just boils down to adding DEBUG=1 to @environment@?|| 
 | 
 |priority|number|Higher number means spend more resources (e.g., go ahead of other queued jobs, bring up more nodes). 
 Zero means a job should not be run. Clients are expected to submit JobRequests with zero priority in order to prevew the job that will be used to satisfy it.|(TC)Do we need something more subtle than a single number? 
 (TC)What if a high priority job is waiting for a low priority job to finish?|@0@, @1000.5@, @-1@| 


 h2. "Job" schema 

 |Attribute|Type|Description|Discussion|Examples| 
 |state, started_at, finished_at, log||Same as current job||| 
 | 
 |input, stdin, stdout, environment, initial_collection, cwd, command, runtime_debugging, git_checkout_dir, temp_dir, output_dir, keep_dir||Copied from the relevant JobRequest(s) and made available to the job process.||| process.| 
 || 
 | 
 |output|hash|Arbitrary hash provided by the job process.| 
 (PA)Changing the basic output type from a collection to a JSON object is important for native CWL support. 
 (TC)Need examples of how "output is one collection", "output is multiple collections", "output is collections plus other stuff(?)", and "output is other stuff without collections" are to be encoded.|| 
 | 
 |pure|boolean|The job's output is thought to be dependent solely on its inputs (i.e., it is expected to produce identical output if repeated)| 
 (TC)Is this merely an assertion by the submitter? Is the job itself expected to set or reset it? Does the system behave differently while running the job (e.g., different firewall rules, some APIs disabled)? [Under what conditions] is the system allowed to change it from true to false? Is null allowed, presumably signifying "not known"?|@null@ (?) 
 @true@ 
 @false@| 
 | 
 |git_commit_sha1|string|Full 40-character commit hash used to run the job.|(TC)Should we store the tree hash as well? Or _instead_ of the commit hash, if we prevent the job from seeing the git metadata, which would be good for reproducibility (consider a job that starts by doing "git checkout master" in its working directory). 
 (TC)Do we need to store git_repository here too? Presumably, the relevant git tree should be in the internal git repository as a prerequisite of Job creation. And if two repositories have the same commit/tree, it shouldn't matter which we pull it from when running the job.|| 
 |docker_image_pdh|string|Portable data hash of a collection containing the docker image used to run the job.|(TC) *If* docker image hashes can be verified efficiently, we can use the native docker image hash here instead of a collection PDH.|| 
 | 
 |progress|number|A number between 0.0 and 1.0 describing the fraction of work done.| 
 (TC)How does this relate to child tasks? E.g., is a job supposed to update this itself as its child tasks complete?|| 
 | 
 |priority|number|Highest priority of all associated JobRequests||| 

 h2. Mount types 

 The "mounts" hash is the primary mechanism for adding data to the container at runtime (beyond what is already in the container image). 

 Each value of the "mounts" hash is itself a hash, whose "kind" key determines the handler used to attach data to the container. 

 |Mount type|@kind@|Expected keys|Description|Examples|Discussion| 
 | 
 |Read-only collection|@collection@| 
 @"portable_data_hash"@ _or_ @"uuid"@ can be provided.| 
 At job startup, the target path will have the same directory structure as the given collection. Files in target path _may_ be read-only. Even if the files/directories are writable, modifications will _not_ be saved when the job ends.| 
 <pre> 
 { 
  "kind":"collection", 
  "uuid":"..." 
 } 
 </pre>| 
 | 
 |Read-only file|@file@| 
 Either @"portable_data_hash"@ or @"uuid"@ must be provided. 
 "path" must be provided, and must indicate a file in the given collection.| 
 Behavior is identical to "Read-only collection" except that the target is a single file.| 
 <pre> 
 { 
  "kind":"file", 
  "uuid":"..." 
 } 
 </pre>| 
 | 
 |Output collection (initially empty)|@output:collection@| 
 None| 
 At job startup, the target path will be empty. When the job ends, the content will be saved as the output of the job.| 
 <pre> 
 { 
  "kind":"output:collection" 
 } 
 </pre>| 
 (TC)Needs a "pre-populate with collection X" feature.| 
 | 
 |Output collection (initially empty)|@output:file@| 
 @"name"@| 
 This is usable only for the @"stdout"@ mount. The standard output of the container process will be written to a file in a new collection, and the resulting file (recorded as "{collection}/{path}") will be taken as the output of the job.| 
 <pre> 
 { 
  "kind":"output:file" 
 } 
 </pre>| 
 | 
 |Output object|@output:object@ (?)| 
 None| 
 This is usable only for the @"stdout"@ mount. The standard output of the container process will be parsed as JSON, and the resulting object will be taken as the output of the job.| 
 <pre> 
 { 
  "kind":"output:object" 
 } 
 </pre>| 
 | 
 |Git tree|@git_tree@| 
 One of {@"git-url"@, @"repository_name"@, @"uuid"@} must be provided. 
 One of {@"tree"@, @"commit"@, @"revisions"@} must be provided.| 
 At job startup, the target path will have the source tree indicated by the given revision. The @.git@ metadata directory _will not_ be available: typically the system will use @git-archive@ rather than @git-checkout@ to prepare the target directory. 
 If a value is given for @"revisions"@, it will be resolved to a set of commits (as desribed in the "ranges" section of git-revisions(1)) and the job request will be satisfiable by any commit in that set. 
 If a value is given for @"commit"@, it will be resolved to a single commit, and the tree resulting from that commit will be used. 
 If a value is given for @"tree"@, the given tree will be used. 
 Note that multiple commit hashes can resolve to the same tree hash: e.g., identical patch with different committer/author/timestamp.| 
 <pre> 
 { 
  "kind":"git_tree", 
  "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx", 
  "commit":"master" 
 } 

 { 
  "kind":"git_tree", 
  "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx", 
  "commit_range":"bugfix^..master" 
 } 

 { 
  "kind":"git_tree", 
  "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx", 
  "tree":"bugfix^..master" 
 } 
 </pre>| 
 (TC)Is the "tree" option here useful, or just unnecessary extra work? It seems to be the right way to test equivalence of two jobs, in any case.| 
 | 
 | 
 |Temporary directory|@tmp@| 
 None| 
 At job startup, the target path will be empty. When the job finishes, the content will be discarded. This will be backed by a memory-based filesystem where possible.| 
 <pre> 
 { 
  "kind":"tmp", 
 } 
 </pre>| 
 (TC)Should add a "max size" feature, to help memfs-backed implementations.| 
 | 


 h2. Permissions 

 Users own JobRequests but the system owns Jobs.    Users get permission to read Jobs by virtue of linked JobRequests. 

 h2. "jobs" API methods 

 Changes from *TODO: bring this section up to speed with distinct JobRequest and Job records.* 

 Reuse and reproducibility require some changes to the usual REST APIs: APIs. 

 h3. arvados.v1.job_requests.create and .update arvados.v1.jobs.create 

 These methods can fail when objects referenced in the "mounts" hash do not exist, Q: How does "find or the acting user has insufficient permission on them. create" work? 

 Q: How does a client submitting job B indicate it shouldn't run unless/until job A succeeds? 

 h3. arvados.v1.job_requests.update arvados.v1.jobs.update 

 The @job_uuid@ attribute is special: 
 * It Most attributes cannot be changed from null to non-null by after a regular client. job starts. Some attributes _can_ be changed: 
 * It cannot be changed from non-null to null by system processes. name, description, priority 
 * It _can_ be reset from non-null to null output, progress, state, finished_at, log (ideally only by the system _during a client-initiated update transaction that modifies attributes other than @state@ and @priority@._ 

 Apart from @job_uuid@, updates are restricted by the current @state@ of the job request. 
 * When @state="Preview"@, all attributes can itself - should this be updated. enforced?) 
 * When @state="Request"@, only @priority@ and @state@ can be updated. modified_* 
 * When @state="Done"@, no attributes can be updated. Q: (any more?) 

 @state@ cannot be null. The following state transitions are the only ones permitted. 
 * Preview &rarr; Request 
 * Preview &rarr; Done 
 * Request &rarr; Done 

 h3. arvados.v1.jobs.create and .update arvados.v1.jobs.get 

 These methods are not callable except Q: Should this omit mutable attributes when retrieved by system processes. 

 h3. arvados.v1.jobs.progress 

 This method permits specific types of updates while a job is running: update progress, record success/failure. 

 Q: [How] can a client submitting job B indicate it shouldn't run unless/until job A succeeds? 

 h2. Debugging 

 Q: Need any infrastructure debug-logging controls in this API? 

 Q: Need any job debug-logging controls in this API? Or just use environment vars? pure job? (Ideally, pure jobs should not be able to retrieve data other than their stated immutable / content-addressed inputs, either through Keep or through the API.) 

 h2. Scheduling and running jobs 

 Q: If two users submit identical pure jobs and ask to reuse existing jobs, whose token does the job get to use? 
 * Should pure jobs be run as a pseudo-user that is given read access to the relevant objects for the duration of the job? (This would make it safer to share jobs -- see #5823) 

 Q: If two users submit identical pure jobs with different priority, which priority is used? 
 * Choices include "whichever is greater" and "sum". 

 Q: If two users submit identical pure jobs and one cancels -- or one user submits two identical jobs and cancels one -- does the work stop, or continue? What do the job records look like after this?