Project

General

Profile

Actions
History

Wiki » Proposals »

Containers API » History » Revision 11

« Previous | Revision 11/64 (diff) | Next »
Tom Clegg, 06/09/2015 03:07 PM

Jobs API (DRAFT)

A Job is a record of a computational process.
  • Its goal is to capture, unambiguously, as much information as possible about the environment in which the process was run. For example, git trees, data collections, and docker images are stored as content addresses. This makes it possible to reason about the difference between two processes, and to replay a process at a different time and place.
  • Clients can read Job records, but only the system can create or modify them.
A JobRequest is a client's expression of interest in knowing the outcome of a computational process.
  • Typically, in this context the client's description of the process is less precise than a Job: a JobRequest describes job constraints which can have different interpretations over time. For example, a JobRequest with a {"kind":"git_tree","commit_range":"abc123..master",...} mount might be satisfiable by any of several different source trees, and this set of satisfying source trees can change when the repository's "master" branch is updated.
  • The system is responsible for finding suitable Jobs and assigning them to JobRequests. (Currently this is expected to be done synchronously during the jobRequests.create and jobRequests.update transactions.)
  • A JobRequest may indicate that it can only be satisfied by a new Job record (i.e., existing results should not be reused). In this case creating a JobRequest amounts to a submission to the job queue. This is appropriate when the purpose of the JobRequest is to test whether a process is repeatable.
  • A JobRequest may indicate that it cannot be satisfied by a new Job record. This is an appropriate way to test whether a result is already available.

When the system has assigned a Job to a JobRequest, anyone with permission to read the JobRequest also has permission to read the Job.

Use cases

Preview

Tell me how you would satisfy job request X. Which pdh/commits would be used? Is the satisfying job already started? finished?

Submit a previewed existing job

I'm happy with the already-running/finished job you showed me in "preview". Give me access to that job, its logs, and [when it finishes] its output.

Submit a previewed new job

I'm happy with the new job the "preview" response proposed to run. Run that job.

Submit a new job (disable reuse)

I don't want to use an already-running/finished job. Run a new job that satisfies my job request.

Submit a new duplicate job (disable reuse)

I'm happy with the already-running/finished job you showed me in "preview". Run a new job exactly like that one.

Select a job and associate it with my JobRequest

I'm not happy with the job you chose, but I know of another job that satisfies my request. Assuming I'm right about that, attach my JobRequest to the existing job of my choice.

Just do the right thing without a preview

Satisfy job request X one way or another, and tell me the resulting job's UUID.

JobRequest/Job life cycle

Illustrating job re-use and preview facility:
  1. Client CA creates a JobRequest JRA with priority=0.
  2. Server creates job JX and assigns JX to JRA, but does not try to run JX yet because max(priority)=0.
  3. Client CA presents JX to the user. "We haven't computed this result yet, so we'll have to run a new job. Is this OK?"
  4. Client CB creates a JobRequest JRB with priority=1.
  5. Server assigns JX to JRB and puts JX in the execution queue with priority=1.
  6. Client CA updates JRA with priority=2.
  7. Server updates JX with priority=2.
  8. Job JX starts.
  9. Client CA updates JRA with priority=0. (This is as close as we get to a "cancel" operation.)
  10. Server updates JX with priority=1. (JRB still wants this job to complete.)
  11. Job JX finishes.
  12. Clients CA and CB have permission to read JX (ever since JX was assigned to their respective JobRequests) as well as its progress indicators, output, and log.

"JobRequest" schema

Attribute Type Description Discussion Examples
uuid, owner_uuid, modified_by_client_uuid, modified_by_user_uuid string Usual Arvados model attributes
created_at, modified_at datetime Usual Arvados model attributes
name string Unparsed
description text Unparsed
state string Once a request is committed, priority is the only attribute that can be modified. "Uncommitted"
"Committed"
job_uuid uuid The job that satisfies this job request.
Can be null if a suitable job has not yet been found or queued.
Assigned by the system: cannot be modified directly by clients.
If null, it can be changed by the system at any time.
If not null, it can be reset to null by a client if priority is zero.
mounts hash Objects to attach to the container's filesystem and stdin/stdout.
Keys starting with a forward slash indicate objects mounted in the container's filesystem.
Other keys are given special meanings here.
We use "stdin" instead of "/dev/stdin" because literally replacing /dev/stdin with a file would have a confusing effect on many unix programs. The stdin feature only affects the standard input of the first process started in the container; after that, the usual rules apply.		 
{
 "/input/foo":{
  "kind":"collection",
  "portable_data_hash":"d41d8cd98f00b204e9800998ecf8427e+0" 
 },
 "stdin":{
  "kind":"collection_file",
  "uuid":"zzzzz-4zz18-yyyyyyyyyyyyyyy",
  "path":"/foo.txt" 
 },
 "stdout":{
  "kind":"regular_file",
  "path":"/tmp/a.out" 
 }
}
runtime_constraints hash Restrict the job's access to compute resources and the outside world (in addition to its explicitly stated inputs and output).
Each key is the name of a capability, like "internet" or "API" or "clock". The corresponding value is true (the capability must be available in the job's runtime environment) or false (must not) or a value or an array of two numbers indicating an inclusive range. If a key is omitted, availability of the corresponding capability is acceptable but not necessary.		 This is a generalized version of "enforce purity restrictions": it is not a claim that the job will be pure. Rather, it helps us control and track runtime restrictions, which can be helpful when reasoning about whether a given job was pure.
In the most basic implementation, no capabilities are defined, and the only acceptable value of this attribute is the empty hash.
(TC)Should this structure be extensible like mounts?		 
{
  "ram":12000000000,
  "vcpus":[1,null]
}
container_image string Docker image repository and tag, docker image hash, collection UUID, or collection PDH.
environment hash environment variables and values that should be set in the container environment (docker run --env). This augments and (when conflicts exists) overrides environment variables given in the image's Dockerfile.
cwd string initial working directory, given as an absolute path (in the container) or a path relative to the WORKDIR given in the image's Dockerfile. The default is ".". 
"/tmp"
command array of strings Command to execute in the container. Default is the CMD given in the image's Dockerfile.
To use a UNIX pipeline, like "echo foo | tr f b", or to interpolate environment variables, make sure your container image has a shell, and use a command like ["sh","-c","echo $PATH | wc"].
output_path string Path to a directory or file inside the container that should be preserved as job's output when it finishes. This path must be, or be inside, one of the mount targets.
For best performance, point output_path to a writable collection mount.
priority number Higher number means spend more resources (e.g., go ahead of other queued jobs, bring up more nodes).
Zero means a job should not be run on behalf of this request. (Clients are expected to submit JobRequests with zero priority in order to prevew the job that will be used to satisfy it.)
Priority is null if and only if state="Uncommitted".
null
0
1000.5
-1
expires_at datetime After this time, priority is considered to be zero. If the assigned job is running at that time, the job may be cancelled to conserve resources.
null
2015-07-01T00:00:01Z
filters array Additional constraints for satisfying the request, given in the same form as the filters parameter accepted by the jobs.list API.
["created_at","<","2015-07-01T00:00:01Z"]

"Job" schema

Attribute Type Description Discussion Examples
uuid, owner_uuid, created_at, modified_at, modified_by_client_uuid, modified_by_user_uuid string Usual Arvados model attributes
state string 
"Queued" 
"Running" 
"Cancelled" 
"Failed" 
"Complete"
started_at, finished_at, log Same as current job
environment hash Must be equal to a JobRequest's environment in order to satisfy the JobRequest. (TC)We could offer a "resolve" process here like we do with mounts: e.g., hash values in the JobRequest environment could be resolved according to the given "kind". I propose we leave room for this feature but don't add it yet.
cwd, command, output_path string Must be equal to the corresponding values in a JobRequest in order to satisfy that JobRequest.
mounts hash Must contain the same keys as the JobRequest being satisfied. Each value must be within the range of values described in the JobRequest at the time the Job is assigned to the JobRequest.
runtime_constraints hash Compute resources, and access to the outside world, that are/were available to the job. Generally this will contain additional keys that are not present in any corresponding JobRequests: for example, even if no JobRequests specified constraints on the number of CPU cores, the number of cores actually used will be recorded here.
Permission/access types will change over time and it may be hard/impossible to translate old types to new. Such cases may cause old Jobs to be inelegible for assignment to new JobRequests.
(TC)Is it permissible for this to gain keys over time? For example, a job scheduler might not be able to predict how many CPU cores will be available until the job starts.
output string Portable data hash of the output collection.
pure boolean The job's output is thought to be dependent solely on its inputs, i.e., it is expected to produce identical output if repeated.
We want a feature along these lines, but "pure" seems to be a conclusion we can come to after examining various facts -- rather than a property of an individual job execution event -- and it probably needs something more subtle than a boolean.
container_image string Portable data hash of a collection containing the docker image used to run the job. (TC) If docker image hashes can be verified efficiently, we can use the native docker image hash here instead of a collection PDH.
progress number A number between 0.0 and 1.0 describing the fraction of work done.
If a job submits jobs of its own, it should update its own progress as the child jobs progress/finish.
priority number Priority assigned by the system, taking into account the priorities of all associated JobRequests.

Mount types

The "mounts" hash is the primary mechanism for adding data to the container at runtime (beyond what is already in the container image).

Each value of the "mounts" hash is itself a hash, whose "kind" key determines the handler used to attach data to the container.

Mount type kind Expected keys Description Examples Discussion
Arvados data collection collection
"portable_data_hash" or "uuid" may be provided. If not provided, a new collection will be created. This is useful when "writable":true and the job's output_path is (or is a subdirectory of) this mount target.
"writable" may be provided with a true or false to indicate the path must (or must not) be writable. If not specified, the system can choose.
"path" may be provided, and defaults to "/".
At job startup, the target path will have the same directory structure as the given path within the collection. Even if the files/directories are writable in the container, modifications will not be saved back to the original collections when the job ends.		 
{
 "kind":"collection",
 "uuid":"...",
 "path":"/foo.txt" 
}

{
 "kind":"collection",
 "uuid":"..." 
}
Git tree git_tree
One of {"git-url", "repository_name", "uuid"} must be provided.
One of {"commit", "revisions"} must be provided.
"path" may be provided. The default path is "/".
At job startup, the target path will have the source tree indicated by the given revision. The .git metadata directory will not be available: typically the system will use git-archive rather than git-checkout to prepare the target directory.
If a value is given for "revisions", it will be resolved to a set of commits (as desribed in the "ranges" section of git-revisions(1)) and the job request will be satisfiable by any commit in that set.
If a value is given for "commit", it will be resolved to a single commit, and the tree resulting from that commit will be used.
"path" can be used to select a subdirectory or a single file from the tree indicated by the selected commit.
Note that multiple commits can resolve to the same tree: for example, the file/directory given in "path" might not have changed between commits A and B.		 
{
 "kind":"git_tree",
 "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx",
 "commit":"master" 
}

{
 "kind":"git_tree",
 "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx",
 "commit_range":"bugfix^..master",
 "path":"/crunch_scripts/grep" 
}
The resolved mount (found in the Job record) will have only the "kind" key and a "blob" or "tree" key indicating the 40-character hash of the git tree/blob used.
Temporary directory tmp
"capacity": capacity (in bytes) of the storage device
At job startup, the target path will be empty. When the job finishes, the content will be discarded. This will be backed by a memory-based filesystem where possible.		 
{
 "kind":"tmp",
 "capacity":10000000000
}

Permissions

Users own JobRequests but the system owns Jobs. Users get permission to read Jobs by virtue of linked JobRequests.

API methods

Changes from the usual REST APIs:

arvados.v1.job_requests.create and .update

These methods can fail when objects referenced in the "mounts" hash do not exist, or the acting user has insufficient permission on them.

If state="Uncommitted":
  • has null priority.
  • can have its job_uuid reset to null by a client.
  • can have its job_uuid set to a non-null value by a system process.
If state="Committed":
  • has non-null priority.
  • cannot be modified, except that its priority can be changed to another non-null value.

arvados.v1.job_requests.cancel

Set priority to zero.

arvados.v1.job_requests.satisfy

Find or create a suitable job, and update job_uuid.

Return an error if job_uuid is not null.

Q: Can this be requested during create? Create+satisfy is a common operation so having a way to do it in a single API call might be a worthwhile convenience.

Q: Better name?

arvados.v1.jobs.create and .update

These methods are not callable except by system processes.

arvados.v1.jobs.progress

This method permits specific types of updates while a job is running: update progress, record success/failure.

Q: [How] can a client submitting job B indicate it shouldn't run unless/until job A succeeds?

Debugging

Q: Need any infrastructure debug-logging controls in this API?

Q: Need any job debug-logging controls in this API? Or just use environment vars?

Scheduling and running jobs

Q: When/how should we implement a hooks for futures/promises: e.g., "run job Y when jobs X0, X1, and X2 have finished"?

Accounting

A complete design for resource accounting and quota is out of scope here, but we do assert here that the API makes it feasible to retain accounting data.

It should be possible to retrieve, for a given job, a complete set of resource allocation intervals, each one including:
  • interval start time
  • interval end time (presented as null or now if the interval hasn't ended yet)
  • user uuid
  • job request id
  • job request priority
  • job state

Updated by Tom Clegg over 9 years ago · 64 revisions