Project

General

Profile

Containers API » History » Version 64

Tom Clegg, 10/10/2018 03:00 PM

1 9 Tom Clegg
{{>TOC}}
2
3 16 Tom Clegg
h1. Containers API (DRAFT)
4 1 Tom Clegg
5 41 Tom Clegg
See also [[Container dispatch]]
6 22 Peter Amstutz
7 16 Tom Clegg
A Container resource is a record of a computational process.
8 1 Tom Clegg
* Its goal is to capture, unambiguously, as much information as possible about the environment in which the process was run. For example, git trees, data collections, and docker images are stored as content addresses. This makes it possible to reason about the difference between two processes, and to replay a process at a different time and place.
9 16 Tom Clegg
* Clients can read Container records, but only the system can create or modify them.
10 1 Tom Clegg
11 16 Tom Clegg
*Note about the term "containers" vs. "jobs" and "services":* Here, we focus on the use of containers as producers of output data. We anticipate extending the feature set to cover service containers as well. The distinguishing feature of a service container is that _having it running_ is inherently valuable because of the way it interacts with the outside world.
12 1 Tom Clegg
13 16 Tom Clegg
A ContainerRequest is a client's expression of interest in knowing the outcome of a computational process.
14
* Typically, in this context the client's description of the process is less precise than a Container: a ContainerRequest describes container _constraints_ which can have different interpretations over time. For example, a ContainerRequest with a @{"kind":"git_tree","commit_range":"abc123..master",...}@ mount might be satisfiable by any of several different source trees, and this set of satisfying source trees can change when the repository's "master" branch is updated.
15
* The system is responsible for finding suitable Containers and assigning them to ContainerRequests. (Currently this is expected to be done synchronously during the containerRequests.create and containerRequests.update transactions.)
16
* A ContainerRequest may indicate that it can _only_ be satisfied by a new Container record (i.e., existing results should not be reused). In this case creating a ContainerRequest amounts to a submission to the container queue. This is appropriate when the purpose of the ContainerRequest is to test whether a process is repeatable.
17
* A ContainerRequest may indicate that it _cannot_ be satisfied by a new Container record. This is an appropriate way to test whether a result is already available.
18 1 Tom Clegg
19 16 Tom Clegg
When the system has assigned a Container to a ContainerRequest, anyone with permission to read the ContainerRequest also has permission to read the Container.
20
21 1 Tom Clegg
h2. Use cases
22
23
h3. Preview
24
25 16 Tom Clegg
Tell me how you would satisfy container request X. Which pdh/commits would be used? Is the satisfying container already started? finished?
26 9 Tom Clegg
27 16 Tom Clegg
h3. Submit a previewed existing container
28 1 Tom Clegg
29 16 Tom Clegg
I'm happy with the already-running/finished container you showed me in "preview". Give me access to that container, its logs, and [when it finishes] its output.
30 1 Tom Clegg
31 16 Tom Clegg
h3. Submit a previewed new container
32 9 Tom Clegg
33 16 Tom Clegg
I'm happy with the new container the "preview" response proposed to run. Run that container.
34 9 Tom Clegg
35 16 Tom Clegg
h3. Submit a new container (disable reuse)
36 9 Tom Clegg
37 16 Tom Clegg
I don't want to use an already-running/finished container. Run a new container that satisfies my container request.
38 9 Tom Clegg
39 16 Tom Clegg
h3. Submit a new duplicate container (disable reuse)
40 9 Tom Clegg
41 16 Tom Clegg
I'm happy with the already-running/finished container you showed me in "preview". Run a new container exactly like that one.
42 9 Tom Clegg
43 16 Tom Clegg
h3. Select a container and associate it with my ContainerRequest
44 9 Tom Clegg
45 16 Tom Clegg
I'm not happy with the container you chose, but I know of another container that satisfies my request. Assuming I'm right about that, attach my ContainerRequest to the existing container of my choice.
46 1 Tom Clegg
47 9 Tom Clegg
h3. Just do the right thing without a preview
48
49 16 Tom Clegg
Satisfy container request X one way or another, and tell me the resulting container's UUID.
50 6 Tom Clegg
51 16 Tom Clegg
h2. ContainerRequest/Container life cycle
52 6 Tom Clegg
53 16 Tom Clegg
Illustrating container re-use and preview facility:
54
# Client ClientA creates a ContainerRequest CRA with priority=0.
55
# Server creates container CX and assigns CX to CRA, but does not try to run CX yet because max(priority)=0.
56
# Client ClientA presents CX to the user. "We haven't computed this result yet, so we'll have to run a new container. Is this OK?"
57
# Client ClientB creates a ContainerRequest CRB with priority=1.
58
# Server assigns CX to CRB and puts CX in the execution queue with priority=1.
59
# Client ClientA updates CRA with priority=2.
60
# Server updates CX with priority=2.
61
# Container CX starts.
62
# Client ClientA updates CRA with priority=0. (This is as close as we get to a "cancel" operation.)
63
# Server updates CX with priority=1. (CRB still wants this container to complete.)
64
# Container CX finishes.
65
# Clients ClientA and ClientB have permission to read CX (ever since CX was assigned to their respective ContainerRequests) as well as its progress indicators, output, and log.
66 1 Tom Clegg
67 16 Tom Clegg
h2. "ContainerRequest" schema
68 1 Tom Clegg
69 6 Tom Clegg
|Attribute|Type|Description|Discussion|Examples|
70
|uuid, owner_uuid, modified_by_client_uuid,  modified_by_user_uuid|string|Usual Arvados model attributes|||
71
|
72
|created_at, modified_at|datetime|Usual Arvados model attributes|||
73 1 Tom Clegg
|
74
|name|string|Unparsed|||
75
|
76 6 Tom Clegg
|description|text|Unparsed|||
77 15 Tom Clegg
|
78 16 Tom Clegg
|properties|object|Client-defined structured data that does not affect how the container is run.|||
79 11 Tom Clegg
|
80 44 Tom Clegg
|state|string|Once a request is committed, the only attributes that can be modified are priority, container_uuid, and container_count_max. A request with @state="Final"@ cannot have any of its functional parts modified (i.e., only name, description, and properties fields can be modified).||@"Uncommitted"@
81 17 Tom Clegg
@"Committed"@
82
@"Final"@|
83 15 Tom Clegg
|
84 60 Tom Clegg
|requesting_container_uuid|string|When the referenced container ends, the container request is automatically cancelled.|Can be null. If changed to a non-null value, it must refer to a container that is running.||
85 1 Tom Clegg
|
86
|container_uuid|uuid|The container that satisfies this container request.|See "methods" below.||
87
|
88 17 Tom Clegg
|container_count_max|positive integer|Maximum number of containers to start ("attempts").|See "methods" below.||
89
|
90 8 Tom Clegg
|mounts|hash|Objects to attach to the container's filesystem and stdin/stdout.
91
Keys starting with a forward slash indicate objects mounted in the container's filesystem.
92
Other keys are given special meanings here.|
93 1 Tom Clegg
We use "stdin" instead of "/dev/stdin" because literally replacing /dev/stdin with a file would have a confusing effect on many unix programs. The stdin feature only affects the standard input of the first process started in the container; after that, the usual rules apply.|
94
<pre>{
95
 "/input/foo":{
96
  "kind":"collection",
97 9 Tom Clegg
  "portable_data_hash":"d41d8cd98f00b204e9800998ecf8427e+0"
98 1 Tom Clegg
 },
99
 "stdin":{
100 55 Tom Clegg
  "kind":"collection",
101 1 Tom Clegg
  "uuid":"zzzzz-4zz18-yyyyyyyyyyyyyyy",
102 8 Tom Clegg
  "path":"/foo.txt"
103 9 Tom Clegg
 },
104 1 Tom Clegg
 "stdout":{
105 31 Tom Clegg
  "kind":"file",
106 1 Tom Clegg
  "path":"/tmp/a.out"
107 9 Tom Clegg
 }
108 1 Tom Clegg
}</pre>|
109 11 Tom Clegg
|
110 16 Tom Clegg
|runtime_constraints|hash|Restrict the container's access to compute resources and the outside world (in addition to its explicitly stated inputs and output).
111 53 Tom Clegg
-- Each key is the name of a capability, like "internet" or "API" or "clock". The corresponding value is @true@ (the capability must be available in the container's runtime environment) or @false@ (must not) or a value or an array of two numbers indicating an inclusive range. Numeric values are given in basic units (e.g., RAM is given in bytes, not KB or MB or MiB). If a key is omitted, availability of the corresponding capability is acceptable but not necessary.|
112
This is a generalized version of "enforce purity restrictions": it is not a claim that the container will be pure. Rather, it helps us control and track runtime restrictions, which can be helpful when reasoning about whether a given container was pure.|
113 10 Tom Clegg
<pre>
114 1 Tom Clegg
{
115
  "ram":12000000000,
116
  "vcpus":2,
117 52 Tom Clegg
  "keep_cache_ram":256000000,
118 1 Tom Clegg
  "API":true
119 53 Tom Clegg
}</pre>|
120
|
121
|scheduling_parameters|hash|Parameters to pass to the container scheduler (e.g., SLURM) when running the container.||
122
<pre>
123
{
124
  "partitions":["fastcpu","vfastcpu"]
125 1 Tom Clegg
}</pre>|
126
|
127 14 Tom Clegg
|container_image|string|Docker image repository and tag, docker image hash, collection UUID, or collection PDH.|||
128 1 Tom Clegg
|
129
|environment|hash|environment variables and values that should be set in the container environment (@docker run --env@). This augments and (when conflicts exists) overrides environment variables given in the image's Dockerfile.|||
130
|
131 8 Tom Clegg
|cwd|string|initial working directory, given as an absolute path (in the container) or a path relative to the WORKDIR given in the image's Dockerfile. The default is @"."@.||<pre>"/tmp"</pre>|
132 1 Tom Clegg
|
133
|command|array of strings|Command to execute in the container. Default is the CMD given in the image's Dockerfile.|
134
To use a UNIX pipeline, like "echo foo &#124; tr f b", or to interpolate environment variables, make sure your container image has a shell, and use a command like @["sh","-c","echo $PATH &#124; wc"]@.||
135
|
136
|output_path|string|Path to a directory or file inside the container that should be preserved as container's output when it finishes.|This path _must_ be, or be inside, one of the mount targets.
137 14 Tom Clegg
For best performance, point output_path to a writable collection mount.||
138 1 Tom Clegg
|
139 58 Tom Clegg
|priority|integer 0&le;N&le;1000|Higher number means spend more resources (e.g., go ahead of other queued containers, bring up more nodes).
140
-- Zero means a container should not be run on behalf of this request. (Clients are expected to submit ContainerRequests with zero priority in order to prevew the container that will be used to satisfy it.)|
141
Priority is ignored when @state!="Committed"@.|
142 11 Tom Clegg
null
143 1 Tom Clegg
@0@
144 58 Tom Clegg
@10@
145
@1000@|
146 11 Tom Clegg
|
147 16 Tom Clegg
|expires_at|datetime|After this time, priority is considered to be zero. If the assigned container is running at that time, the container _may_ be cancelled to conserve resources.||
148 11 Tom Clegg
null
149
@2015-07-01T00:00:01Z@|
150
|
151 50 Tom Clegg
|use_existing|boolean|If possible, use an existing (non-failed) container to satisfy the request instead of creating a new one.|Default is true|
152
@true@
153
@false@|
154
|
155 16 Tom Clegg
|filters|array|Additional constraints for satisfying the request, given in the same form as the @filters@ parameter accepted by the @containers.list@ API.||
156 1 Tom Clegg
@["created_at","<","2015-07-01T00:00:01Z"]@|
157
|
158 51 Tom Clegg
|output_name|string|Name of the output collection that will be created when the container finishes.|If null, a unique name will be assigned automatically.|
159
null
160
@"my container output"@|
161
|
162 56 Tom Clegg
|output_ttl|non-negative integer|Desired lifetime of the output collection, in seconds. This is implemented by setting trash_at and delete_at attributes on the output collection.|If zero, trash_at and delete_at will be null and the output collection will not be deleted automatically.|
163
@0@
164
@86400@|
165
|
166 1 Tom Clegg
167 16 Tom Clegg
h2. "Container" schema
168 1 Tom Clegg
169 9 Tom Clegg
|Attribute|Type|Description|Discussion|Examples|
170
|
171
|uuid, owner_uuid, created_at, modified_at, modified_by_client_uuid,  modified_by_user_uuid|string|Usual Arvados model attributes|||
172
|
173 33 Tom Clegg
|state|string||See "Container states" below|
174 20 Tom Clegg
@"Queued"@
175 25 Tom Clegg
@"Locked"@
176 20 Tom Clegg
@"Running"@
177
@"Cancelled"@
178
-@"Failed"@-
179
@"Complete"@|
180 1 Tom Clegg
|
181 26 Tom Clegg
|locked_by_uuid|string|UUID of a token, indicating which dispatch process changed state to Locked|If null, any token can be used to lock. If not null, only the indicated token can modify.
182
Is null if and only if state&notin;{"Locked","Running"}||
183 25 Tom Clegg
|
184 28 Tom Clegg
|auth_uuid|string|UUID of a token to be passed into the container itself, used to access Keep-backed mounts, etc.|Is null if and only if state&notin;{"Locked","Running"}||
185
|
186 25 Tom Clegg
|started_at, finished_at, log||Same as Job attributes in Crunch1|||
187 8 Tom Clegg
|
188 16 Tom Clegg
|environment|hash|Must be equal to a ContainerRequest's environment in order to satisfy the ContainerRequest.|(TC)We could offer a "resolve" process here like we do with mounts: e.g., hash values in the ContainerRequest environment could be resolved according to the given "kind". I propose we leave room for this feature but don't add it yet.||
189 9 Tom Clegg
|
190 16 Tom Clegg
|cwd, command, output_path|string|Must be equal to the corresponding values in a ContainerRequest in order to satisfy that ContainerRequest.|||
191 9 Tom Clegg
|
192 16 Tom Clegg
|mounts|hash|Must contain the same keys as the ContainerRequest being satisfied. Each value must be within the range of values described in the ContainerRequest _at the time the Container is assigned to the ContainerRequest._|||
193 14 Tom Clegg
|
194 16 Tom Clegg
|runtime_constraints|hash|Compute resources, and access to the outside world, that are/were available to the container.
195
-- Generally this will contain additional keys that are not present in any corresponding ContainerRequests: for example, even if no ContainerRequests specified constraints on the number of CPU cores, the number of cores actually used will be recorded here.|
196
Permission/access types will change over time and it may be hard/impossible to translate old types to new. Such cases may cause old Containers to be inelegible for assignment to new ContainerRequests.
197
-- (TC)Is it permissible for this to gain keys over time? For example, a container scheduler might not be able to predict how many CPU cores will be available until the container starts.||
198 1 Tom Clegg
|
199 54 Tom Clegg
|scheduling_parameters|hash|See Container Request schema above.|||
200
|
201 9 Tom Clegg
|output|string|Portable data hash of the output collection.|||
202
|
203 21 Tom Clegg
|exit_code|integer|Process exit code.|Is null if and only if @state!="Complete"@|
204
@null@
205
@0@
206
@1@
207
@129@|
208
|
209 16 Tom Clegg
|-pure-|-boolean-|-The container's output is thought to be dependent solely on its inputs, i.e., it is expected to produce identical output if repeated.-|
210
We want a feature along these lines, but "pure" seems to be a conclusion we can come to after examining various facts -- rather than a property of an individual container execution event -- and it probably needs something more subtle than a boolean.||
211 8 Tom Clegg
|
212 16 Tom Clegg
|container_image|string|Portable data hash of a collection containing the docker image used to run the container.|(TC) *If* docker image hashes can be verified efficiently, we can use the native docker image hash here instead of a collection PDH.||
213 10 Tom Clegg
|
214 8 Tom Clegg
|progress|number|A number between 0.0 and 1.0 describing the fraction of work done.|
215 16 Tom Clegg
If a container submits containers of its own, it should update its own progress as the child containers progress/finish.||
216 8 Tom Clegg
|
217 16 Tom Clegg
|priority|number|Priority assigned by the system, taking into account the priorities of all associated ContainerRequests.|||
218 62 Tom Clegg
|runtime_status|hash|Details of the contained process's progress/outcome.|Can be updated by the container or the system while @state=="Running"@. If an "error" key exists, the container will not qualify for reuse even if it is still running.|<pre>{
219 63 Tom Clegg
  "activity": "flushing logs",
220 62 Tom Clegg
  "error": "error in foo: bar not found"
221
}</pre>|
222 8 Tom Clegg
223
h2. Mount types
224
225
The "mounts" hash is the primary mechanism for adding data to the container at runtime (beyond what is already in the container image).
226
227
Each value of the "mounts" hash is itself a hash, whose "kind" key determines the handler used to attach data to the container.
228 9 Tom Clegg
229 61 Tom Clegg
|Mount type|@kind@|Expected keys|Description|Examples|
230 9 Tom Clegg
|
231
|Arvados data collection|@collection@|
232 64 Tom Clegg
@"portable_data_hash"@, @"uuid"@, or both _may_ be provided in a container request.
233
If both are provided, the uuid is considered advisory, and the container uses the provided portable_data_hash.
234
If only the uuid is provided, the container uses the portable data hash corresponding to the given uuid _at the time the container is assigned to the container request._
235
If neither is provided, a new collection is created when the container runs. This is useful when @"writable":true@ and the container's @output_path@ is (or is a subdirectory of) this mount target.
236 8 Tom Clegg
@"writable"@ may be provided with a @true@ or @false@ to indicate the path must (or must not) be writable. If not specified, the system can choose.
237 1 Tom Clegg
@"path"@ may be provided, and defaults to @"/"@.|
238 16 Tom Clegg
At container startup, the target path will have the same directory structure as the given path within the collection. Even if the files/directories are writable in the container, modifications will _not_ be saved back to the original collections when the container ends.|
239 8 Tom Clegg
<pre>
240 9 Tom Clegg
{
241 1 Tom Clegg
 "kind":"collection",
242
 "uuid":"...",
243
 "path":"/foo.txt"
244
}
245
246 8 Tom Clegg
{
247 1 Tom Clegg
 "kind":"collection",
248 13 Tom Clegg
 "uuid":"..."
249 8 Tom Clegg
}
250 61 Tom Clegg
</pre>|
251 1 Tom Clegg
|
252
|Git tree|@git_tree@|
253 59 Tom Clegg
One of {@"git_url"@, @"repository_name"@, @"uuid"@} must be provided.
254 1 Tom Clegg
One of {@"commit"@, @"revisions"@} must be provided.
255 14 Tom Clegg
"path" may be provided. The default path is "/".|
256 16 Tom Clegg
At container startup, the target path will have the source tree indicated by the given revision. The @.git@ metadata directory _will not_ be available: typically the system will use @git-archive@ rather than @git-checkout@ to prepare the target directory.
257
-- If a value is given for @"revisions"@, it will be resolved to a set of commits (as desribed in the "ranges" section of git-revisions(1)) and the container request will be satisfiable by any commit in that set.
258 14 Tom Clegg
-- If a value is given for @"commit"@, it will be resolved to a single commit, and the tree resulting from that commit will be used.
259 8 Tom Clegg
-- @"path"@ can be used to select a subdirectory or a single file from the tree indicated by the selected commit.
260 1 Tom Clegg
-- Multiple commits can resolve to the same tree: for example, the file/directory given in @"path"@ might not have changed between commits A and B.
261 16 Tom Clegg
-- The resolved mount (found in the Container record) will have only the "kind" key and a "blob" or "tree" key indicating the 40-character hash of the git tree/blob used.|
262 1 Tom Clegg
<pre>
263 8 Tom Clegg
{
264
 "kind":"git_tree",
265
 "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx",
266 1 Tom Clegg
 "commit":"master"
267
}
268
269
{
270
 "kind":"git_tree",
271 8 Tom Clegg
 "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx",
272 57 Tom Clegg
 "revisions":"bugfix^..master",
273 5 Tom Clegg
 "path":"/crunch_scripts/grep"
274 1 Tom Clegg
}
275 61 Tom Clegg
</pre>|
276 1 Tom Clegg
|
277
|Temporary directory|@tmp@|
278 27 Tom Clegg
@"capacity"@: capacity (in bytes) of the storage device.
279
@"device_type"@ (optional, default "network"): one of @{"ram", "ssd", "disk", "network"}@ indicating the acceptable level of performance.|
280
At container startup, the target path will be empty. When the container finishes, the content will be discarded. This will be backed by a storage mechanism no slower than the specified type.|
281 1 Tom Clegg
<pre>
282 11 Tom Clegg
{
283 1 Tom Clegg
 "kind":"tmp",
284 27 Tom Clegg
 "capacity":100000000000
285 1 Tom Clegg
}
286 27 Tom Clegg
287
{
288 46 Tom Clegg
 "kind":"tmp",
289 27 Tom Clegg
 "capacity":1000000000,
290
 "device_type":"ram"
291
}
292 61 Tom Clegg
</pre>|
293 12 Tom Clegg
|
294
|Keep|@keep@|
295
Expose all readable collections via arv-mount.|Requires suitable runtime constraints.|
296
<pre>
297 13 Tom Clegg
{
298 2 Tom Clegg
 "kind":"keep"
299 8 Tom Clegg
}
300 61 Tom Clegg
</pre>|
301 1 Tom Clegg
|
302 30 Tom Clegg
|Mounted file or directory|@file@|
303
@"path"@: absolute path (inside the container) of a file or directory that is (or is inside) another mount target.|Can be used for "stdin" and "stdout" targets.|
304
<pre>
305
{
306
 "kind":"file",
307
 "path":"/mounted_tmp/a.out"
308
}
309 61 Tom Clegg
</pre>|
310 47 Tom Clegg
|JSON document|@json@|
311 1 Tom Clegg
A JSON-encoded string, array, or object.||
312
<pre>
313
{
314
 "kind":"json",
315
 "content":{"foo":"bar"}
316
}
317 61 Tom Clegg
</pre>|
318
|Text file|@text@|
319
Arbitrary UTF-8 text.|Not suitable for binary data.|
320
<pre>
321
{
322
 "kind":"text",
323
 "content":"Foo bar.\n"
324
}
325 47 Tom Clegg
</pre>|
326 61 Tom Clegg
|
327 33 Tom Clegg
328
h2. Container states
329
330 36 Tom Clegg
|*state*|*significance*|*allowed next*|
331
|Queued|Waiting for a dispatcher to lock it and try to run the container.|Locked, Cancelled|
332
|Locked|A dispatcher has "taken" the container and is allocating resources for it. The container has not started yet.|Queued, Running, Cancelled|
333
|Running|Resources have been allocated and the contained process has been started (or is about to start). Crunch-run _must_ set state to Running _before_ there is any possibility that user code will run in the container.|Complete, Cancelled|
334
|Complete|Container was running, and the contained process/command has exited.|-|
335
|Cancelled|The container did not run long enough to produce an exit code. This includes cases where the container didn't even start, cases where the container was interrupted/killed before it exited by itself (e.g., priority changed to 0), and cases where some problem prevented the system from capturing the contained process's exit status (exit code and output).|-|
336
337 1 Tom Clegg
338
h2. Permissions
339
340 16 Tom Clegg
Users own ContainerRequests but the system owns Containers.  Users get permission to read Containers by virtue of linked ContainerRequests.
341 1 Tom Clegg
342
h2. API methods
343
344
Changes from the usual REST APIs:
345
346 16 Tom Clegg
h3. arvados.v1.container_requests.create and .update
347 8 Tom Clegg
348 11 Tom Clegg
These methods can fail when objects referenced in the "mounts" hash do not exist, or the acting user has insufficient permission on them.
349
350 42 Tom Clegg
These methods accept an optional boolean "satisfy" parameter. If true, and the create/update operation is successful, a "satisfy" API is then called implicitly, and the create/update response reflects the semantics of "satisfy" given below: e.g., it might return a non-200 status (201? 202?) to indicate the container request was created, but has not been satisfied yet: in this case the caller should wait a bit and then call "satisfy" explicitly.
351
352
State-dependent validations:
353
354 11 Tom Clegg
If @state="Uncommitted"@:
355 16 Tom Clegg
* has null @priority@.
356
* can have its @container_uuid@ reset to null by a client.
357 1 Tom Clegg
* can have its @container_uuid@ set to a non-null value by a system process.
358
359
If @state="Committed"@:
360
* has non-null @priority@.
361 17 Tom Clegg
* can have its @priority@ changed (but not to null).
362
* can have its @container_count_max@ changed.
363
* can have its @container_uuid@ changed by the system. (This allows the system to re-attempt a failed container.)
364 45 Tom Clegg
* can have its @name@, @description@, and @properties@ changed.
365
* cannot be modified in other ways.
366 1 Tom Clegg
367
If @state="Final"@:
368 45 Tom Clegg
* can have its @name@, @description@, and @properties@ changed.
369
* cannot be modified in other ways.
370 42 Tom Clegg
371 17 Tom Clegg
372 16 Tom Clegg
h3. arvados.v1.container_requests.cancel
373 8 Tom Clegg
374 11 Tom Clegg
Set @priority@ to zero.
375
376 16 Tom Clegg
h3. arvados.v1.container_requests.satisfy
377 11 Tom Clegg
378 40 Tom Clegg
If container_uuid is null, find or create a suitable container, and update @container_uuid@.
379 11 Tom Clegg
380 40 Tom Clegg
If container_uuid is not null, respond immediately.
381 1 Tom Clegg
382 40 Tom Clegg
Return a retryable error if the container is not known to be unsatisfiable, but was not satisfied in time to respond to this API request. IOW, clients should be prepared to poll until the container is satisfied.
383
384
The premise is that "create container request" should be able to return quickly, even if the system needs some time to decide how/whether to satisfy the new CR -- but it should also be easy to write a client that submits a ContainerRequest and then waits for a Container to be assigned.
385
386 43 Tom Clegg
This behavior can also be requested at creation time; see "create" above.
387 11 Tom Clegg
388 1 Tom Clegg
Q: Better name?
389
390 16 Tom Clegg
h3. arvados.v1.containers.create and .update
391 1 Tom Clegg
392
These methods are not callable except by system processes.
393
394 16 Tom Clegg
h3. arvados.v1.containers.progress
395 1 Tom Clegg
396 39 Tom Clegg
This method permits the container itself (using the token indicated by auth_uuid) to update the progress field.
397 16 Tom Clegg
398 1 Tom Clegg
399 37 Tom Clegg
h3. arvados.v1.containers.auth
400 29 Tom Clegg
401 37 Tom Clegg
@GET /arvados/v1/containers/{uuid}/auth@
402 29 Tom Clegg
403
Given the uuid of a container, return the api_client_authorization record indicated by its auth_uuid. The token used to make this request must be the one indicated by the container's locked_by_uuid.
404
405 1 Tom Clegg
h2. Debugging
406
407
Q: Need any infrastructure debug-logging controls in this API?
408
409 16 Tom Clegg
Q: Need any container debug-logging controls in this API? Or just use environment vars?
410 1 Tom Clegg
411 16 Tom Clegg
h2. Scheduling and running containers
412 11 Tom Clegg
413 16 Tom Clegg
Q: When/how should we implement a hooks for futures/promises: e.g., "run container Y when containers X0, X1, and X2 have finished"?
414 11 Tom Clegg
415 24 Tom Clegg
(PA) Having a field specifying "wait until time X to run this container" would be generally useful for cron-style tasks.
416
417
418 11 Tom Clegg
h2. Accounting
419
420
A complete design for resource accounting and quota is out of scope here, but we do assert here that the API makes it feasible to retain accounting data.
421
422 16 Tom Clegg
It should be possible to retrieve, for a given container, a complete set of resource allocation intervals, each one including:
423 11 Tom Clegg
* interval start time
424
* interval end time (presented as null or now if the interval hasn't ended yet)
425 1 Tom Clegg
* user uuid
426
* container request id
427
* container request priority
428
* container state
429 17 Tom Clegg
430
h2. TBD
431
432
How does a client get a list of previous (presumably failed) container attempts for a given request?
433 23 Tom Clegg
* Add an array property, like previous_ or attempted_container_uuids?
434
435 38 Tom Clegg
Classifying failure/error modes
436
* (PA) I think we should distinguish between infrastructure failure and task failure by distinguishing between "TempFail" and "PermFail" in the container state. "TempFail" shouldn't count againt the container_count_max count, or alternately we only honor container_count_max for "TempFail" tasks and don't retry "PermFail". Ideally, "TempFail" containers should retry forever, but with a backoff. One way to do the backoff is to schedule the container to run at a specific time in the future.
437
* (TC) Classifying failure modes sounds useful, but I think it's wrong to overload the container state field with this information. State should represent the state of the container, not an assessment of how it got into that state. "Success/failure" has no bearing on what state the container can be in next, for example. If anything, I'd consider consolidating "Cancelled" and "Complete" (as "Stopped"?) rather than loading more information into the state field.
438
* (TC) The "temporary/permanent" distinction seems orthogonal to the "infrastructure/user-code" distinction. E.g., if a container cannot run because the static physical hardware does not have enough memory, we shouldn't retry. E.g., if a container fails because the user code timed out trying to read from Keep, retrying would be worthwhile. It seems hard (impossible?) for us to determine automatically (reliably) whether an infrastructure problem is the root cause of a given container's non-zero exit code, and whether there's a reasonable chance retrying now will avoid hitting the same infrastructure problem.
439
* (TC) The concept of "retry" seems to belong in ContainerRequest, not Container. A Container is just a container; if you "run something again", you've got a new container.
440 18 Peter Amstutz
441
h2. References
442
443
Should consider how this fits in with Kubernetes notion of jobs:
444
445
https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/jobs.md