Upgrading to Containers API » History » Version 1
Peter Amstutz, 10/17/2018 06:45 PM
| 1 | 1 | Peter Amstutz | h1. Upgrading to Containers API |
|---|---|---|---|
| 2 | |||
| 3 | The "containers" API is the recommended way to submit compute work to Arvados. It supersedes the "jobs" API, which is deprecated. |
||
| 4 | |||
| 5 | h2. Benefits over the "jobs" API |
||
| 6 | |||
| 7 | * Simpler and more robust execution with fewer points of failure |
||
| 8 | * Automatic retry for containers that fail to run to completion due to infrastructure errors |
||
| 9 | * Scales to thousands of simultaneous containers |
||
| 10 | * Able to support alternate schedulers/dispatchers in addition to slurm |
||
| 11 | * Improved logging, different streams logs/metrics stored in different files in the log collection |
||
| 12 | * Records more upfront detail about the compute node, and additional metrics (such as available disk space over the course of the container run) |
||
| 13 | * Better behavior when deciding whether to reuse past work -- pick the oldest container that matches the criteria |
||
| 14 | * Can reuse running containers between workflows, cancelling a workflow will not cancel containers that are shared with other workflows |
||
| 15 | * Supports setting time-to-live on intermediate output collections for automatic cleanup |
||
| 16 | * Supports "secret" inputs, suitable for passwords or access tokens, which are hidden from the API responses and logs, and forgotten after use |
||
| 17 | * Does not require "git" for dispatching work |
||
| 18 | |||
| 19 | h2. Differences from the "jobs" API |
||
| 20 | |||
| 21 | Containers cannot reuse jobs (but can reuse other containers) |
||
| 22 | |||
| 23 | Must run "crunch-dispatch-slurm":https://doc.arvados.org/install/crunch2-slurm/install-dispatch.html instead of @crunch-dispatch.rb@ |
||
| 24 | |||
| 25 | Containers cannot be run from pipeline templates (there is a new "Workflow" type) |
||
| 26 | |||
| 27 | APIs are incompatible, code which integrates with the "jobs" API must be updated to work with containers |
||
| 28 | |||
| 29 | Containers have network access disabled by default |
||
| 30 | |||
| 31 | The keep mount only exposes collections which are explicitly listed as inputs |
||
| 32 | |||
| 33 | h2. Migrating to "containers" API |
||
| 34 | |||
| 35 | Run your workflows using @arvados-cwl-runner --api=containers@ |
||
| 36 | |||
| 37 | Register your workflows so they can be run from workbench using @arvados-cwl-runner --api=containers@ --create-workflow@ |
||
| 38 | |||
| 39 | Read "Migrating running CWL on jobs API to containers API":https://doc.arvados.org/user/cwl/cwl-style.html#migrate |
||
| 40 | |||
| 41 | Use @arv:APIRequirement: {}@ in the @requirements@ section of your CWL file to enable network access for the container (see "Arvados CWL Extensions":https://doc.arvados.org/user/cwl/cwl-extensions.html) |
||
| 42 | |||
| 43 | For examples on how to manage container requests with the Python SDK, see https://doc.arvados.org/sdk/python/cookbook.html |