Story #17296

Singularity proof of concept

Added by Peter Amstutz 3 months ago. Updated 4 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
5.0

Description

When we run a container on a compute node, we do a container conversion, on the fly, to a SIF file, and run that with singularity instead. Perhaps we even save the SIF file in Keep and do something with another Link object to make it findable in the future, for the corresponding docker image. TODO: check if the framework we built in for the docker image format v1 -> v2 could be used here.

  1. global option that switches between docker or singularity runner
  2. container_request runtime parameters flag that chooses between docker and singularity
  3. crunch-run gets docker tar file from keep (existing docker v2 format images)
  4. crunch-run converts docker tar file to SIF:
$ docker save arvados/jobs:latest > arvados-jobs.latest.tar
$ ls -laF arvados-jobs.latest.tar 
-rw-r--r-- 1 ward ward 295209984 Jan 14 17:16 arvados-jobs.latest.tar
$ singularity build arvados-jobs.latest.sif docker-archive://arvados-jobs.latest.tar
INFO:    Starting build...
...
  1. crunch-run executes singularity with mount points, stdout/stderr captured to logs
  2. slurm dispatcher supports singularity
    1. ideally the backend container runner should be transparent to the dispatcher
  3. proof of concept will be tested on 9tee4
  4. assume that user id inside the container will be the same as the crunch-run user (?)
  5. try to support running containers without setuid, identify specific features that require setuid on singularity binary.

Testing goals / acceptance criteria

  1. MVP: runs a container
  2. default value for singularity binary (/usr/bin/singularity) but can be changed from arvados config.yml
  3. captures stdout/stderr to logs
  4. can bind-mount arv-mount inside the container
  5. can bind mount tmp/output directories inside the container
  6. output files have proper permissions to be read for upload & cleaned up (deleted) by crunch-run
  7. see if it makes sense to have singularity mock the docker API
  8. should have similar test coverage of singularity features as exist to the Docker features

For future tickets:

  1. crunchstat
  2. memory / CPU constraints

Subtasks

Task #17318: reviewNewWard Vandewege


Related issues

Related to Arvados - Story #17241: Scoping/grooming Singularity support workResolved

Related to Arvados Epics - Story #16305: Singularity supportIn Progress01/01/202105/31/2021

History

#1 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)
  • Subject changed from Singularity MVP to Singularity proof of concept

#2 Updated by Ward Vandewege 3 months ago

  • Description updated (diff)

#3 Updated by Ward Vandewege 3 months ago

  • Related to Story #17241: Scoping/grooming Singularity support work added

#4 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 3 months ago

  • Story points set to 5.0
  • Description updated (diff)

#6 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#7 Updated by Ward Vandewege 3 months ago

  • Assigned To set to Tom Clegg

#8 Updated by Peter Amstutz 2 months ago

  • Target version changed from 2021-02-17 sprint to 2021-03-03 sprint

#9 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2021-03-03 sprint to 2021-03-17 sprint

#10 Updated by Peter Amstutz about 2 months ago

  • Category set to Crunch

#11 Updated by Peter Amstutz about 2 months ago

  • Assigned To changed from Tom Clegg to Nico César

#12 Updated by Nico César about 1 month ago

  • Status changed from New to In Progress

#13 Updated by Peter Amstutz about 1 month ago

#14 Updated by Nico César about 1 month ago

very first take trying to create an abstraction that matches 1 to 1 with docker for now,

c1d1f0502b8a0f049dba41da2f6b19a0d4b03d77

https://ci.arvados.org/view/Developer/job/developer-run-tests/2380/

TODO:

  • [DONE] 0a27815bd review ContainerConfig, HostConfig settings and add them to the ThinContainerExecRunner interface as Get/Set methods to abstract from the internal representation
  • review all the networking related options and see if can be simplified
  • make a run with crunch-run --container-runner singularity to see how it behaves
  • add tests related to singularity

#15 Updated by Nico César about 1 month ago

Commit:0a27815bd 17241-singularity-take1

#16 Updated by Peter Amstutz about 1 month ago

  • Target version changed from 2021-03-17 sprint to 2021-03-31 sprint

#17 Updated by Nico César about 1 month ago

As I'm reading all the documentation available about singularity I want to write down some notes:

It is also important to note that the philosophy of Singularity is Integration over Isolation. Most container run times strive to isolate your container from the host system and other containers as much as possible. Singularity, on the other hand, assumes that the user’s primary goals are portability, reproducibility, and ease of use and that isolation is often a tertiary concern.
Therefore, Singularity only isolates the mount namespace by default, and will bind mount several host directories such as $HOME and /tmp into the container at runtime. If needed, additional levels of isolation can be achieved by passing options causing Singularity to enter any or all of the other kernel namespaces and to prevent automatic bind mounting. These measures allow users to interact with the host system from within the container in sensible ways.

(taken from https://sylabs.io/guides/3.7/user-guide/security.html )

I see a potential problem here, since singularity tries to incorporate the HOST files as part of the container in a transparent way, this could cause problems if crunch-run is running everything with the same user and maybe in a shared environment

#19 Updated by Nico César about 1 month ago

  • Description updated (diff)

#20 Updated by Nico César 24 days ago

fccb8a3de 17241-singularity-take1

Tom let's review this branch before I continue down the wrong path

#21 Updated by Nico César 18 days ago

  • Target version changed from 2021-03-31 sprint to 2021-04-14 sprint

#22 Updated by Nico César 4 days ago

43b39915bae3f3c24ab31cfbc7aefdce88f84dcb branch 17296-singularity-take2

we had a conversation with tom and we agree that lib/crunchrun/container_exec.go looks good to start making lib/crunchrun/container_exec_test.go to add all the tests for a contarner exec.

#23 Updated by Peter Amstutz 4 days ago

  • Target version changed from 2021-04-14 sprint to 2021-05-12 sprint

Also available in: Atom PDF