Actions
Idea #17296
closed
Singularity proof of concept
Start date:
05/27/2021
Due date:
Story points:
5.0
Release:
Release relationship:
Auto
Description
Given the following restrictions:
- singularity 3.5.2 installed on the compute node.
- config.yml has "ContainerExec=Singularity" "SingularityBinary=/usr/bin/singularity"
- no privileges, setuid or special linux capabilities are needed to run the container (i.e. ping won't work because it needs CAP_NET_RAW)
- The space required in the compute node is 2 * Docker Image size uncompressed. Since the singularity build step requires a file on disk for the docker-archive:// schema
- When we have a ContainerRequest,
- /usr/bin/singularity build .... to create the .sif file
- /usr/bin/singularity run ... to run the container using --bind for all the container mounts
- date in realtime the container StdErr and StdOut
- store the outputs in a collection
Before starting the support for singularity we need to define an interface for all Container Executors.
type ContainerExecuter interface {
// CheckImageIsLoaded checks if imageID is already in the local environment,
// either something we can reference later in Docker API or similar, or a
// filepath containing the image to be used
CheckImageIsLoaded(imageID ImageID) bool
// LoadImage translates a io.Reader that has a tarball in docker format
// (Created with 'docker save'), and load it to the local
// ContainerExecuter.
//
// Returns an ImageID to be referenced later, note that this could be an image
// identifier or a filepath or something else
LoadImage(containerImage io.Reader) (ImageID, error)
// ImageRemove removes the image loaded using LoadImage()
ImageRemove(imageID ImageID) error
ContainerState() (ContainerState, error)
// CreateContainer will prepare anything in the creation on the container
// env is an array of string "KEY=VALUE" that represents the environment variables
// containerConfig has all the parameters needed to start the container
CreateContainer(containerConfig ContainerConfig) error
// StartContainer will start the container
StartContainer() error
// Kill the container and optionally remove the underlying image returns an
// error if it didn't work (including timeout)
Kill() error
// this is how https://golang.org/pkg/os/exec/#Cmd does it.
StdinPipe() (io.WriteCloser, error)
StdoutPipe() (io.ReadCloser, error)
StderrPipe() (io.ReadCloser, error)
// Wait for the container to finish
Wait() error
// Returns the exit code of the last executed container
ExitCode() (int, error)
}
When we run a container on a compute node, we do a container conversion, on the fly, to a SIF file, and run that with singularity instead.
- global option that switches between docker or singularity runner
- container_request runtime parameters flag that chooses between docker and singularity
- crunch-run gets docker tar file from keep (existing docker v2 format images)
- crunch-run converts docker tar file to SIF:
$ docker save arvados/jobs:latest > arvados-jobs.latest.tar $ ls -laF arvados-jobs.latest.tar -rw-r--r-- 1 ward ward 295209984 Jan 14 17:16 arvados-jobs.latest.tar $ singularity build arvados-jobs.latest.sif docker-archive://arvados-jobs.latest.tar INFO: Starting build... ...
- crunch-run executes singularity with mount points, stdout/stderr captured to logs
- slurm dispatcher supports singularity
- ideally the backend container runner should be transparent to the dispatcher
- proof of concept will be tested on 9tee4
- assume that user id inside the container will be the same as the crunch-run user (?)
- try to support running containers without setuid, identify specific features that require setuid on singularity binary.
Testing goals / acceptance criteria
- MVP: runs a container
- default value for singularity binary (/usr/bin/singularity) but can be changed from arvados config.yml
- captures stdout/stderr to logs
- can bind-mount arv-mount inside the container
- can bind mount tmp/output directories inside the container
- output files have proper permissions to be read for upload & cleaned up (deleted) by crunch-run
- see if it makes sense to have singularity mock the docker API
- should have similar test coverage of singularity features as exist to the Docker features
For future tickets:
- crunchstat
- memory / CPU constraints
- Save the SIF file in Keep and do something with another Link object to make it findable in the future, for the corresponding docker image. TODO: check if the framework we built in for the docker image format v1 -> v2 could be used here.
Related issues
Updated by 
Updated by 
Updated by 
Updated by Javier Bértoli 
Updated by
Actions