Arvados

Singularity .sif images include a runscript that assembles Entrypoint, Cmd, and the arguments given at runtime. It mangles the arguments by wrapping each in double quotes and then passing them to eval, which produces entertaining results.

$ sh -c 'echo "hello world"'
hello world
$ singularity exec docker://debian:11 sh -c 'echo "hello world"'
hello world
$ singularity run docker://debian:11 sh -c 'echo "hello world"'
hello

...because the runscript calls eval on this:

set "sh" "-c" "echo "hello world"" 

...which sets $1 to 'sh', $2 to '-c', $3 to 'echo hello', and $4 to 'world', so exec "$@" is equivalent to

sh -c 'echo hello' world

which, naturally, means execute the shell script 'echo hello' with $0 set to 'world'.

Escaping a shell script to survive this transformation is a fun exercise.

$ sh -c 'foo=bar; echo \$foo'
$foo
$ sh -c 'foo=bar; echo "\$foo"'
$foo
$ singularity run docker://debian:11 sh -c 'foo=bar; echo \$foo'
bar
$ singularity run docker://debian:11 sh -c 'foo=bar; echo "\$foo"'
bar
$ singularity run docker://debian:11 sh -c 'foo=bar; echo \\\$foo'
bar
$ singularity run docker://debian:11 sh -c 'foo=bar; echo "\\\$foo"'
$foo

$ sh -c 'echo \"hello world\"'
"hello world" 
$ singularity run docker://debian:11 sh -c 'echo \"hello world\"'
hello world
$ singularity run docker://debian:11 sh -c 'echo "\\\""hello world"\\\""'
"hello world" 

I haven't been able to reproduce any of this with "singularity exec" -- tried 3.7.4 and 3.9.9, tried Dockerfile with ENTRYPOINT, tried Dockerfile with CMD -- only with "singularity run". And crunch-run has never used "singularity run", only "singularity exec". So I don't see how this could have mangled an Arvados container command line.

I tried this on 9tee4 (9tee4-xvhdp-p4nvkhpec2vqpfz) and it worked correctly, producing "foo bar\nfoo bar\nfoo bar\n":

"command":["/bin/bash","-c","echo foo bar; echo \"foo bar\"; echo foo bar"]

Meanwhile, environment variables also get mangled (shell.EscapeDoubleQuotes()) and evaluated by the shell, instead of being passed through literally, even by "singularity exec":

$ echo $("whoami")
tom
$ FOO='$("whoami")' sh -c 'echo "$FOO"'
$("whoami")
$ SINGULARITYENV_FOO='$("whoami")' singularity exec docker://debian:11 sh -c 'echo "$FOO"'
"\"whoami\"": executable file not found in $PATH
$ SINGULARITYENV_FOO='$("whoami")' singularity exec docker://debian:11 sh -c 'echo hello world'
"\"whoami\"": executable file not found in $PATH
hello world
$ SINGULARITYENV_FOO='$(uname >&2)' singularity exec docker://debian:11 sh -c 'echo hello world'
Linux
hello world

(yikes)

Future versions of singularity will look for SINGULARITY_NO_EVAL=1 env var and use "OCI compatible" mode, i.e., don't mangle arguments or env vars (https://github.com/sylabs/singularity/pull/704). I think crunch-run should always use that mode.

Given all this, despite the description being pretty clear, I'm wondering whether the bug report could have come from a "singularity run" experience rather than actually submitting to Arvados...?

Or, perhaps "singularity exec" mangles command lines in a more subtle way -- can we find out the exact container command?