Project

General

Profile

Actions

Bug #19081

open

Possible bug passing cmd line arguments with spaces to singularity

Added by Peter Amstutz 2 months ago. Updated 5 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
Start date:
05/17/2022
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Customer reported a job that worked correctly with Docker runtime, did not work with the Singularity runtime.

The command line looked like this:

["/bin/bash", "-c", "command1 --option1 --option2"]

However it acts is if it were invoked as

/bin/bash -c command1

or possibly

/bin/bash -c command1 --option1 --option2

Further bolstering this hypothesis, the workaround was to not run it as a shell command (which seems to have been unnecessary, anyway), this worked as expected:

["command1", "--option1", "--option2"]


Subtasks 1 (0 open1 closed)

Task #19120: Review 19081-singularity-no-evalResolvedPeter Amstutz05/17/2022

Actions

Related issues

Related to Arvados - Bug #18765: engine configuration too big > 1048448 with singularityNew

Actions
Actions #1

Updated by Peter Amstutz 2 months ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz 2 months ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Actions #4

Updated by Tom Clegg about 2 months ago

  • Assigned To set to Tom Clegg
Actions #5

Updated by Tom Clegg about 2 months ago

  • Status changed from New to In Progress

Singularity .sif images include a runscript that assembles Entrypoint, Cmd, and the arguments given at runtime. It mangles the arguments by wrapping each in double quotes and then passing them to eval, which produces entertaining results.

$ sh -c 'echo "hello world"'
hello world
$ singularity exec docker://debian:11 sh -c 'echo "hello world"'
hello world
$ singularity run docker://debian:11 sh -c 'echo "hello world"'
hello

...because the runscript calls eval on this:

set "sh" "-c" "echo "hello world"" 

...which sets $1 to 'sh', $2 to '-c', $3 to 'echo hello', and $4 to 'world', so exec "$@" is equivalent to

sh -c 'echo hello' world

which, naturally, means execute the shell script 'echo hello' with $0 set to 'world'.

Escaping a shell script to survive this transformation is a fun exercise.

$ sh -c 'foo=bar; echo \$foo'
$foo
$ sh -c 'foo=bar; echo "\$foo"'
$foo
$ singularity run docker://debian:11 sh -c 'foo=bar; echo \$foo'
bar
$ singularity run docker://debian:11 sh -c 'foo=bar; echo "\$foo"'
bar
$ singularity run docker://debian:11 sh -c 'foo=bar; echo \\\$foo'
bar
$ singularity run docker://debian:11 sh -c 'foo=bar; echo "\\\$foo"'
$foo

$ sh -c 'echo \"hello world\"'
"hello world" 
$ singularity run docker://debian:11 sh -c 'echo \"hello world\"'
hello world
$ singularity run docker://debian:11 sh -c 'echo "\\\""hello world"\\\""'
"hello world" 

I haven't been able to reproduce any of this with "singularity exec" -- tried 3.7.4 and 3.9.9, tried Dockerfile with ENTRYPOINT, tried Dockerfile with CMD -- only with "singularity run". And crunch-run has never used "singularity run", only "singularity exec". So I don't see how this could have mangled an Arvados container command line.

I tried this on 9tee4 (9tee4-xvhdp-p4nvkhpec2vqpfz) and it worked correctly, producing "foo bar\nfoo bar\nfoo bar\n":

"command":["/bin/bash","-c","echo foo bar; echo \"foo bar\"; echo foo bar"]

Meanwhile, environment variables also get mangled (shell.EscapeDoubleQuotes()) and evaluated by the shell, instead of being passed through literally, even by "singularity exec":

$ echo $("whoami")
tom
$ FOO='$("whoami")' sh -c 'echo "$FOO"'
$("whoami")
$ SINGULARITYENV_FOO='$("whoami")' singularity exec docker://debian:11 sh -c 'echo "$FOO"'
"\"whoami\"": executable file not found in $PATH
$ SINGULARITYENV_FOO='$("whoami")' singularity exec docker://debian:11 sh -c 'echo hello world'
"\"whoami\"": executable file not found in $PATH
hello world
$ SINGULARITYENV_FOO='$(uname >&2)' singularity exec docker://debian:11 sh -c 'echo hello world'
Linux
hello world

(yikes)

Future versions of singularity will look for SINGULARITY_NO_EVAL=1 env var and use "OCI compatible" mode, i.e., don't mangle arguments or env vars (https://github.com/sylabs/singularity/pull/704). I think crunch-run should always use that mode.

Given all this, despite the description being pretty clear, I'm wondering whether the bug report could have come from a "singularity run" experience rather than actually submitting to Arvados...?

Or, perhaps "singularity exec" mangles command lines in a more subtle way -- can we find out the exact container command?

Actions #6

Updated by Tom Clegg about 2 months ago

Addressing the mangled environment variable issue I happened to notice while investigating the mangled command issue:

19081-singularity-no-eval @ f7954ab0a45cbc302aa07fa60697363895395dde -- developer-run-tests: #3143

This won't have any effect right away, but future releases of Singularity will notice the SINGULARITY_NO_EVAL env var, and pass environment variables into the container without eval/mangling them.

Actions #7

Updated by Peter Amstutz about 2 months ago

Tom Clegg wrote:

Addressing the mangled environment variable issue I happened to notice while investigating the mangled command issue:

19081-singularity-no-eval @ f7954ab0a45cbc302aa07fa60697363895395dde -- developer-run-tests: #3143

This won't have any effect right away, but future releases of Singularity will notice the SINGULARITY_NO_EVAL env var, and pass environment variables into the container without eval/mangling them.

This LGTM.

Actions #8

Updated by Peter Amstutz about 1 month ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Actions #9

Updated by Peter Amstutz about 1 month ago

  • Release set to 51
Actions #10

Updated by Tom Clegg 26 days ago

  • Related to Bug #18765: engine configuration too big > 1048448 with singularity added
Actions #11

Updated by Tom Clegg 25 days ago

  • Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint
Actions #12

Updated by Peter Amstutz 11 days ago

Peter to try this on customer cluster with original command line with leading "echo"

Actions #13

Updated by Peter Amstutz 11 days ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06
Actions

Also available in: Atom PDF