Arvados

As part of testing in #17755, it appears that there is a problem with /tmp (?) not being writable in our singularity containers. a-c-r relies on this, and many other tools will likely do too. It seems a bit unreasonable for /tmp to be unwritable, if that is really the problem. Maybe we can change something about the way we are invoking singularity?

See e.g. ce8i5-xvhdp-1ubjhuo87i24ora:

2021-06-16T20:15:21.003947318Z INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20210610215458, arvados-python-client 2.3.0.dev20210610215458, cwltool 3.0.20210319143721
2021-06-16T20:15:21.022483370Z INFO Resolved '/var/lib/cwl/workflow.json#main' to 'file:///var/lib/cwl/workflow.json#main'
2021-06-16T20:15:23.231706231Z INFO Using cluster ce8i5 (https://workbench2.ce8i5.arvadosapi.com/)
2021-06-16T20:15:30.386974208Z INFO Using collection cache size 256 MiB
2021-06-16T20:15:30.448702283Z INFO Running inside container ce8i5-dz642-9asvb2g41z514n8
2021-06-16T20:15:30.665440997Z INFO [workflow workflow.json#main] start
2021-06-16T20:15:30.666220299Z INFO [workflow workflow.json#main] starting step substep
2021-06-16T20:15:30.666896701Z INFO [step substep] start
2021-06-16T20:15:31.473008286Z WARNING X-Keep-Storage-Classes header not supported by the cluster
2021-06-16T20:15:31.700830931Z INFO Using collection ce8i5-4zz18-df50zijeqkpbdaf
2021-06-16T20:15:36.635135414Z ERROR Unexpected exception
2021-06-16T20:15:36.635135414Z Traceback (most recent call last):
2021-06-16T20:15:36.635135414Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 436, in job
2021-06-16T20:15:36.635135414Z     runtimeContext,
2021-06-16T20:15:36.635135414Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 964, in job
2021-06-16T20:15:36.635135414Z     j.stagedir = runtimeContext.create_tmpdir()
2021-06-16T20:15:36.635135414Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/context.py", line 181, in create_tmpdir
2021-06-16T20:15:36.635135414Z     return tempfile.mkdtemp(prefix=tmp_prefix, dir=tmp_dir)
2021-06-16T20:15:36.635135414Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/tempfile.py", line 505, in mkdtemp
2021-06-16T20:15:36.635135414Z     _os.mkdir(file, 0o700)
2021-06-16T20:15:36.635135414Z OSError: [Errno 30] Read-only file system: 'tmpph0scv23'
2021-06-16T20:15:36.831953372Z ERROR Cannot make scatter job: [Errno 30] Read-only file system: 'tmpph0scv23'
2021-06-16T20:15:37.120042388Z INFO [step substep] start
2021-06-16T20:15:37.240278629Z ERROR Unexpected exception
2021-06-16T20:15:37.240278629Z Traceback (most recent call last):
2021-06-16T20:15:37.240278629Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 436, in job
2021-06-16T20:15:37.240278629Z     runtimeContext,
2021-06-16T20:15:37.240278629Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 964, in job
2021-06-16T20:15:37.240278629Z     j.stagedir = runtimeContext.create_tmpdir()
2021-06-16T20:15:37.240278629Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/context.py", line 181, in create_tmpdir
2021-06-16T20:15:37.240278629Z     return tempfile.mkdtemp(prefix=tmp_prefix, dir=tmp_dir)
2021-06-16T20:15:37.240278629Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/tempfile.py", line 505, in mkdtemp
2021-06-16T20:15:37.240278629Z     _os.mkdir(file, 0o700)
2021-06-16T20:15:37.240278629Z OSError: [Errno 30] Read-only file system: 'tmpi4ayzhq8'
2021-06-16T20:15:37.383785035Z ERROR Cannot make scatter job: [Errno 30] Read-only file system: 'tmpi4ayzhq8'
2021-06-16T20:15:37.546091895Z INFO [step substep] start
2021-06-16T20:15:37.660351019Z ERROR Unexpected exception
2021-06-16T20:15:37.660351019Z Traceback (most recent call last):
2021-06-16T20:15:37.660351019Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 436, in job
2021-06-16T20:15:37.660351019Z     runtimeContext,
2021-06-16T20:15:37.660351019Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 964, in job
2021-06-16T20:15:37.660351019Z     j.stagedir = runtimeContext.create_tmpdir()
2021-06-16T20:15:37.660351019Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/context.py", line 181, in create_tmpdir
2021-06-16T20:15:37.660351019Z     return tempfile.mkdtemp(prefix=tmp_prefix, dir=tmp_dir)
2021-06-16T20:15:37.660351019Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/tempfile.py", line 505, in mkdtemp
2021-06-16T20:15:37.660351019Z     _os.mkdir(file, 0o700)
2021-06-16T20:15:37.660351019Z OSError: [Errno 30] Read-only file system: 'tmpybc3dv2f'
2021-06-16T20:15:37.867057605Z ERROR Cannot make scatter job: [Errno 30] Read-only file system: 'tmpybc3dv2f'
2021-06-16T20:15:37.985364440Z INFO [step substep] start
2021-06-16T20:15:38.084313721Z ERROR Unexpected exception
2021-06-16T20:15:38.084313721Z Traceback (most recent call last):
2021-06-16T20:15:38.084313721Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 436, in job
2021-06-16T20:15:38.084313721Z     runtimeContext,
2021-06-16T20:15:38.084313721Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 964, in job
2021-06-16T20:15:38.084313721Z     j.stagedir = runtimeContext.create_tmpdir()
2021-06-16T20:15:38.084313721Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/context.py", line 181, in create_tmpdir
2021-06-16T20:15:38.084313721Z     return tempfile.mkdtemp(prefix=tmp_prefix, dir=tmp_dir)
2021-06-16T20:15:38.084313721Z   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/tempfile.py", line 505, in mkdtemp
2021-06-16T20:15:38.084313721Z     _os.mkdir(file, 0o700)
2021-06-16T20:15:38.084313721Z OSError: [Errno 30] Read-only file system: 'tmp8_atoptz'
2021-06-16T20:15:38.254878304Z ERROR Cannot make scatter job: [Errno 30] Read-only file system: 'tmp8_atoptz'
2021-06-16T20:15:38.424077683Z WARNING [step substep] completed permanentFail
2021-06-16T20:15:38.585621241Z INFO [workflow workflow.json#main] completed permanentFail
2021-06-16T20:15:38.585821742Z ERROR Overall process status is permanentFail