Arvados

Saving a docker image into keep using `arv keep docker` fails if there is not enough space in my home directory to `docker save` the image tar file.

The error given was:

$ arv keep docker mercury/test
0
Collection saved as 'Docker image mercury/test:latest 91897da3ec59'
7lnae-4zz18-rr3k3lmqavbwd76
Traceback (most recent call last):
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20150827-f2afe64/arvados-python-venv/bin/arv-keepdocker", line 4, in <module>
import('pkg_resources').run_script('arvados-python-client==0.1.20150818124406', 'arv-keepdocker')
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20150827-f2afe64/arvados-python-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 652, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20150827-f2afe64/arvados-python-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1573, in run_script
exec(code, namespace, namespace)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20150827-f2afe64/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20150818124406-py2.7.egg/EGG-INFO/scripts/arv-keepdocker", line 4, in <module>
main()
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20150827-f2afe64/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20150818124406-py2.7.egg/arvados/commands/keepdocker.py", line 403, in main
image_tar = tarfile.open(fileobj=image_file)
File "/software/hgi/pkglocal/python-2.7.8-ucs4/lib/python2.7/tarfile.py", line 1665, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully

I ended up putting in some debug prints to figure out what tar file it is talking about and why it wasn't written.

I think `docker save` is partly to blame here, as it returns 0 regardless of whether the write fails (this is with docker v1.2 so perhaps it has since been fixed):

$ docker save 91897da3ec59ed61dd4aee3fbfe4881873ff93bf1a8a0dd3a3a2a254b18cae23 > ~/.cache/arvados/docker/foo.tar
$ echo $?
0
$ ls l ~/.cache/arvados/docker/foo.tar
-rw-r--r- 1 jr17 hgi 0 Sep 3 12:24 /nfs/users/nfs_j/jr17/.cache/arvados/docker/foo.tar

I think there are two separate issues here:
- it would be nice if `arv keep docker` checked whether `docker save` successfully created a tar file and gave an error regarding the specific location involved rather than giving a stack trace.
- it would be really good if the location used by these commands to create the temporary image was (a) configurable and (b) not in one's home directory by default -- wouldn't something under ${TMPDIR} (defaulting to "/tmp") be more appropriate as a default? Isn't this more of a temporary file than a "cache" file?

I have temporarily worked around this issue by symlinking ~/.cache/arvados/docker to a tmp dir with available space.