Project

General

Profile

Actions

Idea #8568

closed

[SDKs] `arv keep docker` supports Docker 1.10+

Added by Brett Smith about 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
SDKs
Target version:
Story points:
1.0

Description

If you try to run arv keep docker today on a host running Docker 1.10+, you'll get an exception when the tool tries to find a specific file in the generated tar file:

$ arv keep docker --project-uuid=qr1hi-j7d0g-rerv400sudof0mn name
1474M / 1474M 100.0% 
Collection saved as 'Docker image name:latest sha256:... (2)'
qr1hi-4zz18-k4b79jr1uamnila
Traceback (most recent call last):
  File "/usr/local/bin/arv-keepdocker", line 5, in <module>
    pkg_resources.run_script('arvados-python-client==0.1.20160128214108', 'arv-keepdocker')
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 517, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 1436, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/arvados_python_client-0.1.20160128214108-py2.7.egg/EGG-INFO/scripts/arv-keepdocker", line 4, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/arvados_python_client-0.1.20160128214108-py2.7.egg/arvados/commands/keepdocker.py", line 401, in main
    json_file = image_tar.extractfile(image_tar.getmember(image_hash + '/json'))
  File "/usr/lib/python2.7/tarfile.py", line 1800, in getmember
    raise KeyError("filename %r not found" % name)
KeyError: "filename 'sha256:...d2b6/json' not found"

This happens because Docker changed the image tar format in Docker 1.10. Recognize and correctly upload images generated by Docker 1.10.

Changes to make

The output of docker images --no-trunc has changed so the image hash now includes the cryptographic hash name. It looks like sha256:4fe79ae514594715386c226e89a53c8037e081603f93f65a47c82573359f70cb.

Right now arv-keepdocker looks for a JSON metadata file in the .tar under <image hash>/json:

json_file = image_tar.extractfile(image_tar.getmember(image_hash + '/json'))

This needs to be updated to look for a file named <unmarked hash>.json. So, in the example hash above, arv-keepdocker needs to look for a metadata file named 4fe79ae514594715386c226e89a53c8037e081603f93f65a47c82573359f70cb.json.

Suggest finding the metadata file with something like the following:

image_hash_type, _, raw_image_hash = image_hash.rpartition(':')
if image_hash_type:
    json_filename = raw_image_hash + '.json'
else:
    json_filename = raw_image_hash + '/json'
json_file = image_tar.extractfile(image_tar.getmember(json_filename))

I have confirmed that the JSON file still has the information arv-keepdocker needs in the new image format.


Subtasks 1 (0 open1 closed)

Task #9641: Review branch 8568-docker-version-supportResolvedRadhika Chippada02/26/2016Actions

Related issues

Related to Arvados - Support #9628: CWL Tutorial Does not RunResolvedTom Morris07/18/2016Actions
Actions

Also available in: Atom PDF