Bug #5790
closed
[SDKs] arv-copy does not select the correct Docker image to copy
Added by Brett Smith about 9 years ago.
Updated almost 9 years ago.
Description
A user just reported a bug where arv-copy was failing trying to copy a nonexistent collection UUID while copying a pipeline template. This collection was a Docker image that had since been deleted and replaced. Docker metadata links were still hanging around pointing to this collection, and apparently arv-copy followed them to try to copy it instead of the replacement collection.
Given this scenario, and that arv-copy worked after deleting the links, it seems likely that arv-copy is not using the appropriate metadata to decide which image to copy (the same logic in API server's Collection.find_all_for_docker_image).
- Target version changed from Bug Triage to 2015-06-10 sprint
- Assigned To set to Peter Amstutz
- Assigned To changed from Peter Amstutz to Brett Smith
5790-copy-most-recent-docker-image-wip is up for review. It ends up revamping the Docker image list function in arv-keepdocker, because that's what arv-copy calls to find what it should copy. I ended up fixing several bugs while I was here. Relevant to arv-copy:
- The original sorting bug: the code previously used created_at a sort key when image_timestamp was not available. This did not match the API server's behavior, which is to follow links with any image_timestamp before ones without.
- Also like API server, ignore links that point to unknown collections.
- Always fetch all Docker links, not just the first batch returned by the API server.
- Docker images can be referred to by Docker image hash, so I added that search functionality to the list function.
Some listing improvements:
- Show image hashes when listing images by name.
- Like Docker itself, when an image has multiple names and we're not filtering by name, list each one.
- Status changed from New to In Progress
I finally checked and tested out.
This is a comparison of the output from master and the branch :
$ arv-keepdocker
REPOSITORY TAG IMAGE ID COLLECTION CREATED
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa latest d8309758b8fe zzzzz-4zz18-1v45jub259sjjgb Tue Jun 10 14:30:00 2014
arvados/apitestfixture latest d8309758b8fe zzzzz-4zz18-t68oksiu9m80s4y Thu Jun 10 14:30:00 2010
$ arv-keepdocker
REPOSITORY TAG IMAGE ID COLLECTION CREATED
arvados/apitestfixture latest d8309758b8fe zzzzz-4zz18-1v45jub259sjjgb Tue Jun 10 14:30:00 2014
arvados/apitestfixture june10 d8309758b8fe zzzzz-4zz18-1v45jub259sjjgb Tue Jun 10 14:30:00 2014
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa latest d8309758b8fe zzzzz-4zz18-1v45jub259sjjgb Tue Jun 10 14:30:00 2014
arvados/apitestfixture latest d8309758b8fe zzzzz-4zz18-t68oksiu9m80s4y Thu Jun 10 14:30:00 2010
this looks good to me. merge it!
- Status changed from In Progress to Resolved
- % Done changed from 50 to 100
Applied in changeset arvados|commit:e8ae9364dce380d305833f35dfb25578175664d7.
- Status changed from Resolved to In Progress
Looks like this change broke arv keep docker on 4xphq:
shell.4xphq:~$ arv keep docker
REPOSITORY TAG IMAGE ID COLLECTION CREATED
Traceback (most recent call last):
File "/usr/local/bin/arv-keepdocker", line 4, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/keepdocker.py", line 291, in main
for i, j in list_images_in_arv(api, args.retries):
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/keepdocker.py", line 241, in list_images_in_arv
filters=search_filters + [['link_class', '=', 'docker_image_hash']])
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/keepdocker.py", line 187, in _get_docker_links
links.sort(key=itemgetter('_sort_key'), reverse=True)
TypeError: can't compare datetime.datetime to int
If I comment out line 187 (and also line 274 where the same sort is done), then it works again.
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:679ea52a117414fc33768aac887615f9b701fdf8.
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by 
Updated by 
Updated by