Arvados

Docs and interface¶

Not sure of the background wrt prioritizing this with thin requirements/expectations, but for starters, the story should be updated to say why "arv keep docker --download" is important even though "docker load < ~/keep/by_id/collectionuuid/dockerimage.tar" exists.

The branch needs some documentation for users, too, to explain when/how this is meant to be used. "Fetch docker image from Arvados and load locally" in the usage message is the extent of the docs I can find, and it's a bit terse. I think I understand what this is for, but even if I'm right, we need to tell our users about it, so they know it's there when they need it. Is there somewhere in /doc where it would be appropriate to mention this?

• Should we consider switching to the terms "get" and "put" here -- matching "arv put" and "arv get" -- instead of "download" and "upload", which tend to sound (at least to me) like they do things out there on the internet with non-Arvados entities?
• Or perhaps "arv keep docker load" instead of --download, since this is essentially "docker load" using Keep instead of POSIX?

• --pull --upload
• --no-pull --upload
• --download

• --pull --download (probably throw a usage error instead of just ignoring --pull and fetching from Keep?)
• --no-pull --download (usage error? or "of course --no-pull, ignoring that part"?)
• --pull --no-upload (I assume this would be easy, and we might as well allow it, so people can use the same tool for this use case too, instead of switching to regular "docker pull". OTOH I'm not keen on --pull --no-upload and switching to "docker pull" isn't that hard, so maybe leave this alone?)
• --pull --download (usage error, presumably? Can't think of an obvious meaning for this off hand.)
• --download --force (usage error? "force" seems to apply only on upload. Tangent: "force" looks like it was a bad choice here, since it's far from obvious which of the various conflicts will be "forced" through...)

• Leave the --image argument so it still just refers to a docker image
• Let the --tag argument continue to specify a tag to search for
• Allow specifying a collection with --collection X (we can reliably distinguish UUID from PDH here, but --uuid and --portable-data-hash might be worth considering as alternatives/synonyms)
• Avoid doing different searches depending on whether previous searches turned up any results. (If there's a use case for "I don't know what kind of ID this is but I do expect it to turn up a docker image" we should consider that feature, but surely it shouldn't be the only way to specify an image.)

Tests¶

Seems like we should be able to add something...

Code¶

Couple of initial observations:

The new list_images_in_arv() function signature is rather unwieldy. It accepts three image arguments (image_name, image_tag, and image_hash) but image_hash is ignored if image_name is provided, etc. The "if image_name or image_hash" part might be a clue this function is trying to do too many different things.

64000 and 1024*1024 should be named constants.

_group.add_argument('--upload') should probably say action='store_false', default=False, dest='download' if that's the way we're using the args.download flag. Surely the current code works because of the mutually exclusive group, but still, creating an args.upload flag that is either True or False but never gets read seems like asking for trouble/confusion later.

The usage messages for --tag and --image should be updated now that they're used for --download mode as well.

It looks like arv keep docker --download {collection_uuid} will fail "Docker image '%s' not found" if the collection isn't tagged as a docker image. It seems like it would be better to use it anyway, and either [a] treat the missing tag as a warning instead of an error, or [b] skip the tag lookup entirely, since the collection UUID already specifies exactly one (or zero) available image. This would also permit downloading a docker image using a portable_data_hash, which seems like a really useful feature for careful scripts/people to use.

It looks like arv keep docker --download --image X --tag Y looks in Arvados for an image called X (with any tag), downloads it, and then -- only if X was a collection uuid -- pushes tag Y to the upstream docker repository. Am I following correctly, and is this intentional? I would have guessed this command would restrict its search to images that are tagged with Y, which is rather different, and it seems especially surprising for the tagging behavior to change based on whether X was a collection UUID or a PDH or a docker image id.

I still find myself wanting this feature occasionally, although we may want to abandon this particular branch.