Bug #14880

arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wrong

Added by Julien Fouret 8 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
03/08/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Hi,

I am trying to set up an arvados instance in our institution. I am having trouble to use the arv-keepdocker when setting up a compute node.

All previous step from "install" part of the documentation were OK. For information, arvados is intended to be tested on a private network so keep-proxy was not necessary.

I had some trouble with using keep storage through python SDK because I choose not to install keep-proxy for now and httplib2 from python was not parsing fine my "no_proxy" environment variable (see https://github.com/httplib2/httplib2/issues/132).

I got an error in the following step (using super_user API token from the computing node I am trying to set up (id206)):

`arv-keepdocker --project-uuid $project_uuid --pull arvados/jobs latest`

```
latest: Pulling from arvados/jobs
3d77ce4481b1: Already exists
e9d5d7a7ec48: Already exists
0275f7e96071: Already exists
e5c7ad7bde76: Already exists
354b6c89659e: Pull complete
2946e252f657: Pull complete
a58f890f60d4: Pull complete
803ff01c62a8: Pull complete
Digest: sha256:47dadb94b92346cf6adad2e11cef84331e7ee07fcef5630e12ab1cd9297938c6
Status: Downloaded newer image for arvados/jobs:latest
usage: arv-keepdocker [-h] [--version] [--normalize | --dry-run]
[--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw]
[--update-collection UUID] [--use-filename FILENAME]
[--filename FILENAME] [--portable-data-hash]
[--replication N] [--storage-classes STORAGE_CLASSES]
[--threads N] [--project-uuid UUID] [--name NAME]
[--exclude PATTERN]
[--progress | --no-progress | --batch-progress]
[--silent] [--resume | --no-resume]
[--follow-links | --no-follow-links]
[--cache | --no-cache] [--retries RETRIES]
[path [path ...]]
arv-keepdocker: error: unrecognized arguments: /home/default_admin/.cache/arvados/docker/sha256:a9171d8a433188c522dfe94938afa1b8cee91635f6d281e52595e1ee09a8447c.tar
```

For information here are our DNS record (where I have replaced our institution domain name by example.org) for nodes that should be involved in the process:

core-pfx.example.org > API server
keep0-pfx.example.org > keepstore server
id206.example.org ==> compute node

This is not your convention but the DNS team for our institution does not want us to create other domains if not explicitly required by the app we deploy. We have self-signed certificate with the appropriate root and intermediate certificates installed in our servers and workstation to enable ssl authentication.

Of note `arv-put` `arv-get` are working fine on `id206.example.org`.

here is the output of `$docker version` on `id206.example.org`

```
Client:
Version: 18.09.2
API version: 1.39
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:47 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 03:42:13 2019
OS/Arch: linux/amd64
Experimental: false
```

All OS are ubuntu 18.04 and I am using this repository for arvados `deb http://apt.arvados.org/ bionic main`

Arvados version is 1.3.0.20190211211047

Have you ever seen this error ? Do you have any idea of what is causing this ?

Thanks,

system.info (1.5 KB) system.info Julien Fouret, 02/25/2019 09:26 AM

Subtasks

Task #14899: Review 14880-keepdocker-argparseResolvedPeter Amstutz

Associated revisions

Revision 61a02d59 (diff)
Added by Ward Vandewege 8 months ago

Change argument order for arv-keepdocker to the one that works, as it is
currently sensitive to the order.

refs #14880

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision fb9f29b3
Added by Peter Amstutz 7 months ago

Merge branch '14880-keepdocker-argparse' refs #14880

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Julien Fouret 8 months ago

The error happens at this line from keepdocker.py file

coll_uuid = arv_put.main(
            put_args + ['--filename', outfile_name, image_file.name], stdout=stdout,
            install_sig_handlers=install_sig_handlers).strip()

with

>>>print put_args
['--project-uuid', 'arvados/jobs', 'latest', '--name', 'Docker image arvados/jobs:latest sha256:a9171']

and

image_file.name: /home/default_admin/.cache/arvados/docker/sha256:a9171d8a433188c522dfe94938afa1b8cee91635f6d281e52595e1ee09a8447c.tar
install_sig_handlers: true
outfile_name: sha256:a9171d8a433188c522dfe94938afa1b8cee91635f6d281e52595e1ee09a8447c.tar

It seems that the project UUID is not present in `put_args` list whereas it should be ?

#2 Updated by Julien Fouret 8 months ago

Finally,

Still with the command "arv-keepdocker --project-uuid "id241-j7d0g-54qgzcohscf9k1v" --pull arvados/jobs latest", as written in the documentation

I got :

>>>print keepdocker_parser.parse_known_args(arguments)
(Namespace(force=False, force_image_format=False, image='id241-j7d0g-54qgzcohscf9k1v', pull=True, tag=None), ['--project-uuid', 'arvados/jobs', 'latest'])

... weird

Thus, I tried "arv-keepdocker arvados/jobs latest --project-uuid "id241-j7d0g-54qgzcohscf9k1v" --pull"

and I got:

>>>print keepdocker_parser.parse_known_args(arguments)
(Namespace(force=False, force_image_format=False, image='arvados/jobs', pull=True, tag='latest'), ['--project-uuid', 'id241-j7d0g-54qgzcohscf9k1v'])

and no error!

It seems there was a problem parsing the arguments (a list of my system info is attached), I won't look any further as it is working by switching arguments order.

Bests,

#3 Updated by Ward Vandewege 8 months ago

  • Subject changed from setting compute node according to doc : "arv-keepdocker: error" to arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wrong

Hi Julien,

Thanks for this bug report. I can reproduce this bug. We will look into fixing it (patches are welcome, of course), and in the mean time I have updated the documentation so that the argument order suggested there is the one that works.

$ arv-keepdocker --project-uuid qr1hi-j7d0g-6ljkv8ab0cb6ot0 --pull cure/arvados-runtime latest
latest: Pulling from cure/arvados-runtime
7b722c1070cd: Pull complete 
5fbf74db61f1: Pull complete 
ed41cb72e5c9: Pull complete 
7ea47a67709e: Pull complete 
703661372883: Pull complete 
8664f41efa89: Pull complete 
e4ff173ab779: Pull complete 
ea495f19c483: Pull complete 
Digest: sha256:63c0c7bee37f932f8a68d5d7eda9f7b7a2cb8149dbd07e1766866974273760c3
Status: Downloaded newer image for cure/arvados-runtime:latest
usage: arv-keepdocker [-h] [--version] [--normalize | --dry-run]
                      [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw]
                      [--update-collection UUID] [--use-filename FILENAME]
                      [--filename FILENAME] [--portable-data-hash]
                      [--replication N] [--storage-classes STORAGE_CLASSES]
                      [--threads N] [--project-uuid UUID] [--name NAME]
                      [--exclude PATTERN]
                      [--progress | --no-progress | --batch-progress]
                      [--silent] [--resume | --no-resume]
                      [--follow-links | --no-follow-links]
                      [--cache | --no-cache] [--retries RETRIES]
                      [path [path ...]]
arv-keepdocker: error: unrecognized arguments: /home/wvandewege/.cache/arvados/docker/sha256:1f05b3c98bdd0dd48373942257ff15a1aec1d886deaf8a18ac1ec7a07da9c1bd.tar
$ arv-keepdocker --pull cure/arvados-runtime latest --project-uuid qr1hi-j7d0g-6ljkv8ab0cb6ot0
latest: Pulling from cure/arvados-runtime
Digest: sha256:63c0c7bee37f932f8a68d5d7eda9f7b7a2cb8149dbd07e1766866974273760c3
Status: Image is up to date for cure/arvados-runtime:latest
2019-02-27 13:38:10 arvados.arv_put[26443] INFO: Creating new cache file at /home/wvandewege/.cache/arvados/arv-put/b99012c27a1a3b3f0d01969f07cf17ac
139M / 139M 100.0% 2019-02-27 13:38:17 arvados.arv_put[26443] INFO: 

2019-02-27 13:38:17 arvados.arv_put[26443] INFO: Collection saved as 'Docker image cure/arvados-runtime:latest sha256:1f05b'
qr1hi-4zz18-l4995i7dcwldpzi

#4 Updated by Tom Morris 8 months ago

  • Target version set to 2019-03-13 Sprint

#5 Updated by Peter Amstutz 8 months ago

  • Assigned To set to Peter Amstutz

#6 Updated by Tom Morris 8 months ago

  • Release set to 15

#8 Updated by Peter Amstutz 7 months ago

14880-keepdocker-argparse @ b26b8ced25b767648f493a48956a7b40d95e1b20

arv-keepdocker passes through some flags to arv-put. The strategy it
uses is to remove known flags and pass through unknown flags. However
it doesn't know if an unknown flag takes a value or not, so it treats
the values as a positional argument rather than the value
corresponding to the unknown flag.

  • Move positional arguments from "keepdocker_parser" to "arg_parser"
  • Strip positional arguments (image name, tag) from arguments before
    parsing known keepdocker_parser flags.
  • Additionally move a couple of flags that were mistakenly added to
    put.run_opts (and thus appearing in the arv-keepdocker help despite
    making no sense) that should have been in put.upload_opts

https://ci.curoverse.com/job/run-tests/2580/

#9 Updated by Eric Biagiotti 7 months ago

Peter Amstutz wrote:

14880-keepdocker-argparse @ b26b8ced25b767648f493a48956a7b40d95e1b20

arv-keepdocker passes through some flags to arv-put. The strategy it
uses is to remove known flags and pass through unknown flags. However
it doesn't know if an unknown flag takes a value or not, so it treats
the values as a positional argument rather than the value
corresponding to the unknown flag.

  • Move positional arguments from "keepdocker_parser" to "arg_parser"
  • Strip positional arguments (image name, tag) from arguments before
    parsing known keepdocker_parser flags.
  • Additionally move a couple of flags that were mistakenly added to
    put.run_opts (and thus appearing in the arv-keepdocker help despite
    making no sense) that should have been in put.upload_opts

https://ci.curoverse.com/job/run-tests/2580/

Tiny typo in put.py line 163. Missing 'it'. And tests failed. Other than that LGTM.

#10 Updated by Peter Amstutz 7 months ago

  • Status changed from New to Resolved

Also available in: Atom PDF