https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422019-02-25T08:40:12ZArvadosArvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=716752019-02-25T08:40:12ZJulien Fouret
<ul></ul><p>The error happens at this line from keepdocker.py file</p>
<pre><code class="python syntaxhl"><span class="n">coll_uuid</span> <span class="o">=</span> <span class="n">arv_put</span><span class="p">.</span><span class="n">main</span><span class="p">(</span>
<span class="n">put_args</span> <span class="o">+</span> <span class="p">[</span><span class="s">'--filename'</span><span class="p">,</span> <span class="n">outfile_name</span><span class="p">,</span> <span class="n">image_file</span><span class="p">.</span><span class="n">name</span><span class="p">],</span> <span class="n">stdout</span><span class="o">=</span><span class="n">stdout</span><span class="p">,</span>
<span class="n">install_sig_handlers</span><span class="o">=</span><span class="n">install_sig_handlers</span><span class="p">).</span><span class="n">strip</span><span class="p">()</span>
</code></pre>
<p>with</p>
<pre><code class="python syntaxhl"><span class="o">>>></span><span class="k">print</span> <span class="n">put_args</span>
<span class="p">[</span><span class="s">'--project-uuid'</span><span class="p">,</span> <span class="s">'arvados/jobs'</span><span class="p">,</span> <span class="s">'latest'</span><span class="p">,</span> <span class="s">'--name'</span><span class="p">,</span> <span class="s">'Docker image arvados/jobs:latest sha256:a9171'</span><span class="p">]</span>
</code></pre>
<p>and</p>
<pre><code class="yaml syntaxhl"><span class="na">image_file.name</span><span class="pi">:</span> <span class="s">/home/default_admin/.cache/arvados/docker/sha256:a9171d8a433188c522dfe94938afa1b8cee91635f6d281e52595e1ee09a8447c.tar</span>
<span class="na">install_sig_handlers</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">outfile_name</span><span class="pi">:</span> <span class="s">sha256:a9171d8a433188c522dfe94938afa1b8cee91635f6d281e52595e1ee09a8447c.tar</span>
</code></pre>
<p>It seems that the project UUID is not present in `put_args` list whereas it should be ?</p> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=716762019-02-25T09:27:39ZJulien Fouret
<ul><li><strong>File</strong> <a href="/attachments/2351">system.info</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2351/system.info">system.info</a> added</li></ul><p>Finally,</p>
<p>Still with the command "arv-keepdocker --project-uuid "<a href="https://arvadosapi.com/id241-j7d0g-54qgzcohscf9k1v">id241-j7d0g-54qgzcohscf9k1v</a>" --pull arvados/jobs latest", <ins><strong>as written in the documentation</strong></ins></p>
<p>I got :</p>
<pre><code class="python syntaxhl"><span class="o">>>></span><span class="k">print</span> <span class="n">keepdocker_parser</span><span class="p">.</span><span class="n">parse_known_args</span><span class="p">(</span><span class="n">arguments</span><span class="p">)</span>
<span class="p">(</span><span class="n">Namespace</span><span class="p">(</span><span class="n">force</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">force_image_format</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">image</span><span class="o">=</span><span class="s">'<a href="https://arvadosapi.com/id241-j7d0g-54qgzcohscf9k1v">id241-j7d0g-54qgzcohscf9k1v</a>'</span><span class="p">,</span> <span class="n">pull</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">tag</span><span class="o">=</span><span class="bp">None</span><span class="p">),</span> <span class="p">[</span><span class="s">'--project-uuid'</span><span class="p">,</span> <span class="s">'arvados/jobs'</span><span class="p">,</span> <span class="s">'latest'</span><span class="p">])</span>
</code></pre>
<p>... weird</p>
<p>Thus, I tried "arv-keepdocker arvados/jobs latest --project-uuid "<a href="https://arvadosapi.com/id241-j7d0g-54qgzcohscf9k1v">id241-j7d0g-54qgzcohscf9k1v</a>" --pull"</p>
<p>and I got:</p>
<pre><code class="python syntaxhl"><span class="o">>>></span><span class="k">print</span> <span class="n">keepdocker_parser</span><span class="p">.</span><span class="n">parse_known_args</span><span class="p">(</span><span class="n">arguments</span><span class="p">)</span>
<span class="p">(</span><span class="n">Namespace</span><span class="p">(</span><span class="n">force</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">force_image_format</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">image</span><span class="o">=</span><span class="s">'arvados/jobs'</span><span class="p">,</span> <span class="n">pull</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">tag</span><span class="o">=</span><span class="s">'latest'</span><span class="p">),</span> <span class="p">[</span><span class="s">'--project-uuid'</span><span class="p">,</span> <span class="s">'<a href="https://arvadosapi.com/id241-j7d0g-54qgzcohscf9k1v">id241-j7d0g-54qgzcohscf9k1v</a>'</span><span class="p">])</span>
</code></pre>
<p>and no error!</p>
<p>It seems there was a problem parsing the arguments (a list of my system info is attached), I won't look any further as it is working by switching arguments order.</p>
<p>Bests,</p> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=717552019-02-27T13:47:26ZWard Vandewegeward@curii.com
<ul><li><strong>Subject</strong> changed from <i>setting compute node according to doc : "arv-keepdocker: error"</i> to <i>arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wrong</i></li></ul><p>Hi Julien,</p>
<p>Thanks for this bug report. I can reproduce this bug. We will look into fixing it (patches are welcome, of course), and in the mean time I have updated the documentation so that the argument order suggested there is the one that works.</p>
<pre>
$ arv-keepdocker --project-uuid <a href="https://arvadosapi.com/qr1hi-j7d0g-6ljkv8ab0cb6ot0">qr1hi-j7d0g-6ljkv8ab0cb6ot0</a> --pull cure/arvados-runtime latest
latest: Pulling from cure/arvados-runtime
7b722c1070cd: Pull complete
5fbf74db61f1: Pull complete
ed41cb72e5c9: Pull complete
7ea47a67709e: Pull complete
703661372883: Pull complete
8664f41efa89: Pull complete
e4ff173ab779: Pull complete
ea495f19c483: Pull complete
Digest: sha256:63c0c7bee37f932f8a68d5d7eda9f7b7a2cb8149dbd07e1766866974273760c3
Status: Downloaded newer image for cure/arvados-runtime:latest
usage: arv-keepdocker [-h] [--version] [--normalize | --dry-run]
[--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw]
[--update-collection UUID] [--use-filename FILENAME]
[--filename FILENAME] [--portable-data-hash]
[--replication N] [--storage-classes STORAGE_CLASSES]
[--threads N] [--project-uuid UUID] [--name NAME]
[--exclude PATTERN]
[--progress | --no-progress | --batch-progress]
[--silent] [--resume | --no-resume]
[--follow-links | --no-follow-links]
[--cache | --no-cache] [--retries RETRIES]
[path [path ...]]
arv-keepdocker: error: unrecognized arguments: /home/wvandewege/.cache/arvados/docker/sha256:1f05b3c98bdd0dd48373942257ff15a1aec1d886deaf8a18ac1ec7a07da9c1bd.tar
</pre>
<pre>
$ arv-keepdocker --pull cure/arvados-runtime latest --project-uuid <a href="https://arvadosapi.com/qr1hi-j7d0g-6ljkv8ab0cb6ot0">qr1hi-j7d0g-6ljkv8ab0cb6ot0</a>
latest: Pulling from cure/arvados-runtime
Digest: sha256:63c0c7bee37f932f8a68d5d7eda9f7b7a2cb8149dbd07e1766866974273760c3
Status: Image is up to date for cure/arvados-runtime:latest
2019-02-27 13:38:10 arvados.arv_put[26443] INFO: Creating new cache file at /home/wvandewege/.cache/arvados/arv-put/b99012c27a1a3b3f0d01969f07cf17ac
139M / 139M 100.0% 2019-02-27 13:38:17 arvados.arv_put[26443] INFO:
2019-02-27 13:38:17 arvados.arv_put[26443] INFO: Collection saved as 'Docker image cure/arvados-runtime:latest sha256:1f05b'
<a href="https://arvadosapi.com/qr1hi-4zz18-l4995i7dcwldpzi">qr1hi-4zz18-l4995i7dcwldpzi</a>
</pre> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=718062019-02-27T16:53:04ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>2019-03-13 Sprint</i></li></ul> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=718112019-02-27T16:57:42ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=719572019-03-01T19:34:01ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Release</strong> set to <i>15</i></li></ul> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=721372019-03-08T15:32:49ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>14880-keepdocker-argparse @ <a class="changeset" title="14880: Fix argument parsing issues arv-keepdocker passes through some flags to arv-put. The str..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/b26b8ced25b767648f493a48956a7b40d95e1b20">b26b8ced25b767648f493a48956a7b40d95e1b20</a></p>
<p>arv-keepdocker passes through some flags to arv-put. The strategy it<br />uses is to remove known flags and pass through unknown flags. However<br />it doesn't know if an unknown flag takes a value or not, so it treats<br />the values as a positional argument rather than the value<br />corresponding to the unknown flag.</p>
<ul>
<li>Move positional arguments from "keepdocker_parser" to "arg_parser" </li>
<li>Strip positional arguments (image name, tag) from arguments before<br /> parsing known keepdocker_parser flags.</li>
<li>Additionally move a couple of flags that were mistakenly added to<br /> put.run_opts (and thus appearing in the arv-keepdocker help despite<br /> making no sense) that should have been in put.upload_opts</li>
</ul>
<p><a class="external" href="https://ci.curoverse.com/job/run-tests/2580/">https://ci.curoverse.com/job/run-tests/2580/</a></p> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=721662019-03-08T20:53:24ZEric Biagiotti
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>14880-keepdocker-argparse @ <a class="changeset" title="14880: Fix argument parsing issues arv-keepdocker passes through some flags to arv-put. The str..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/b26b8ced25b767648f493a48956a7b40d95e1b20">b26b8ced25b767648f493a48956a7b40d95e1b20</a></p>
<p>arv-keepdocker passes through some flags to arv-put. The strategy it<br />uses is to remove known flags and pass through unknown flags. However<br />it doesn't know if an unknown flag takes a value or not, so it treats<br />the values as a positional argument rather than the value<br />corresponding to the unknown flag.</p>
<ul>
<li>Move positional arguments from "keepdocker_parser" to "arg_parser" </li>
<li>Strip positional arguments (image name, tag) from arguments before<br />parsing known keepdocker_parser flags.</li>
<li>Additionally move a couple of flags that were mistakenly added to<br />put.run_opts (and thus appearing in the arv-keepdocker help despite<br />making no sense) that should have been in put.upload_opts</li>
</ul>
<p><a class="external" href="https://ci.curoverse.com/job/run-tests/2580/">https://ci.curoverse.com/job/run-tests/2580/</a></p>
</blockquote>
<p>Tiny typo in put.py line 163. Missing 'it'. And tests failed. Other than that LGTM.</p> Arvados - Bug #14880: arv-keepdocker is very picky about the order of its command line arguments and crashes with an obscure error if the order is wronghttps://dev.arvados.org/issues/14880?journal_id=721762019-03-08T21:30:38ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul>