https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422014-11-21T20:50:34ZArvadosArvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=184552014-11-21T20:50:34ZTim Piercetwp@curoverse.com
<ul><li><strong>Category</strong> set to <i>SDKs</i></li><li><strong>Target version</strong> set to <i>Bug Triage</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=184562014-11-21T20:50:47ZTim Piercetwp@curoverse.com
<ul><li><strong>Subject</strong> changed from <i>arv copy did not pay attention to --project-uuid given</i> to <i>[SDKs] arv copy did not pay attention to --project-uuid given</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=191762014-12-11T17:16:35ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Subject</strong> changed from <i>[SDKs] arv copy did not pay attention to --project-uuid given</i> to <i>[SDKs] arv copy does not respect --project-uuid when copying Collection</i></li></ul><p>This might be related to <a class="issue tracker-1 status-3 priority-4 priority-default closed" title="Bug: [SDK] --project-uuid switch behaves strangely without = (seen in arv-keepdocker, arv-copy) (Resolved)" href="https://dev.arvados.org/issues/3847">#3847</a>.</p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=193642014-12-18T20:24:41ZTom Cleggtom@curii.com
<ul><li><strong>Subject</strong> changed from <i>[SDKs] arv copy does not respect --project-uuid when copying Collection</i> to <i>[SDKs] arv copy should respect --project-uuid when saving destination Collection</i></li><li><strong>Story points</strong> set to <i>0.5</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=193652014-12-18T20:24:45ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> changed from <i>Bug Triage</i> to <i>Arvados Future Sprints</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=200942015-01-16T20:20:03ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2015-02-18 sprint</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=202052015-01-20T15:48:44ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Assigned To</strong> set to <i>Radhika Chippada</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=204952015-01-26T15:02:59ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Assigned To</strong> deleted (<del><i>Radhika Chippada</i></del>)</li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=205932015-01-28T19:32:05ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=206252015-01-28T21:37:56ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> deleted (<del><i>Peter Amstutz</i></del>)</li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=207222015-01-29T19:58:36ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=213162015-02-17T21:16:22ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Notes:</p>
<p>It should now correctly respect --project-uuid for collections, templates, pipeline instances, and docker image metadata.</p>
<p>If a collection already exists on the target system, it will create a new collection in the target project using the existing manifest_text from the existing collection.</p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=213172015-02-17T21:20:27ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=213892015-02-18T20:10:20ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2015-02-18 sprint</i> to <i>2015-03-11 sprint</i></li></ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215082015-02-23T03:07:15ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Reviewing <a class="changeset" title="4520: tested copying, fixed bad 'properties' field" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/74784b33fd7b8b519b0ce0cbbdfa16dba2dad219">74784b3</a>.</p>
<ul>
<li>The code to take a source collection and turn it into a collection on dst (including necessary munging) seems long enough to be worth DRYing up at this point.</li>
<li>Why did the log level of the "Skipping message" collection upgrade from DEBUG to INFO? It seems like this could occasionally make the program noticeably more verbose, and I'm not sure how it helps the user to know this.</li>
</ul>
<p>While I was reading this I noticed some other bugs in the code you're touching. If you're up for fixing them, I'd say go for it. If not, we should file separate bugs about them.</p>
<ul>
<li>The collection create calls don't respect args.retries.</li>
<li>The docker_image_hash link is copied incorrectly. The name should be Docker's native "image ID," not the portable data hash of the corresponding collection.</li>
</ul>
<p>Thanks.</p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215342015-02-23T19:15:11ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Brett Smith wrote:</p>
<blockquote>
<p>Reviewing <a class="changeset" title="4520: tested copying, fixed bad 'properties' field" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/74784b33fd7b8b519b0ce0cbbdfa16dba2dad219">74784b3</a>.</p>
<ul>
<li>The code to take a source collection and turn it into a collection on dst (including necessary munging) seems long enough to be worth DRYing up at this point.</li>
</ul>
</blockquote>
<p>Done.</p>
<blockquote>
<ul>
<li>Why did the log level of the "Skipping message" collection upgrade from DEBUG to INFO? It seems like this could occasionally make the program noticeably more verbose, and I'm not sure how it helps the user to know this.</li>
</ul>
</blockquote>
<p>Took that out entirely and revamped the copying the logic to actually check the <code>owner_uuid</code>, <code>name</code> and <code>portable_data_hash</code> of existing collection at the destination before deciding whether to make a copy.</p>
<blockquote>
<p>While I was reading this I noticed some other bugs in the code you're touching. If you're up for fixing them, I'd say go for it. If not, we should file separate bugs about them.</p>
<ul>
<li>The collection create calls don't respect args.retries.</li>
</ul>
</blockquote>
<p>Fixed.</p>
<blockquote>
<ul>
<li>The docker_image_hash link is copied incorrectly. The name should be Docker's native "image ID," not the portable data hash of the corresponding collection.</li>
</ul>
</blockquote>
<p>Fixed. Also looks for (and copies) Docker metadata any time it is copying a collection and not only when copying pipelines.</p>
<p>Bonus assertion added to <code>keep.py</code> because it turns out arv-copy was passing unicode strings to <code>KeepClient.put()</code> (whoops!)</p>
<p>Now at <a class="changeset" title="4520: Better checking to see if collection already exists at the destination. Set args.project_uu..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/a47dc2bc994e89347fd94d088fa3756236c7fd52">a47dc2b</a></p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215382015-02-23T19:51:03ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Reviewing <a class="changeset" title="4520: Better checking to see if collection already exists at the destination. Set args.project_uu..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/a47dc2bc994e89347fd94d088fa3756236c7fd52">a47dc2b</a></p>
<p>Peter Amstutz wrote:</p>
<blockquote><blockquote>
<ul>
<li>The docker_image_hash link is copied incorrectly. The name should be Docker's native "image ID," not the portable data hash of the corresponding collection.</li>
</ul>
</blockquote>
<p>Fixed. Also looks for (and copies) Docker metadata any time it is copying a collection and not only when copying pipelines.</p>
</blockquote>
<p>Thanks, that basically covers <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: [SDKs] Make arv-copy useful for copying a Docker image (Closed)" href="https://dev.arvados.org/issues/5294">#5294</a>.</p>
<blockquote>
<p>Bonus assertion added to <code>keep.py</code> because it turns out arv-copy was passing unicode strings to <code>KeepClient.put()</code> (whoops!)</p>
</blockquote>
<p>Please change this condition to <code>isinstance(data, str)</code> to accept subclasses, and add a test.</p>
<p>The new line <code>args.project_uuid = dst_arv.users().current().execute()["uuid"]</code> should respect args.retries.</p>
<p>Thanks.</p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215442015-02-23T20:16:10ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Brett Smith wrote:</p>
<blockquote>
<p>Please change this condition to <code>isinstance(data, str)</code> to accept subclasses, and add a test.</p>
</blockquote>
<p>Fixed.</p>
<blockquote>
<p>The new line <code>args.project_uuid = dst_arv.users().current().execute()["uuid"]</code> should respect args.retries.</p>
</blockquote>
<p>Fixed.</p>
<p>Now at <a class="changeset" title="4520: manifest_text() is utf-8 encoded by default so it can be safely put() to Keep. Add test th..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/7b0d3939e19b6c62effdb5c808b811b38f4ffd57">7b0d393</a></p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215512015-02-23T20:38:10ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>Brett Smith wrote:</p>
<blockquote>
<p>Please change this condition to <code>isinstance(data, str)</code> to accept subclasses, and add a test.</p>
</blockquote>
<p>Fixed.</p>
<blockquote>
<p>The new line <code>args.project_uuid = dst_arv.users().current().execute()["uuid"]</code> should respect args.retries.</p>
</blockquote>
<p>Fixed.</p>
</blockquote>
<p>Both of these changes are good. Unfortunately, the change to make CollectionWriter.manifest_text() encode the return string isn't going to fly. The entire interface to CollectionWriter is already bytes-based, so users need to pass in encoded strings for filenames already. For anybody already passing in non-ASCII, this change will basically attempt a double-encode and break:</p>
<pre>>>> import arvados
>>> cw = arvados.CollectionWriter()
>>> with cw.open(u'♥'.encode('utf-8')) as cf:
... pass
...
>>> cw.manifest_text()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "arvados/collection.py", line 588, in manifest_text
return manifest.encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 41: ordinal not in range(128)
</pre>
<p>If you're okay reverting that change, I think this is good to merge. If there's some nuance I'm missing, let's talk about it. Thanks.</p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215542015-02-23T21:35:33ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><ul>
<li>Don't encode manifest_text by default to avoid double-encode faults.</li>
<li>Coerce unicode strings to ascii in put(). Will raise an error if it is not a valid conversion.</li>
<li>Use result.content (returns literal result bytes) instead result.text (returns unicode) when processing Keep responses.</li>
<li>Updated test.</li>
</ul> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215562015-02-23T21:40:17ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<ul>
<li>Updated test.</li>
</ul>
</blockquote>
<p>Please assert that the first put returns the right hash (so we're sure it encodes correctly), and ditch the noop <code>foo_locator</code> assignments here. Then go ahead and merge. Thanks.</p> Arvados - Bug #4520: [SDKs] arv copy should respect --project-uuid when saving destination Collectionhttps://dev.arvados.org/issues/4520?journal_id=215612015-02-23T21:50:09ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset arvados|commit:f0fe7273c1851cb93e9edd58c0b60d3590b222ed.</p>