https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422015-10-27T18:14:18ZArvadosArvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=317972015-10-27T18:14:18ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/31797/diff?detail_id=31213">diff</a>)</li><li><strong>Category</strong> set to <i>FUSE</i></li><li><strong>Story points</strong> set to <i>1.0</i></li></ul> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=319292015-10-28T19:46:05ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Assigned To</strong> set to <i>Radhika Chippada</i></li></ul> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=319482015-10-28T20:42:19ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/31948/diff?detail_id=31383">diff</a>)</li></ul><p>I've updated the description with implementation ideas, and points about where documentation is likely to need updating. I have not changed any functional requirements, so I don't think this requires a change in story points. I hope these are useful pointers to help start the ball rolling on implementation work.</p> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=319492015-10-28T20:44:21ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/31949/diff?detail_id=31384">diff</a>)</li></ul> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=320522015-11-03T15:13:15ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>commit <a class="changeset" title="7661: add --by-pdh option to FUSE and use this option in crunch-job. Do not start web socket clie..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/64027668d7b5bd47a863c86277c83961d8a29914">64027668d7b5bd47a863c86277c83961d8a29914</a></p>
<p>Implemented the items listed in the description. I chose option three and added a by_pdh optional argument to MagicDirectory since method overloading seemed to be the most straightforward and non-intrusive.</p>
<p>I spent a couple hours trying to see how I may write a test to pass by_pdh instead of by_id to the MagicDirectory based tests. I could not figure out where to improve existing test implementation to do this. So, did not add any tests for this.</p> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=320542015-11-03T15:14:45ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=321712015-11-09T14:52:43ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>In <code>MagicDirectory</code>, suggest calling it <code>pdh_only</code> instead of <code>by_pdh</code>.</p>
<p>This statement:</p>
<pre>
if self.by_pdh and uuid_pattern.match(k):
raise llfuse.FUSEError(errno.ENOENT)
</pre>
<p>should be folded into the existing conditional. Also it should return False, not raise an exception:</p>
<pre>
if not portable_data_hash_pattern.match(k) and (self.pdh_only or not uuid_pattern.match(k)):
return False
</pre>
<p>I don't think it's necessary or desirable to modify the behavior of arv-mount --all based on --by-pdh. This means you can revert the changes to the text of <code>--all</code> (line 35) and behavior of <code>e._entries['by_id']</code> (line 135).</p>
<p>For testing, suggest starting from <code>FuseMagicTest</code></p>
<ol>
<li><code>make_mount</code> takes keyword arguments, which are passed through, so you can write tests for each of <code>make_mount(fuse.MagicDirectory, by_pdh=True)</code> and <code>make_mount(fuse.MagicDirectory, by_pdh=False)</code></li>
<li><code>self.api.collections().create()...</code> should return an API object with a UUID of the collection</li>
<li>When by_pdh is False, test that the collection is accessible by both the PDH and the UUID</li>
<li>When by_pdh is True, perform the same test except check that the collection is only accessable by PDH and fails when accessing by UUID</li>
</ol> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=322252015-11-10T21:36:18ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><ul>
<li>I figured out why <code>FuseMagicTestPDHOnly</code> is failing.</li>
</ul>
<p>Line 514 of fusedir.py:</p>
<pre>
self._entries['by_id'] = self.inodes.add_entry(MagicDirectory(
self.inode, self.inodes, self.api, self.num_retries))
</pre>
<p>You need to add <code>self.pdh_only</code>:</p>
<pre>
self._entries['by_id'] = self.inodes.add_entry(MagicDirectory(
self.inode, self.inodes, self.api, self.num_retries, self.pdh_only))
</pre>
<ul>
<li>Can you add the same access-by-UUID test to FuseMagicTest? Except it should succeed, of course.</li>
</ul>
<ul>
<li>tests.test_mount.MagicDirApiError is failing for me:<br /><pre>
ERROR: runTest (tests.test_mount.MagicDirApiError)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/peter/work/arvados/services/fuse/tests/test_mount.py", line 1030, in runTest
llfuse.listdir(os.path.join(self.mounttmp, self.testcollection))
File "llfuse/fuse_api.pxi", line 43, in llfuse.capi.listdir (src/llfuse/capi_linux.c:22621)
OSError: [Errno 2] No such file or directory: '/tmp/tmpBpoluv/97d180c4f916faf61fb3d64aa2263961+52'
</pre></li>
</ul> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=322262015-11-10T21:41:26ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>tests.test_mount.MagicDirApiError seems to be failing in master as well for me, so it is likely unrelated to this change.</p> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=322302015-11-11T00:21:34ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>Thanks for the help with line 514 of fusedir.py. The test is now working.</p>
<p>I also added the other test variations (by_id, pdh_only=False) etc that I was holding back until the main test passed.</p>
<p>All fuse tests are passing for me.</p> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=322422015-11-11T15:59:12ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p><a class="changeset" title="7661: Pass pdh_only when adding by_id subdir; test now passes." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/768f114b1746dddec0f0224dee92a58c8ddc5499">768f114b1746dddec0f0224dee92a58c8ddc5499</a> Looks good to me. Please merge.</p> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=322442015-11-11T16:05:10ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>50</i> to <i>100</i></li></ul><p>Applied in changeset arvados|commit:3cdc4e8e3668b783f42f887f6a8662b66ecb7f9a.</p> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=326102015-11-19T09:57:22ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>This change appears to be causing a problem for me - I appear to have a crunch-job that passes `--by-pdh` to arv-mount, but my arv-mount does not support that option so it exits with an error. I guess this change requires that crunch-job and arv-mount be upgraded in sync. I think in my case arv-mount is coming from the Ubuntu apt packages, and I can confirm that now that I've updated python-arvados-fuse on all nodes, this issue appears to have been resolved.</p> Arvados - Idea #7661: [FUSE] Add --by-pdh switch for Crunch's usehttps://dev.arvados.org/issues/7661?journal_id=326112015-11-19T13:05:07ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Sorry for the trouble, Josh. Getting cross-component dependencies declared correctly is always a challenge. I should've given you a heads-up about that when we discussed upgrading yesterday.</p>