https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422016-09-14T14:29:57ZArvadosArvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=430112016-09-14T14:29:57ZTom Cleggtom@curii.com
<ul></ul><p>WIP: 10008-flaky-token-test</p>
Unfortunately, the new test seems to indicate the token-refresh behavior is actually broken:
<ul>
<li>time is mocked so it appears to advance by 13 days each time open() is called</li>
<li>open() is called 3 times in a collection directory</li>
<li>API server logs show a single "get collection" request (expect 3)</li>
</ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=430182016-09-14T15:29:36ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li><li><strong>Story points</strong> set to <i>0.5</i></li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=440442016-10-13T16:37:01ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> set to <i>2016-10-26 sprint</i></li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=443662016-10-21T19:38:13ZTom Cleggtom@curii.com
<ul></ul><p>10008-check-token-exp-on-open</p>
<p>test <a class="changeset" title="10008: Move test setup from __init__() to setUp()" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/b73985d8a0c9173aec57f6a81fe540b2813a5bff">b73985d8a0c9173aec57f6a81fe540b2813a5bff</a></p> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=444102016-10-25T13:04:34ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><ul>
<li>File <code>services/fuse/arvados_fuse/__init__.py</code>:
<ul>
<li>Line 504: Could you please explain in a comment why is the parent traversal needed on <code>open()</code>?</li>
<li>Line 595: Is the method <code>opendir()</code> needing the same parent traversal to <code>touch()</code> all of its ancestors too?</li>
</ul>
</li>
<li>Is there any test covering the case when the token is expired? If I’m understanding this test correctly, it checks that the token is refreshed if the fuse is accessed every 13 days, right?</li>
</ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=444342016-10-25T16:24:07ZTom Cleggtom@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<ul>
<li>File <code>services/fuse/arvados_fuse/__init__.py</code>:
<ul>
<li>Line 504: Could you please explain in a comment why is the parent traversal needed on <code>open()</code>?</li>
</ul></li>
</ul>
</blockquote>
<p>Added.</p>
<blockquote>
<ul>
<li>Line 595: Is the method <code>opendir()</code> needing the same parent traversal to <code>touch()</code> all of its ancestors too?</li>
</ul>
</blockquote>
<p>The passage of time (and the resulting invalidation of the signature tokens) is the only kind of event we don't expect websocket to tell us about. So opendir() doesn't really need this.</p>
<p>Of course, I suppose that means this really belongs in read(): if a file is read after being left open for 2 weeks, even open() won't have a chance to refresh.</p>
<blockquote>
<ul>
<li>Is there any test covering the case when the token is expired? If I’m understanding this test correctly, it checks that the token is refreshed if the fuse is accessed every 13 days, right?</li>
</ul>
</blockquote>
<p>Yes, that's right. There test covers only "old enough to be nearly expired", not "actually expired". I'm not too worried about it. This would be a good integration test, but for that we'd also have to figure out how to convince keepstore to expire tokens immediately...</p>
<p><a class="changeset" title="10008: Comment to explain ancestor update." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/7bf89e17945bb6610d6339409a6a411803851434">7bf89e17945bb6610d6339409a6a411803851434</a></p>
Notes from meeting:
<ul>
<li>Checking in read() shouldn't wreck performance</li>
<li>Even checking in read() might not be enough
<ul>
<li>we will initiate an update, but will the read progress before the updated response arrives?</li>
<li>if we have a new collection when we're fulfilling the read(), but we created the collection, subdir, and file inodes back when we had the old collection, will we use the new collection's block locators, or will the cached inodes still cause us to use the old ones?</li>
</ul>
</li>
<li>It would be better to fix this (re-fetch the manifest when reading data, if we notice the tokens expire soon) in the Python SDK so arv-mount wouldn't have to do anything special. However, if it's not too hard to fix arv-mount, we might as well do that in the meantime.</li>
</ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=444432016-10-25T17:44:13ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>A few notes:</p>
<ol>
<li><code>Operations.lookup()</code> calls <code>p[name]</code> which turns into <code>Directory.__getitem__</code> which is wrapped by <code>@check_update</code>. This means when a lookup happens, it is supposed to synchronously refresh the underlying directory if it is stale.</li>
<li>FUSE directory entry caching behavior has changed over kernel versions, so it is possible that <code>lookup()</code> used to always occur before <code>open()</code>, but now sometimes does not happen if the kernel has the directory entry in cache.</li>
<li><code>CollectionDirectory.update</code> has an early exit on <code>self.collection_record is not None and portable_data_hash_pattern.match(self.collection_locator)</code> which suggests that collections referenced by portable data hash never get refreshed, even if they need to be in order to refresh tokens.</li>
<li>ArvadosFiles which are the same except for token updates are applied as <code>TOK</code> changes (in other words, Collection.update() should be presumed to work, and the problem is that it isn't being called when it is supposed to.)</li>
</ol>
<p>Updating is synchronous and it will hold a lock on the collection which will prevent any other activity on the collection while it is updating, so there should not be an update race.</p> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=444472016-10-25T17:52:48ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Suggested fix:</p>
<p><code>CollectionDirectory.update</code> needs to force update when poll time is exceeded, even for read only collections.</p>
<p><code>FuseArvadosFile.readfrom()</code> should be wrapped with <code>@check_update</code>. When created, <code>FuseArvadosFile</code> should have poll time set to match the parent.</p>
<p><code>FuseArvadosFile</code> needs to have a reference to either the parent directory object or the inodes table (to look up parent_inode) so that it can initiate an <code>update()</code>.</p> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=445652016-10-26T19:27:55ZTom Cleggtom@curii.com
<ul><li><strong>Subject</strong> changed from <i>[FUSE] [Tests] Fix flaky test TokenExpiryTest</i> to <i>[FUSE] [SDKs] When reading data through Collection et al., signatures should refresh automatically when needed</i></li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=445672016-10-26T19:28:17ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> deleted (<del><i>2016-10-26 sprint</i></del>)</li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=445692016-10-26T19:28:49ZTom Cleggtom@curii.com
<ul><li><strong>Category</strong> changed from <i>Tests</i> to <i>FUSE</i></li><li><strong>Assigned To</strong> deleted (<del><i>Tom Clegg</i></del>)</li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=484402017-02-15T21:24:46ZTom Cleggtom@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li><li><strong>Target version</strong> set to <i>2017-03-01 sprint</i></li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=485952017-02-21T18:55:37ZTom Cleggtom@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/48595/diff?detail_id=46761">diff</a>)</li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=485962017-02-21T18:55:59ZTom Cleggtom@curii.com
<ul><li><strong>Subject</strong> changed from <i>[FUSE] [SDKs] When reading data through Collection et al., signatures should refresh automatically when needed</i> to <i>[SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when needed</i></li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=485972017-02-21T18:56:21ZTom Cleggtom@curii.com
<ul><li><strong>Category</strong> changed from <i>FUSE</i> to <i>SDKs</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>New</i></li><li><strong>Assigned To</strong> deleted (<del><i>Tom Clegg</i></del>)</li><li><strong>Target version</strong> deleted (<del><i>2017-03-01 sprint</i></del>)</li><li><strong>Story points</strong> deleted (<del><i>0.5</i></del>)</li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=545072017-08-29T13:37:47ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>Arvados Future Sprints</i></li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=947862021-07-07T18:23:16ZWard Vandewegeward@curii.com
<ul><li><strong>Target version</strong> deleted (<del><i>Arvados Future Sprints</i></del>)</li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=1120322023-02-14T22:23:30ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>60</i></li></ul> Arvados - Bug #10008: [SDKs] [Python] When reading data through Collection et al., signatures should refresh automatically when neededhttps://dev.arvados.org/issues/10008?journal_id=1234032024-03-01T21:13:26ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> set to <i>Future</i></li></ul>