https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422015-11-11T15:55:00ZArvadosArvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=322392015-11-11T15:55:00ZTom Cleggtom@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/32239/diff?detail_id=31644">diff</a>)</li></ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=322412015-11-11T15:55:58ZTom Cleggtom@curii.com
<ul><li><strong>Category</strong> set to <i>SDKs</i></li><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2015-12-02 sprint</i></li></ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=323132015-11-12T15:34:30ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Story points</strong> set to <i>1.0</i></li></ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=323162015-11-12T15:40:08ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li></ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=323252015-11-12T15:42:49ZTom Cleggtom@curii.com
<ul><li><strong>Story points</strong> changed from <i>1.0</i> to <i>2.0</i></li></ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=326562015-11-20T20:33:47ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=326712015-11-20T23:46:39ZTom Cleggtom@curii.com
<ul></ul>7751-mount-tmp @ <a class="changeset" title="7751: Suppress help messages when testing usage errors." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/273221309b13c6ab1c299095a811041b95ce4fa9">2732213</a> includes:
<ul>
<li>"scratch space" mount mode: data blocks get written to Keep, but a collection never gets saved (at least not by arv-mount -- you can get a manifest by reading the magic <code>.arvados#collection</code> file before unmounting, and save that to a collection).</li>
<li>Rearrange arv-mount args a bit so caller can specify a custom layout: <code>--mount-tmp foo --mount-by-pdh by_id /tmp/mnt</code> → <code>/tmp/mnt/foo</code> is scratch space, <code>/tmp/mnt/by_id/</code> is magic by-id dir.</li>
<li>Move big pile of code out of <code>bin/arv-mount</code> into the <code>arvados_fuse</code> module dir
<ul>
<li>→ argument-parsing and nearly all of the actual FUSE setup code is now used in the new integration tests (was untested before).</li>
<li>→ now possible to test argument handling without the overhead of subscribing to websockets and bringing up a fuse mount</li>
</ul>
</li>
<li>Small bugfix for <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [SDK] Python websockets client sometimes hangs on shutdown (Resolved)" href="https://dev.arvados.org/issues/7654">#7654</a> where we accidentally monkey-patched a ws4py <code>closed()</code> method with a bool, causing a stack trace during websocket shutdown.</li>
</ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=326942015-11-23T20:32:30ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Test failures:</p>
<pre>
======================================================================
ERROR: runTest (tests.test_mount.MagicDirApiError)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/peter/work/arvados/services/fuse/tests/test_mount.py", line 1034, in runTest
llfuse.listdir(os.path.join(self.mounttmp, self.testcollection))
File "llfuse/fuse_api.pxi", line 43, in llfuse.capi.listdir (src/llfuse/capi_linux.c:22621)
OSError: [Errno 2] No such file or directory: '/tmp/tmpSbUVZB/97d180c4f916faf61fb3d64aa2263961+52'
======================================================================
FAIL: test_tmp_snapshots (tests.test_tmp_collection.TmpCollectionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/peter/work/arvados/services/fuse/tests/integration_test.py", line 66, in wrapper
return func(self, *args, **kwargs)
File "/home/peter/work/arvados/services/fuse/tests/test_tmp_collection.py", line 101, in test_tmp_snapshots
self.pool_test(os.path.join(self.mnt, 'zzz'))
File "/home/peter/work/arvados/services/fuse/tests/integration_test.py", line 34, in pool_test
(modName, clsName, '_'+funcName, args, kwargs))
File "/usr/lib/python2.7/multiprocessing/pool.py", line 244, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
AssertionError: Regexp didn't match: '^\\. 37b51d194a7513e45b56f6524f2d51f2\\+3(\\+\\S+)? 0:3:bar\\n$' not found in u'. 37b51d194a7513e45b56f6524f2d51f2+3+A90de89bcf2ee65cbe320f2eac348858e7725be20@5665e6c3 acbd18db4cc2f85cedef654fccc4a4d8+3+A77c716f7ea2d409fdb116f8b4c52afbd0013fcff@5665e6c3 0:3:bar 3:3:foo\n'
</pre>
<p><code>tests.test_mount.MagicDirApiError</code> is failing consistently. <code>test_tmp_snapshot</code> seems to be a race condition that only fails sometimes.</p>
<p>This came up as an exception but the test is still marked as success (?):</p>
<pre>
test_two_tmp (tests.test_tmp_collection.TmpCollectionTest) ... [keep1] 2015/11/23 15:23:55 [[::1]:58834] PUT acbd18db4cc2f85cedef654fccc4a4d8 0.000169s 200 86 "OK"
[keep0] 2015/11/23 15:23:55 [[::1]:42365] PUT acbd18db4cc2f85cedef654fccc4a4d8 0.000154s 200 86 "OK"
[keep1] 2015/11/23 15:23:55 [[::1]:58836] PUT 37b51d194a7513e45b56f6524f2d51f2 0.000424s 200 86 "OK"
[keep0] 2015/11/23 15:23:55 [[::1]:42367] PUT 37b51d194a7513e45b56f6524f2d51f2 0.002501s 200 86 "OK"
2015-11-23 15:23:56 arvados.arvados_fuse[16092] ERROR: Unhandled exception during FUSE operation
Traceback (most recent call last):
File "/home/peter/work/arvados/services/fuse/arvados_fuse/__init__.py", line 277, in catch_exceptions_wrapper
return orig_func(self, *args, **kwargs)
File "/home/peter/work/arvados/services/fuse/arvados_fuse/__init__.py", line 472, in forget
ent = self.inodes[inode]
File "/home/peter/work/arvados/services/fuse/arvados_fuse/__init__.py", line 215, in __getitem__
return self._entries[item]
KeyError: 1L
Exception in thread Thread-31:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/peter/work/arvados/services/fuse/arvados_fuse/command.py", line 118, in <lambda>
t = threading.Thread(None, lambda: llfuse.main())
File "llfuse/fuse_api.pxi", line 317, in llfuse.capi.main (src/llfuse/capi_linux.c:25028)
File "llfuse/handlers.pxi", line 56, in llfuse.capi.fuse_forget (src/llfuse/capi_linux.c:2730)
File "llfuse/handlers.pxi", line 57, in llfuse.capi.fuse_forget (src/llfuse/capi_linux.c:2682)
File "/home/peter/work/arvados/services/fuse/arvados_fuse/__init__.py", line 290, in catch_exceptions_wrapper
raise llfuse.FUSEError(errno.EIO)
FUSEError: Input/output error
</pre>
<p><code>arvados_fuse.command.Mount.Run</code> should be lowercase <code>run</code> according to PEP8 (<a class="external" href="https://www.python.org/dev/peps/pep-0008/#function-names">https://www.python.org/dev/peps/pep-0008/#function-names</a>)</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=326972015-11-23T20:54:33ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Turns out <code>tests.test_mount.MagicDirApiError</code> is failing is master too, so that may be unrelated.</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=326982015-11-23T21:22:21ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>Turns out <code>tests.test_mount.MagicDirApiError</code> is failing is master too, so that may be unrelated.</p>
</blockquote>
<p>My suspicion is that around v4.0 the Linux VFS may have changed the way directories listings are cached, so this test which checks if something exists (OSError on induced API fail) then checks again (API success) is failing the 2nd time because the 2nd request is being served from the Linux VFS cache instead of asking arv-mount again.</p>
<p>I'll file a separate issue.</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327002015-11-23T21:43:39ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Task <a class="issue tracker-5 status-3 priority-4 priority-default closed child" title="Task: Python SDK helpers (Resolved)" href="https://dev.arvados.org/issues/7793">#7793</a> (Python SDK helpers) should include updating <code>run-command</code>.</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327012015-11-23T21:45:50ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p><code>bin/arv-mount</code> is missing <code>import arvados_fuse.command</code></p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327022015-11-23T21:48:23ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><pre>
$ arv-mount --foreground --mount-tmp=foo ~/keep
Traceback (most recent call last):
File "/home/peter/work/scripts/venv/bin/arv-mount", line 4, in <module>
__import__('pkg_resources').run_script('arvados-fuse==0.1.20151122102613', 'arv-mount')
File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 735, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1652, in run_script
exec(code, namespace, namespace)
File "/home/peter/work/scripts/venv/lib/python2.7/site-packages/arvados_fuse-0.1.20151122102613-py2.7.egg/EGG-INFO/scripts/arv-mount", line 8, in <module>
arvados_fuse.command.Mount(args).Run()
File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_fuse-0.1.20151122102613-py2.7.egg/arvados_fuse/command.py", line 130, in Run
self._run_standalone(self.args)
TypeError: _run_standalone() takes exactly 1 argument (2 given)
</pre> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327032015-11-23T22:27:45ZTom Cleggtom@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p><code>tests.test_mount.MagicDirApiError</code> is failing consistently.</p>
</blockquote>
<p>I haven't seen this one yet (but I've only tried ~5 times so far)</p>
<blockquote>
<p><code>test_tmp_snapshot</code> seems to be a race condition that only fails sometimes.</p>
</blockquote>
<p>I see this one if I run it enough times.</p>
<p>It seems like unlink() is special here: this test (and several others) are deliberately testing race conditions and so far I haven't seen any other failures.</p>
<blockquote>
<p>This came up as an exception but the test is still marked as success (?):</p>
</blockquote>
<p>I see this one too, occasionally, and indeed it doesn't fail the test. Another shutdown race, I suppose. But so far I've only seen it in test_two_tmp, which also has an <code>unlink()</code> -- coincidence?</p>
<blockquote>
<p><code>arvados_fuse.command.Mount.Run</code> should be lowercase <code>run</code> according to PEP8 (<a class="external" href="https://www.python.org/dev/peps/pep-0008/#function-names">https://www.python.org/dev/peps/pep-0008/#function-names</a>)</p>
</blockquote>
<p>Fixed @ <a class="changeset" title="7751: Lowercase method name for PEP-0008." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/796c33a1efdd90f34856e4d24437d9a52367a778">796c33a</a></p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327042015-11-23T22:45:36ZTom Cleggtom@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>Task <a class="issue tracker-5 status-3 priority-4 priority-default closed child" title="Task: Python SDK helpers (Resolved)" href="https://dev.arvados.org/issues/7793">#7793</a> (Python SDK helpers) should include updating <code>run-command</code>.</p>
</blockquote>
<p>I'll probably just do the Python SDK helpers in <a class="issue tracker-5 status-3 priority-4 priority-default closed child" title="Task: Python SDK helpers (Resolved)" href="https://dev.arvados.org/issues/7793">#7793</a>. I've added <a class="issue tracker-2 status-6 priority-4 priority-default closed" title="Feature: [SDKs] Update run-command to use arv-mount --mount-tmp instead of staging directory (Rejected)" href="https://dev.arvados.org/issues/7847">#7847</a> to make use of those in run-command (it obviously stands to benefit from this) but IIRC we deliberately left run-command out of this because the first jobs we want to try this on don't use run-command anyway. Also, it would be reassuring to see real-world results from some deliberate experiments before we start making all run-command jobs rely on it...</p>
<p>Total arv-mount breakage should be fixed as of <a class="changeset" title="7751: Fix shell wrapper code path." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f6bd4a2d80a5092a4306bac61dac90bffdf6211c">f6bd4a2</a> (whoops)</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327092015-11-24T16:05:21ZTom Cleggtom@curii.com
<ul></ul>The "KeyError: 1L" shutdown race is fixed in <a class="changeset" title="7751: Fix stack trace on shutdown caused by race between destroy() and llfuse unmount." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/4574dbbacdd10d24cf5ed312d7b8e91ec9d9c14a">4574dbb</a>. Here's what happened:
<ul>
<li>When "fusermount -u" is called from userspace, llfuse makes a list of inodes to forget, and makes multiple calls to operations.forget(). Each call to forget() has llfuse.lock, but the lock is released between calls.</li>
<li>If operations.destroy() is called from arv-mount (or the test suite), it clears inodes. Therefore, subsequent calls to forget() -- e.g., arising from an external unmount that was already in progress -- will crash.</li>
</ul>
Fix:
<ul>
<li>Acquire llfuse.lock in destroy() so it doesn't race forget().</li>
<li>forget() is a no-op if shutdown has already cleared inodes.</li>
</ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327122015-11-24T16:13:19ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>I'm not keen on monkey-patching the collection object to intercept save() and save_new(). The following approach would be more in line with the existing cache invalidation strategy used in the fuse driver:</p>
<ol>
<li>Subclass <code>Collection</code> and override save() and save_new() to "pass" (no-op)</li>
<li>TmpCollectionDirectory overrides <code>on_event()</code> and calls <code>self.inodes.invalidate_inode()</code> and <code>self.collection_record_file.invalidate()</code> then calls <code>super()</code></li>
<li>Subclass ObjectFile and override read() and size() to regenerate <code>self.contents</code> from the collection when self.stale is True</li>
</ol>
<p>The idea is to aggressively invalidate ".arvados#collection" and generate an up-to-date version on demand rather than serving a cached copy.</p>
<p>Unfortunately, <code>invalidate_inode</code> in llfuse is an asynchronous operation. This is because (in the kernel VFS) the invalidate operation acquires a lock on the inode; if you call <code>invalidate_inode</code> synchronously on the current inode, it will result in a deadlock.</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327232015-11-24T18:47:04ZTom Cleggtom@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>1. Subclass <code>Collection</code> and override save() and save_new() to "pass" (no-op)<br />3. Subclass ObjectFile and override read() and size() to regenerate <code>self.contents</code> from the collection when self.stale is True</p>
</blockquote>
<p>Refactored. <a class="changeset" title="7751: Refactor TmpCollectionDirectory: generate .arvados#collection less often." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/4c0d3027e0297df45e01dd7ededddb3cfecb74fd">4c0d302</a></p>
<blockquote>
<p>2. TmpCollectionDirectory overrides <code>on_event()</code> and calls <code>self.inodes.invalidate_inode()</code> and <code>self.collection_record_file.invalidate()</code> then calls <code>super()</code></p>
</blockquote>
<p>Would CollectionDirectory.update() do anything for us here? I skipped CollectionDirectory and subclassed CollectionDirectoryBase instead because CollectionDirectory seems to be concerned with merging (which we don't want) and has its own <code>.arvados#collection</code> implementation that focuses on getting a current API server record (which we also don't want).</p>
<p>Calling invalidate_inode() from the update() callback instead of unlink()→flush()→lock_released→save() doesn't seem to affect the race bug at all. But invalidating the <em>entry</em> (not just the inode) does: I've updated the test to do 10 iterations of adding and removing files, which nearly always fails before that change, and hasn't failed yet after that change. <a class="changeset" title="7751: Fix race by invalidating name entry (not just inode) when .arvados#collection content is st..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c333c47a1334a36876017ab616a3646cbf1468cd">c333c47</a></p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327892015-11-24T22:38:51ZTom Cleggtom@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>Calling invalidate_inode() from the update() callback instead of unlink()→flush()→lock_released→save() doesn't seem to affect the race bug at all. But invalidating the <em>entry</em> (not just the inode) does: I've updated the test to do 10 iterations of adding and removing files, which nearly always fails before that change, and hasn't failed yet after that change. <a class="changeset" title="7751: Fix race by invalidating name entry (not just inode) when .arvados#collection content is st..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c333c47a1334a36876017ab616a3646cbf1468cd">c333c47</a></p>
</blockquote>
<p>Nope, that didn't work either. With 400 iterations that fails too. But setting <code>llfuse.EntryAttributes().entry_timeout = 0</code> survives 1400 iterations. <a class="external" href="http://pythonhosted.org/llfuse/data.html">http://pythonhosted.org/llfuse/data.html</a></p>
<p><a class="changeset" title="7751: Fix race by telling fuse not to cache the .arvados#collection dirent." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9d80d31df8ed8251651f851bc9b6ccf12f182c70">9d80d31</a></p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327972015-11-25T14:37:08ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>For the record, here's a trace of a failing test:</p>
<pre>
[...]
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount unlink: 2 'foo'
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: collection notify del <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f3006218150> foo <arvados.arvfile.ArvadosFile object at 0x7f3006334bd0>
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: del_entry on inode 3 with refcount 1
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f3006218190> invalidated collection record
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount lookup: parent_inode 2 name '.arvados#collection' inode 4
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount forget: inode 3 nlookup 1 ref_count 1
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount read 6 0 4096
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount unlink: 2 'bar'
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: collection notify del <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f3006218150> bar <arvados.arvfile.ArvadosFile object at 0x7f3006270f10>
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: del_entry on inode 5 with refcount 1
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f3006218190> invalidated collection record
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount forget: inode 5 nlookup 1 ref_count 1
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount forget: inode 4 nlookup 1 ref_count 1
2015-11-25 09:26:49 arvados.arvados_fuse[21735] DEBUG: arv-mount forget: inode 2 nlookup 1 ref_count 1
FAIL
Sent SIGTERM to 21748 (/home/peter/work/arvados/tmp/keep0.pid)
[keep0] 2015/11/25 09:26:49 caught signal: terminated
[keep0] 2015/11/25 09:26:49 keepstore exiting, pid 21748
Sent SIGTERM to 21756 (/home/peter/work/arvados/tmp/keep1.pid)
[keep1] 2015/11/25 09:26:49 caught signal: terminated
[keep1] 2015/11/25 09:26:49 keepstore exiting, pid 21756
======================================================================
FAIL: test_tmp_snapshots (tests.test_tmp_collection.TmpCollectionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/peter/work/arvados/services/fuse/tests/integration_test.py", line 66, in wrapper
return func(self, *args, **kwargs)
File "/home/peter/work/arvados/services/fuse/tests/test_tmp_collection.py", line 101, in test_tmp_snapshots
self.pool_test(os.path.join(self.mnt, 'zzz'))
File "/home/peter/work/arvados/services/fuse/tests/integration_test.py", line 34, in pool_test
(modName, clsName, '_'+funcName, args, kwargs))
File "/usr/lib/python2.7/multiprocessing/pool.py", line 244, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
AssertionError: Regexp didn't match: '^$' not found in u'. 37b51d194a7513e45b56f6524f2d51f2+3+Adfa8a6470676bd3530c8a6594bd76f2dda806fe7@56683a29 0:3:bar\n'
----------------------------------------------------------------------
Ran 1 test in 3.007s
</pre>
<p>Here's a successful one:</p>
<pre>
[...]
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount unlink: 2 'foo'
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: collection notify del <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f09dd93a150> foo <arvados.arvfile.ArvadosFile object at 0x7f09dd379c10>
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: del_entry on inode 3 with refcount 1
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f09dd93a190> invalidated collection record
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount forget: inode 3 nlookup 1 ref_count 1
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount lookup: parent_inode 2 name '.arvados#collection' inode 4
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount read 6 0 4096
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount unlink: 2 'bar'
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: collection notify del <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f09dd93a150> bar <arvados.arvfile.ArvadosFile object at 0x7f09dc237f10>
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: del_entry on inode 5 with refcount 1
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f09dd93a190> invalidated collection record
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount forget: inode 5 nlookup 1 ref_count 1
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount read 7 0 4096
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount forget: inode 4 nlookup 1 ref_count 1
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount lookup: parent_inode 2 name 'foo' not found
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount create: 2 'foo' 100644
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: collection notify add <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f09dd93a150> foo <arvados.arvfile.ArvadosFile object at 0x7f09dd379f50>
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f09dd93a190> invalidated collection record
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: collection notify mod <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f09dd93a150> foo (<arvados.arvfile.ArvadosFile object at 0x7f09dd379f50>, <arvados.arvfile.ArvadosFile object at 0x7f09dd379f50>)
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f09dd93a190> invalidated collection record
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount write 8 0 3
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: collection notify write <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f09dd93a150> foo (<arvados.arvfile.ArvadosFile object at 0x7f09dd379f50>, <arvados.arvfile.ArvadosFile object at 0x7f09dd379f50>)
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f09dd93a190> invalidated collection record
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: arv-mount lookup: parent_inode 2 name '.arvados#collection' inode 4
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: collection notify mod <arvados_fuse.fusedir.UnsaveableCollection object at 0x7f09dd93a150> foo (<arvados.arvfile.ArvadosFile object at 0x7f09dd379f50>, <arvados.arvfile.ArvadosFile object at 0x7f09dd379f50>)
2015-11-25 09:35:43 arvados.arvados_fuse[22738] DEBUG: <arvados_fuse.fusedir.TmpCollectionDirectory object at 0x7f09dd93a190> invalidated collection record
[keep1] 2015/11/25 09:35:43 [[::1]:41523] PUT 37b51d194a7513e45b56f6524f2d51f2 0.000374s 200 86 "OK"
[keep0] 2015/11/25 09:35:43 [[::1]:46603] PUT 37b51d194a7513e45b56f6524f2d51f2 0.000245s 200 86 "OK"
.
.
.
----------------------------------------------------------------------
Ran 1 test in 3.500s
OK
</pre> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327982015-11-25T15:21:00ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>It's working better with the no-cache trick in <a class="changeset" title="7751: Fix race by telling fuse not to cache the .arvados#collection dirent." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9d80d31df8ed8251651f851bc9b6ccf12f182c70">9d80d31</a></p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=327992015-11-25T15:33:32ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>7751-mount-tmp looks good to me.</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=328482015-11-30T15:00:49ZTom Cleggtom@curii.com
<ul></ul><p>7751-crunch-fuse-output @ <a class="changeset" title="7751: Add convenience class for staging task output in $TASK_KEEPMOUNT_TMP." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/70111b9be8ef9b85832e4d2c41f99fe18dbf0a39">70111b9</a> lets you do stuff like this included example script:</p>
<pre><code class="python syntaxhl"><span class="c1">#!/usr/bin/env python
</span>
<span class="kn">import</span> <span class="nn">arvados</span>
<span class="kn">import</span> <span class="nn">arvados.crunch</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">arvados</span><span class="p">.</span><span class="n">crunch</span><span class="p">.</span><span class="n">TaskOutputDir</span><span class="p">()</span>
<span class="n">string</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">__file__</span><span class="p">).</span><span class="n">read</span><span class="p">()</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">out</span><span class="p">.</span><span class="n">path</span><span class="p">,</span> <span class="s">'example.out'</span><span class="p">),</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">string</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">out</span><span class="p">.</span><span class="n">path</span><span class="p">,</span> <span class="s">'example.out.SHA1'</span><span class="p">),</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">hashlib</span><span class="p">.</span><span class="n">sha1</span><span class="p">(</span><span class="n">string</span><span class="p">).</span><span class="n">hexdigest</span><span class="p">()</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="n">arvados</span><span class="p">.</span><span class="n">current_task</span><span class="p">().</span><span class="n">set_output</span><span class="p">(</span><span class="n">out</span><span class="p">.</span><span class="n">manifest_text</span><span class="p">())</span>
</code></pre> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=328792015-11-30T21:11:16ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><pre>
--volume=/tmp/crunch-job-1001/task/localhost.1.keep/tmp:/keep_tmp:ro
</pre>
<p>I don't think mounting the <code>tmp</code> directory as read-only is going to do what you want :-)</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=329282015-12-01T16:54:34ZTom Cleggtom@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>I don't think mounting the <code>tmp</code> directory as read-only is going to do what you want :-)</p>
</blockquote>
<p>Indeed. Fixed, now at <a class="changeset" title="7751: Add convenience class for staging task output in $TASK_KEEPMOUNT_TMP." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9ce4db92fc33ecd4e98b1ff2a675969153faaf89">9ce4db9</a></p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=329352015-12-01T18:29:45ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>Peter Amstutz wrote:</p>
<blockquote>
<p>I don't think mounting the <code>tmp</code> directory as read-only is going to do what you want :-)</p>
</blockquote>
<p>Indeed. Fixed, now at <a class="changeset" title="7751: Add convenience class for staging task output in $TASK_KEEPMOUNT_TMP." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9ce4db92fc33ecd4e98b1ff2a675969153faaf89">9ce4db9</a></p>
</blockquote>
<p>Tested at it work now.</p>
<p>Rest of it looks good to me.</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=330482015-12-02T19:22:05ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Merge of the new branch is blocked until we build and deploy new compute node images with the FUSE branch.</p> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=330832015-12-02T20:39:04ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Target version</strong> changed from <i>2015-12-02 sprint</i> to <i>2015-12-16 sprint</i></li><li><strong>Story points</strong> changed from <i>2.0</i> to <i>0.0</i></li></ul> Arvados - Feature #7751: [Crunch] [SDKs] [FUSE] Convenient way to write job output to Keep via writable arv-mount, as an alternative to staging output on scratch and then copying when finished.https://dev.arvados.org/issues/7751?journal_id=331352015-12-03T16:55:09ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>86</i> to <i>100</i></li></ul><p>Applied in changeset arvados|commit:5590c9ac669f2d74858e6c994afe1a2e9df8d104.</p>