https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422017-04-17T20:45:16ZArvadosArvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=508502017-04-17T20:45:16ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Subject</strong> changed from <i>S3 multipart downloads fail truncated writing to keep</i> to <i>S3 multipart download truncated writing to keep</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/50850/diff?detail_id=49082">diff</a>)</li></ul> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=508522017-04-17T20:53:20ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/50852/diff?detail_id=49083">diff</a>)</li></ul> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=508532017-04-17T20:55:57ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><pre>
$ python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("ttt", "wb")
>>> f.seek(10, 1)
>>> f.tell()
10
>>> f.write("blub")
>>> f.close()
$ ls -l ttt
-rw-r--r-- 1 peter peter 14 Apr 17 16:54 ttt
</pre>
<p>Oh dear.</p> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=508542017-04-17T21:09:54ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>So apparently this is POSIX behavior for creating sparse files, and the Arvados Python SDK (and hence FUSE) currently don't support it (both writeto() and truncate() will throw exceptions if you try to increase the size of a file or start writing at a point past the end of the file).</p>
<p>This messes up tools which rely on it this behavior that we need to support. We should implement it.</p> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=508582017-04-18T13:37:57ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Subject</strong> changed from <i>S3 multipart download truncated writing to keep</i> to <i>[SDK] Support writes to offsets beyond end of file</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li><li><strong>Target version</strong> set to <i>2017-04-26 sprint</i></li></ul> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=509302017-04-20T16:44:40ZTom Cleggtom@curii.com
<ul></ul>11510-sdk-extend-files @ <a class="changeset" title="11510: Fix longstanding bug in replace_range() when appending data from a buffer that was written..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f5638851ddb5a859a54a16fe963ee98591af17a9">f5638851ddb5a859a54a16fe963ee98591af17a9</a>
<ul>
<li>remove debugging print statements in arvfile.py → ArvadosFile → writeto()</li>
<li>remove unused PADDING_BLOCK_LOCATOR in config.py (or perhaps move it into arvados_testutil and use it in the test cases)</li>
<li>I don't have any particular reason to suspect this won't work, and it doesn't become any more necessary now than before, but... it seems like we should test truncating a file in such a way that one or more blocks become unreferenced.</li>
<li>test_sparse_write3 might be even more edge-casey if it left out i=3, so <code>self.assertEqual(writer.read(), "000011112222\x00\x00\x00\x004444")</code> ...?</li>
<li>test seek to a negative offset</li>
<li>test seek past end of file, read (returns ''), seek back to non-sparse portion using SEEK_CUR, check file size did not change</li>
<li>test return value of seek() is new file position</li>
</ul> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=509312017-04-20T17:28:03ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
11510-sdk-extend-files @ <a class="changeset" title="11510: Fix longstanding bug in replace_range() when appending data from a buffer that was written..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f5638851ddb5a859a54a16fe963ee98591af17a9">f5638851ddb5a859a54a16fe963ee98591af17a9</a>
<ul>
<li>remove debugging print statements in arvfile.py → ArvadosFile → writeto()</li>
</ul>
</blockquote>
<p>Fixed.</p>
<blockquote>
<ul>
<li>remove unused PADDING_BLOCK_LOCATOR in config.py (or perhaps move it into arvados_testutil and use it in the test cases)</li>
</ul>
</blockquote>
<p>Moved it into a docstring.</p>
<blockquote>
<ul>
<li>I don't have any particular reason to suspect this won't work, and it doesn't become any more necessary now than before, but... it seems like we should test truncating a file in such a way that one or more blocks become unreferenced.</li>
</ul>
</blockquote>
<p>Added test_truncate3() which does this.</p>
<blockquote>
<ul>
<li>test_sparse_write3 might be even more edge-casey if it left out i=3, so <code>self.assertEqual(writer.read(), "000011112222\x00\x00\x00\x004444")</code> ...?</li>
</ul>
</blockquote>
<p>Added test_sparse_write4() which does this.</p>
<blockquote>
<ul>
<li>test seek to a negative offset</li>
</ul>
</blockquote>
<p>test_seek_min_zero() tests this.</p>
<blockquote>
<ul>
<li>test seek past end of file, read (returns ''), seek back to non-sparse portion using SEEK_CUR, check file size did not change</li>
<li>test return value of seek() is new file position</li>
</ul>
</blockquote>
<p>Also tested in test_truncate3().</p>
<p>Now @ <a class="changeset" title="11510: Fix truncate() to not change file pointer. Additional tests." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5180238a10bd15302a1c15b9a428f2fdeeabdf4e">5180238a10bd15302a1c15b9a428f2fdeeabdf4e</a></p> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=509362017-04-20T17:53:38ZTom Cleggtom@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote><blockquote>
<ul>
<li>test return value of seek() is new file position</li>
</ul>
</blockquote>
<p>Also tested in test_truncate3().</p>
</blockquote>
<p>seek return value is still untested and seems to be incorrect. If I add a check:</p>
<pre>
Traceback (most recent call last):
File "/home/tom/src/arvados/sdk/python/tests/test_arvfile.py", line 169, in test_write_to_end
self.assertEqual(5, writer.seek(5, os.SEEK_SET))
AssertionError: 5 != None
</pre>
<p><a class="external" href="https://docs.python.org/2/library/io.html#io.IOBase.seek">https://docs.python.org/2/library/io.html#io.IOBase.seek</a></p>
<p>Rest LGTM, thanks</p> Arvados - Bug #11510: [SDK] Support writes to offsets beyond end of filehttps://dev.arvados.org/issues/11510?journal_id=510632017-04-26T14:07:21ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul>