https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422019-05-21T19:27:47ZArvadosArvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=746392019-05-21T19:27:47ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Parent task</strong> deleted (<del><i>#14532</i></del>)</li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=746402019-05-21T19:28:08ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Tracker</strong> changed from <i>Task</i> to <i>Idea</i></li><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>To Be Groomed</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=746422019-05-21T19:28:25ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed behind-schedule" href="/issues/14532">Idea #14532</a>: [Epic] Port to Python 3 to for Python 2 sunset in December 2019</i> added</li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=746442019-05-21T19:28:55ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-1 status-7 priority-4 priority-default closed" href="/issues/15251">Bug #15251</a>: arv-mount supports python 3</i> added</li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=749522019-05-29T14:49:24ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>To Be Groomed</i> to <i>Arvados Future Sprints</i></li><li><strong>Story points</strong> set to <i>2.0</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=752102019-06-05T15:37:06ZEric Biagiotti
<ul><li><strong>Assigned To</strong> set to <i>Eric Biagiotti</i></li><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2019-06-19 Sprint</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=753672019-06-10T14:16:24ZWard Vandewegeward@curii.com
<ul><li><strong>Release</strong> set to <i>22</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=756372019-06-19T13:13:58ZEric Biagiotti
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=756382019-06-19T13:14:08ZEric Biagiotti
<ul><li><strong>Target version</strong> changed from <i>2019-06-19 Sprint</i> to <i>2019-07-03 Sprint</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=760332019-07-03T13:15:12ZEric Biagiotti
<ul><li><strong>Target version</strong> changed from <i>2019-07-03 Sprint</i> to <i>2019-07-17 Sprint</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=763442019-07-16T19:13:23ZEric Biagiotti
<ul></ul><p>Latest at <a class="changeset" title="Merge branch '14965-arv-mount-py-three' of git.curoverse.com:arvados into 14965-arv-mount-py-thre..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/16d9c611d5026e50692e1b3d0fefb951de5afa7b">16d9c611d5026e50692e1b3d0fefb951de5afa7b</a></p>
<ul>
<li>arv-mount supports Python 3</li>
<li>Besides the futurize changes, I mostly fixed test encoding issues. I tried to minimize changes to our actual fuse code.</li>
</ul>
<p>Testing:</p>
<ul>
<li><a class="external" href="https://ci.curoverse.com/view/Developer/job/developer-run-tests/1400/">https://ci.curoverse.com/view/Developer/job/developer-run-tests/1400/</a></li>
<li>Also Manually tested arv-mount installation and some functionality in py3/py2 virtualenvs</li>
</ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=763612019-07-17T13:46:23ZEric Biagiotti
<ul><li><strong>Target version</strong> changed from <i>2019-07-17 Sprint</i> to <i>Arvados Future Sprints</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=763632019-07-17T13:48:45ZEric Biagiotti
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2019-07-31 Sprint</i></li></ul> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=764072019-07-17T19:00:01ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Sorry for the delay, I was doing some manual testing on arvbox and it took me some time.</p>
<p>As we have had performance issues in the past with the fuse driver, some of my comments are headed towards that, following this Python 2/3 "cheat sheet": <a class="external" href="http://python-future.org/compatible_idioms.html">http://python-future.org/compatible_idioms.html</a></p>
<ul>
<li>When running tests, I’m getting lots of warning messages like this one: <code>/usr/lib/python3.5/json/decoder.py:355: ResourceWarning: unclosed <ssl.SSLSocket fd=21, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1', 42742), raddr=('127.0.0.1', 47393)></code> — as we talked on zoom, I’m worried about resources not being fully freed after a test run (may create issues on Jenkins slaves), but if that’s not the case, I think it would still be good to avoid them for log readability purposes.</li>
<li>Also got some deprecation warnings: <code>arvados/services/fuse/tests/test_exec.py:65: DeprecationWarning: Please use assertRegex instead.</code> (check <a class="external" href="https://six.readthedocs.io/#unittest-assertions">https://six.readthedocs.io/#unittest-assertions</a> for py2/py3 compatibility)</li>
<li><code>services/fuse/arvados_fuse/fusedir.py</code> - Lines 170 & 178: There’re some <code>dict.itervalues()</code> that were translated to <code>dict.values()</code> calls, that would use extra memory on python2, as it’s described on <a class="external" href="http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items">http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items</a>, seems that the way to go is calling <code>itervalues(dict)</code>?</li>
<li><code>services/fuse/arvados_fuse/__init__.py:L276</code> — Using <code>dict.items()</code> is inefficient on Py2 as described on the previously mentioned guide list, maybe use <code>viewitems(dict)</code> or <code>iteritems(dict)</code>?</li>
<li><code>services/fuse/arvados_fuse/fusedir.py:L99</code> and <code>services/fuse/arvados_fuse/__init__.py:L727</code> — Using <code>list(dict.items())</code> is inefficient on Py2 as described on the previously mentioned guide list, maybe use <code>listitems(dict)</code>?</li>
</ul>
<p>As for the manual tests, at first I initially got my arvbox shell instance to freeze when trying to write a file using py3 after trying it first with py2, but later on I did a proper test initially starting a shell and trying py2 and later closing the shell and re-starting it and doing the same with py3, and it worked without issues, so maybe I didn't unmounted the filesystem correctly at the first try.</p>
<p>Also, I think the documentation needs updating to state that it's compatible with Python3 <a class="external" href="https://doc.arvados.org/sdk/python/arvados-fuse.html">https://doc.arvados.org/sdk/python/arvados-fuse.html</a></p> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=764312019-07-19T13:26:01ZEric Biagiotti
<ul></ul><p>Latest at <a class="changeset" title="14965: Improves inode item view Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@verita..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/3ce47c5d012f9dbd6335ad0ae092eb9cc8a3022e">3ce47c5d012f9dbd6335ad0ae092eb9cc8a3022e</a></p>
<p>Unit tests: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1404/">https://ci.curoverse.com/job/developer-run-tests/1404/</a> - Some workbench integration failures, but I ran them locally and they passed. <br />Conformance tests: <a class="external" href="https://ci.curoverse.com/view/CWL/job/arvados-cwl-conformance-tests/193/">https://ci.curoverse.com/view/CWL/job/arvados-cwl-conformance-tests/193/</a></p>
<p>Also retested manually in virtualenvs.</p>
<p>Lucas Di Pentima wrote:</p>
<blockquote>
<ul>
<li>When running tests, I’m getting lots of warning messages like this one: <code>/usr/lib/python3.5/json/decoder.py:355: ResourceWarning: unclosed <ssl.SSLSocket fd=21, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1', 42742), raddr=('127.0.0.1', 47393)></code> — as we talked on zoom, I’m worried about resources not being fully freed after a test run (may create issues on Jenkins slaves), but if that’s not the case, I think it would still be good to avoid them for log readability purposes.</li>
</ul>
</blockquote>
<p>I was able to reduce some of these with <a class="changeset" title="14965: Fixes resource warning when resetting run_test_server Arvados-DCO-1.1-Signed-off-by: Eric..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/a1a1a8756149d5d05526503fd1e85115ea083fac">a1a1a8756149d5d05526503fd1e85115ea083fac</a>, but the json and yaml ones have been difficult to pin down. I'm fairly certain these pertain to http2libs keep-alive feature <a class="external" href="https://github.com/httplib2/httplib2#keep-alive">https://github.com/httplib2/httplib2#keep-alive</a>. Similar issues have been reported here <a class="external" href="https://github.com/kennethreitz/requests/issues/3912">https://github.com/kennethreitz/requests/issues/3912</a>. The reason these only show up in testing is because they are normally suppressed by default in python, but unittest resets warning suppression.</p>
<p>They don't seem to be causing problems, but their impact on jenkins slaves is worth keeping in mind. It's also worth noting that these warnings show up in majority of our py3 test suites (sdk/python, sdk/cwl). For these reasons, I would like to leave the output as is and make a separate issue to investigate further. Let me know if you think this is satisfactory and I will make the issue.</p>
<blockquote>
<ul>
<li>Also got some deprecation warnings: <code>arvados/services/fuse/tests/test_exec.py:65: DeprecationWarning: Please use assertRegex instead.</code> (check <a class="external" href="https://six.readthedocs.io/#unittest-assertions">https://six.readthedocs.io/#unittest-assertions</a> for py2/py3 compatibility)</li>
</ul>
</blockquote>
<p>Thanks for finding that! Fixed in latest.</p>
<blockquote>
<ul>
<li><code>services/fuse/arvados_fuse/fusedir.py</code> - Lines 170 & 178: There’re some <code>dict.itervalues()</code> that were translated to <code>dict.values()</code> calls, that would use extra memory on python2, as it’s described on <a class="external" href="http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items">http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items</a>, seems that the way to go is calling <code>itervalues(dict)</code>?</li>
<li><code>services/fuse/arvados_fuse/__init__.py:L276</code> — Using <code>dict.items()</code> is inefficient on Py2 as described on the previously mentioned guide list, maybe use <code>viewitems(dict)</code> or <code>iteritems(dict)</code>?</li>
<li><code>services/fuse/arvados_fuse/fusedir.py:L99</code> and <code>services/fuse/arvados_fuse/__init__.py:L727</code> — Using <code>list(dict.items())</code> is inefficient on Py2 as described on the previously mentioned guide list, maybe use <code>listitems(dict)</code>?</li>
</ul>
</blockquote>
<p>Fixed all three in latest.</p>
<blockquote>
<p>Also, I think the documentation needs updating to state that it's compatible with Python3 <a class="external" href="https://doc.arvados.org/sdk/python/arvados-fuse.html">https://doc.arvados.org/sdk/python/arvados-fuse.html</a></p>
</blockquote>
<p>That page will be updated once we create the py3 arv-mount package. I'm tracking it here: <a class="external" href="https://dev.arvados.org/issues/15473">https://dev.arvados.org/issues/15473</a></p> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=764322019-07-19T14:02:28ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Eric Biagiotti wrote:</p>
<blockquote>
<p>Latest at <a class="changeset" title="14965: Improves inode item view Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@verita..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/3ce47c5d012f9dbd6335ad0ae092eb9cc8a3022e">3ce47c5d012f9dbd6335ad0ae092eb9cc8a3022e</a> <br />[...]<br />They don't seem to be causing problems, but their impact on jenkins slaves is worth keeping in mind. It's also worth noting that these warnings show up in majority of our py3 test suites (sdk/python, sdk/cwl). For these reasons, I would like to leave the output as is and make a separate issue to investigate further. Let me know if you think this is satisfactory and I will make the issue.</p>
</blockquote>
<p>Yes, thanks for doing the research! I wasn't aware of the warnings happening on other py3 test runs.</p>
<blockquote>
<p>[...]</p>
<blockquote>
<p>Also, I think the documentation needs updating to state that it's compatible with Python3 <a class="external" href="https://doc.arvados.org/sdk/python/arvados-fuse.html">https://doc.arvados.org/sdk/python/arvados-fuse.html</a></p>
</blockquote>
<p>That page will be updated once we create the py3 arv-mount package. I'm tracking it here: <a class="external" href="https://dev.arvados.org/issues/15473">https://dev.arvados.org/issues/15473</a></p>
</blockquote>
<p>Great, so this LGTM, thanks!</p> Arvados - Idea #14965: Port arv-mount to Python 3https://dev.arvados.org/issues/14965?journal_id=764372019-07-19T15:34:41ZEric Biagiotti
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul>