https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422015-11-09T19:45:53ZArvadosArvados - Bug #6923: [Crunch] crunch-job install script calls `dpkg-query` - needs a cross-distro solutionhttps://dev.arvados.org/issues/6923?journal_id=321892015-11-09T19:45:53ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assigned To</strong> set to <i>Brett Smith</i></li><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2015-11-11 sprint</i></li></ul> Arvados - Bug #6923: [Crunch] crunch-job install script calls `dpkg-query` - needs a cross-distro solutionhttps://dev.arvados.org/issues/6923?journal_id=321982015-11-09T22:19:45ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Story points</strong> set to <i>0.5</i></li></ul> Arvados - Bug #6923: [Crunch] crunch-job install script calls `dpkg-query` - needs a cross-distro solutionhttps://dev.arvados.org/issues/6923?journal_id=322632015-11-11T22:45:04ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Target version</strong> changed from <i>2015-11-11 sprint</i> to <i>2015-12-02 sprint</i></li></ul> Arvados - Bug #6923: [Crunch] crunch-job install script calls `dpkg-query` - needs a cross-distro solutionhttps://dev.arvados.org/issues/6923?journal_id=325822015-11-18T18:28:18ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Reviewing <a class="changeset" title="6923: Improve Arvados SDK version logging in Crunch run script. * Use a mechanism that works in ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/6c40b57dad64bea44694dd5cafd719f0338dd745">6c40b57</a></p>
<p>A few notes:</p>
<ul>
<li>I tested it by running a crunch job that includes <code>arvados_sdk_version</code> and it ran successfully.</li>
<li>Since <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: [CWL] binary run-command shim for CWL (Resolved)" href="https://dev.arvados.org/issues/7582">#7582</a> it only runs the "pipe to perl" bootstrap script when <code>arvados_sdk_version</code> is specified in <code>runtime_constraints</code>. I think this means that "Crunch script run mode" can be streamlined for just that case. This also suggests an inability to install the SDK (such as missing virtualenv) should be a fatal error instead of a warning.</li>
<li>Lines 965-970 are the simplified bash bootstrap script for when <code>arvados_sdk_version</code> is <em>not</em> specified; it would make sense to move <code>pysdk_version_cmd</code> to here (with errors suppressed, since this script is intended to work even if neither the Arvados SDK or even Python are available)</li>
</ul>
<p>I'm not sure if this constitutes scope creep; the existing branch does solve the stated problem so I'd be willing to consider merging as-is.</p> Arvados - Bug #6923: [Crunch] crunch-job install script calls `dpkg-query` - needs a cross-distro solutionhttps://dev.arvados.org/issues/6923?journal_id=325872015-11-18T20:37:52ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<ul>
<li>Since <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: [CWL] binary run-command shim for CWL (Resolved)" href="https://dev.arvados.org/issues/7582">#7582</a> it only runs the "pipe to perl" bootstrap script when <code>arvados_sdk_version</code> is specified in <code>runtime_constraints</code>. I think this means that "Crunch script run mode" can be streamlined for just that case. This also suggests an inability to install the SDK (such as missing virtualenv) should be a fatal error instead of a warning.</li>
</ul>
</blockquote>
<p>Technically, the Python SDK isn't the only SDK we have. The <code>__DATA__</code> section makes at least some effort to install the Ruby and Perl SDKs as well, by extracting them and extending RUBYLIB and PERLLIB accordingly. It's arguable whether that's actually useful, but it's the current behavior, and that means that failing to install any one SDK is not fatal—maybe the Crunch script doesn't care about that specific language SDK. In turn, I don't think there's much opportunity to streamline this section.</p>
<p>I'm open to a discussion of narrowing the scope of this code, but because that's a bigger design discussion than this relatively straightforward bugfix, I'd rather handle it separately.</p>
<blockquote>
<ul>
<li>Lines 965-970 are the simplified bash bootstrap script for when <code>arvados_sdk_version</code> is <em>not</em> specified; it would make sense to move <code>pysdk_version_cmd</code> to here (with errors suppressed, since this script is intended to work even if neither the Arvados SDK or even Python are available)</li>
</ul>
</blockquote>
<p>Added in <a class="changeset" title="6923: crunch-job reports PySDK version with minimal bootstrapping." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/883322d0ad56aba5b6aa6ffc2bc67d8c30e3f2a0">883322d</a>. Thanks.</p> Arvados - Bug #6923: [Crunch] crunch-job install script calls `dpkg-query` - needs a cross-distro solutionhttps://dev.arvados.org/issues/6923?journal_id=325902015-11-18T21:45:01ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Brett Smith wrote:</p>
<blockquote>
<p>Peter Amstutz wrote:</p>
<blockquote>
<ul>
<li>Since <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: [CWL] binary run-command shim for CWL (Resolved)" href="https://dev.arvados.org/issues/7582">#7582</a> it only runs the "pipe to perl" bootstrap script when <code>arvados_sdk_version</code> is specified in <code>runtime_constraints</code>. I think this means that "Crunch script run mode" can be streamlined for just that case. This also suggests an inability to install the SDK (such as missing virtualenv) should be a fatal error instead of a warning.</li>
</ul>
</blockquote>
<p>Technically, the Python SDK isn't the only SDK we have. The <code>__DATA__</code> section makes at least some effort to install the Ruby and Perl SDKs as well, by extracting them and extending RUBYLIB and PERLLIB accordingly. It's arguable whether that's actually useful, but it's the current behavior, and that means that failing to install any one SDK is not fatal—maybe the Crunch script doesn't care about that specific language SDK. In turn, I don't think there's much opportunity to streamline this section.</p>
<p>I'm open to a discussion of narrowing the scope of this code, but because that's a bigger design discussion than this relatively straightforward bugfix, I'd rather handle it separately.</p>
</blockquote>
<p>I agree.</p>
<blockquote><blockquote>
<ul>
<li>Lines 965-970 are the simplified bash bootstrap script for when <code>arvados_sdk_version</code> is <em>not</em> specified; it would make sense to move <code>pysdk_version_cmd</code> to here (with errors suppressed, since this script is intended to work even if neither the Arvados SDK or even Python are available)</li>
</ul>
</blockquote>
<p>Added in <a class="changeset" title="6923: crunch-job reports PySDK version with minimal bootstrapping." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/883322d0ad56aba5b6aa6ffc2bc67d8c30e3f2a0">883322d</a>. Thanks.</p>
</blockquote>
<p>Tested successfully. LGTM, please merge.</p> Arvados - Bug #6923: [Crunch] crunch-job install script calls `dpkg-query` - needs a cross-distro solutionhttps://dev.arvados.org/issues/6923?journal_id=325942015-11-18T23:05:10ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset arvados|commit:155b1aa3772eb2d9421c5bfc678320401eb3b058.</p>