Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422024-03-27T16:11:09ZArvados
Redmine Arvados - Task #21631 (New): Reviewhttps://dev.arvados.org/issues/216312024-03-27T16:11:09ZPeter Amstutzpeter.amstutz@curii.comArvados - Task #21630 (New): Reviewhttps://dev.arvados.org/issues/216302024-03-27T16:10:40ZPeter Amstutzpeter.amstutz@curii.comArvados - Task #21629 (New): Reviewhttps://dev.arvados.org/issues/216292024-03-27T16:10:35ZPeter Amstutzpeter.amstutz@curii.comArvados - Bug #21601 (In Progress): fpm virtualenv packages not using branch versions for depende...https://dev.arvados.org/issues/216012024-03-15T20:38:09ZPeter Amstutzpeter.amstutz@curii.com
<p><a class="external" href="https://dev.arvados.org/issues/19744#note-30">https://dev.arvados.org/issues/19744#note-30</a></p>
<p>The python3-arvados-cwl-runner_2.8.0~dev20240314145937-1_amd64.deb package has arvados-python-client 2.7.1 and crunchstat-summary 2.7.1, when it should have the dev versions from the same commit.</p>
<p>I went back and looked at earlier packages: python3-arvados-cwl-runner_2.7.1~rc3-1_amd64.deb has arvados-python-client 2.7.1rc3 (as expected) and python3-arvados-cwl-runner_2.7.0~dev20230908133938-1_amd64.deb has arvados-python-client 2.7.0.dev20230908133938 (also as expected).</p>
<p>My current theory is that this behavior got lost in the changes made in 20846-package-build-fixes, but I need to find out how it worked before.</p> Arvados - Task #21587 (In Progress): Review 21583-railsapi-base64-gemhttps://dev.arvados.org/issues/215872024-03-13T16:00:31ZPeter Amstutzpeter.amstutz@curii.comArvados - Bug #21583 (In Progress): Running RailsAPI with Passenger implicity requires Ruby 3.3 v...https://dev.arvados.org/issues/215832024-03-13T11:03:08ZBrett Smithbrett.smith@curii.com
<p>Some useful background:</p>
<ul>
<li>base64 has been a default Gem for a while, but it will not be included in Ruby 3.4, and Ruby 3.3 warns you about this.</li>
<li>To make the warning go away, libraries have started declaring a dependency on the base64 gem. The library that's relevant to our story is <a href="https://github.com/lostisland/faraday/commit/ea30bd0b543882f1cf26e75ac4e46e0705fa7e68" class="external">faraday</a>, which is used by our ruby-google-api-client fork.</li>
<li>The current version of base64 is 0.2.0. This version is included in Ruby 3.3. Older versions of Ruby we support have 0.1.1.</li>
</ul>
<p>With this background, when we started using our ruby-google-api-client fork in RailsAPI in <a class="changeset" title="21384: Update arvados-google-api-client in RailsAPI There's no functional need for this. The mai..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cbfdb1b66ab9c1b6e69d1c9cd589633386267177">cbfdb1b66ab9c1b6e69d1c9cd589633386267177</a>, <a class="source" href="https://dev.arvados.org/projects/arvados/repository/arvados/entry/services/api/Gemfile.lock">source:services/api/Gemfile.lock</a> gained a dependency on base64 0.2.0. This works fine in development, because Bundler can load this newer version before any code requires it.</p>
<p>However, Passenger loads the base64 Gem <em>before</em> it starts anything related to your application, including Bundler. Because of this, running RailsAPI behind Passenger with Ruby<3.3 now fails with this error in the Passenger log:</p>
<pre>[ E 2024-03-12 15:12:44.8347 907382/Tf age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/arvados-api/current: The application encountered the following error: You have already activated base64 0.1.1, but your Gemfile requires base64 0.2.0. Since base64 is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports base64 as a default gem. (Gem::LoadError)
</pre>
<p>This is the root cause of <a class="issue tracker-1 status-1 priority-4 priority-default parent" title="Bug: test-provision-ubuntu2004 intermittently times out waiting for the controller to come up (New)" href="https://dev.arvados.org/issues/21524">#21524</a>. Note how test-provision-ubuntu2004 started failing immediately after <a class="changeset" title="21384: Update arvados-google-api-client in RailsAPI There's no functional need for this. The mai..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cbfdb1b66ab9c1b6e69d1c9cd589633386267177">cbfdb1b66ab9c1b6e69d1c9cd589633386267177</a>:</p>
<p><img src="https://dev.arvados.org/attachments/download/3541/clipboard-202403130702-u2cmy.png" alt="" /></p>
<p>What can we do?</p>
<p>There is no single version of base64 that we can lock to that will keep everyone happy. The current lock breaks Ruby<3.3. If we change the lock to base64 0.1.1, we'll break Ruby>=3.3.</p>
<p>We cannot address the problem indirectly by tweaking our faraday dependency. We need version ~>2.8.0 to keep compatibility with the range of Ruby versions we're trying to support, and all those releases declare the base64 dependency.</p>
<p><a href="https://myrtana.sk/articles/my-passenger-was-really-old" class="external">This random blog post with cool styling</a> says you can upgrade Passenger, but note they upgrade Passenger to the version in bookworm, which is the version I'm testing and have reproduced this problem with:</p>
<pre>% apt list --installed '*passenger*'
libnginx-mod-http-passenger/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed]
passenger-dev/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed,automatic]
passenger-doc/bookworm,now 1:6.0.20-1~bookworm1 all [installed,automatic]
passenger/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed,automatic]
</pre>
<p>We could just revert <a class="changeset" title="21384: Update arvados-google-api-client in RailsAPI There's no functional need for this. The mai..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cbfdb1b66ab9c1b6e69d1c9cd589633386267177">cbfdb1b66ab9c1b6e69d1c9cd589633386267177</a>. That has all the downsides implied by the commit message, but it would work.</p>
<p>We can just cheat and remove the lock by hand, but then we have to remember to keep doing that every time we update a RailsAPI gem for as long as we support Ruby<3.3. That sucks. We could write an extremely stupid test to help us remember this I guess.</p>
<p>🤷</p> Arvados - Feature #21572 (New): Reorganize doc/user/getting_started/setup-cli.html to prioritize ...https://dev.arvados.org/issues/215722024-03-04T20:31:38ZBrett Smithbrett.smith@curii.com
<p>User requested that this page just have a block of commands they can copy and paste to set everything up. I don't know if we can get 100% there right now, but we could probably get a lot closer, and good enough, with reasonable effort. Right now the page expects you to read a separate install page for each tool, which is a lot of hoop-jumping. Planned sections:</p>
<ol>
<li>Install with Debian/Ubuntu packages: Set up the apt repository, update, install all the client tool packages</li>
<li>Install with RHEL family packages: Set up the yum repository, update, install all the client tool packages</li>
<li>Install without privileges: Probably break this up into subsections to go from what most people will find easiest vs. hardest. Explain what you get at each step and let users know they can jump off whenever.
<ol>
<li>The Python tools: run <code>python3 -m venv</code>, <code>pip install arvados-cwl-runner arvados_fuse crunchstat_summary</code></li>
<li>If <a class="issue tracker-6 status-1 priority-4 priority-default" title="Idea: Publish standalone binaries for arvados-client, other Go client tools (New)" href="https://dev.arvados.org/issues/20727">#20727</a> is done: Download <code>arvados-client</code> to the right place</li>
<li>The Ruby tools: <code>gem install arvados-cli</code>—I'm less sure what's an appropriate incantation for this</li>
</ol></li>
</ol>
<p>The page can then link to separate pages for individual tools, for more information or for people who want to be more selective for whatever reason.</p> Arvados - Bug #21571 (New): Documentation should call it "arv-mount" rather than "FUSE Driver"https://dev.arvados.org/issues/215712024-03-04T17:09:03ZBrett Smithbrett.smith@curii.com
<p>"FUSE Driver" is a meaningless name to people who don't know what "FUSE" is, which is most people. The documentation should refer to the tool as "arv-mount" as much as possible, since that's a distinctive tool name and more people understand what a "mount" is generally (not a ton more, but still). If necessary the documentation can explain that arv-mount is implemented using FUSE, but that shouldn't be an identifier.</p> Arvados - Task #21563 (New): Reviewhttps://dev.arvados.org/issues/215632024-02-28T17:04:25ZPeter Amstutzpeter.amstutz@curii.comArvados - Bug #21434 (New): Follow up fix to schema saladhttps://dev.arvados.org/issues/214342024-01-31T16:46:46ZPeter Amstutzpeter.amstutz@curii.com
<p><a class="external" href="https://github.com/common-workflow-language/schema_salad/issues/766">https://github.com/common-workflow-language/schema_salad/issues/766</a></p> Arvados - Task #20510 (In Progress): Reviewhttps://dev.arvados.org/issues/205102023-05-10T20:46:40ZPeter Amstutzpeter.amstutz@curii.comArvados - Idea #20435 (In Progress): CWL user guide releasehttps://dev.arvados.org/issues/204352023-04-26T16:03:36ZPeter Amstutzpeter.amstutz@curii.comArvados - Support #20053 (In Progress): schema-salad codegen returns line numbershttps://dev.arvados.org/issues/200532023-02-01T18:12:43ZPeter Amstutzpeter.amstutz@curii.com
<p><a class="external" href="https://github.com/common-workflow-language/schema_salad/pull/647">https://github.com/common-workflow-language/schema_salad/pull/647</a></p> Arvados - Feature #19982 (In Progress): Ability to know when a container died because of spot ins...https://dev.arvados.org/issues/199822023-01-25T17:04:12ZPeter Amstutzpeter.amstutz@curii.com
<p>New arvados-cwl-runner behavior when spot instances are enabled</p>
<ul>
<li>When submitting spot instance, don't retry</li>
<li>Ability to detect when a container failed due to reclaimed spot instance (<a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: Detect and log spot instance interruption notices (Resolved)" href="https://dev.arvados.org/issues/19961">#19961</a>)</li>
<li>Exit code to indicate workflow failed due to spot instance</li>
<li>Option to automatically re-submit as reserved instance</li>
</ul> Arvados - Feature #16316 (In Progress): a-c-r handles resource range requests (especially CPU) an...https://dev.arvados.org/issues/163162020-04-08T12:48:43ZPeter Amstutzpeter.amstutz@curii.com
<p>Implement a version of <a href="https://github.com/common-workflow-language/cwltool/blob/72e4ffb7814ec846c12ec15df82a4243654fe053/cwltool/executors.py#L286" class="external">select_resources</a> for Arvados.</p>
<p>You can get a dictionary of instance types with this:</p>
<p><code>api.config()["InstanceTypes"]</code></p>
<p>The select_resources method should, at minimum, accept a range of CPU core values (e.g. coresMin: 4, coresMax: 16) and then check the available InstanceTypes and assign the greatest core count available. For example, if the system is only configured with 2, 4, and 8 core nodes, it should assign 8 cores since it is in the range (4 - 16).</p>
<p>RAM and disk can also have a range. Just return the minimum value for now (this is the existing behavior).</p>
<p>Tell cwltool to use your <code>select_resources</code> method by setting the object field <code>runtimeContext.select_resources</code>.</p>