Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422024-02-06T04:58:13ZArvados
Redmine Arvados - Idea #21451 (In Progress): running test suite in dockerhttps://dev.arvados.org/issues/214512024-02-06T04:58:13ZPeter Amstutzpeter.amstutz@curii.comArvados - Idea #21446 (New): Stop using `apt-key add` everywherehttps://dev.arvados.org/issues/214462024-02-02T18:52:50ZBrett Smithbrett.smith@curii.com
<p>Quoting the man page:</p>
<blockquote>
<p>apt-key(8) will last be available in Debian 11 and Ubuntu 22.04.</p>
<p>Note: Instead of using [<code>apt-key add</code>] a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either "gpg" or "asc" as file extension.</p>
</blockquote> Arvados - Idea #21409 (In Progress): Curii bioinformatics producthttps://dev.arvados.org/issues/214092024-01-23T16:12:12ZPeter Amstutzpeter.amstutz@curii.com
<p>WIP</p>
<p><a class="external" href="https://docs.google.com/document/d/1-BdkC9qjvIlhjQLW1QF2IO8Q4DRx-QAdGKieHzpjBwE/edit?usp=sharing">https://docs.google.com/document/d/1-BdkC9qjvIlhjQLW1QF2IO8Q4DRx-QAdGKieHzpjBwE/edit?usp=sharing</a></p> Arvados - Idea #21331 (New): website noteshttps://dev.arvados.org/issues/213312024-01-03T14:11:10ZPeter Amstutzpeter.amstutz@curii.com
<p>Some thoughts about things we could emphasize on arvados.org</p>
<ul>
<li>We address scale as a separate point but I think it makes more sense to include it in the data/workflow management points:</li>
</ul>
<blockquote>
<p>Manage large biomedical data sets for everything from genomics to imaging</p>
</blockquote>
<p>"everything from genomics to imaging" isn't that broad. Something like</p>
<p>"Manage file-based biomedical data at petabyte scale for genomics, imaging, video, or anything else."</p>
<blockquote>
<p>reliably run containerized workflows at scale.</p>
</blockquote>
<p>Should be more specific like "reliably run thousands of containerized workflows in parallel on cloud or HPC"</p>
<p>Maybe we can replace "scale" with something else.</p>
<ul>
<li>Also, it occurs to me that we don't have a link to our github page. Even though github isn't our primary development platform, people are going to look for it. Especially since it isn't particularly obvious how to access our main repository, which is public but still mostly for our internal use.</li>
</ul> Arvados - Idea #21017 (In Progress): Convert cookbook container recipes into SDK methodshttps://dev.arvados.org/issues/210172023-09-28T15:56:22ZBrett Smithbrett.smith@curii.com
<p>A lot of code in the Python SDK cookbook that coordinates low-level API calls should just be methods that are defined in the SDK itself. Why force every user to copy-paste these code snippets when we can just give them the method directly?</p>
<p>Here are common general-purpose container/request functions that could be useful. I'm not sure what module they should go in. It might make sense to add them to a new <code>arvados.container</code> module.</p>
<pre><code class="python syntaxhl"><span class="k">def</span> <span class="nf">lookup</span><span class="p">(</span>
<span class="n">client</span><span class="p">:</span> <span class="n">ArvadosAPIClient</span><span class="p">,</span>
<span class="n">container</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">Container</span> <span class="o">|</span> <span class="n">ContainerRequest</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-></span> <span class="n">Container</span> <span class="o">|</span> <span class="bp">None</span><span class="p">:</span> <span class="p">...</span>
<span class="c1"># If container is a str, it's a UUID for one of the other types, fetch that.
# If it's a container request, return None if container_uuid is None,
# else fetch that.
# Return the resulting container.
# Note that lookup should return None *only* if it handles a container
# request with no container_uuid set.
# Other problems like malformed UUID, other object types, container request
# without container_uuid selected, etc. should all raise their corresponding
# exceptions.
</span>
<span class="k">def</span> <span class="nf">container_started</span><span class="p">(</span>
<span class="n">client</span><span class="p">:</span> <span class="n">ArvadosAPIClient</span><span class="p">,</span>
<span class="n">container</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">Container</span> <span class="o">|</span> <span class="n">ContainerRequest</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="n">container_obj</span> <span class="o">=</span> <span class="n">lookup</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">container</span><span class="p">)</span>
<span class="k">if</span> <span class="n">container_obj</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">container_obj</span><span class="p">[</span><span class="s">'status'</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">{</span><span class="s">'Queued'</span><span class="p">,</span> <span class="s">'Locked'</span><span class="p">}</span>
<span class="k">def</span> <span class="nf">container_finished</span><span class="p">(</span>
<span class="n">client</span><span class="p">:</span> <span class="n">ArvadosAPIClient</span><span class="p">,</span>
<span class="n">container</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">Container</span> <span class="o">|</span> <span class="n">ContainerRequest</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="n">container_obj</span> <span class="o">=</span> <span class="n">lookup</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">container</span><span class="p">)</span>
<span class="k">if</span> <span class="n">container_obj</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">container_obj</span><span class="p">[</span><span class="s">'status'</span><span class="p">]</span> <span class="ow">in</span> <span class="p">{</span><span class="s">'Cancelled'</span><span class="p">,</span> <span class="s">'Complete'</span><span class="p">}</span>
<span class="k">def</span> <span class="nf">container_succeeded</span><span class="p">(</span>
<span class="n">client</span><span class="p">:</span> <span class="n">ArvadosAPIClient</span><span class="p">,</span>
<span class="n">container</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">Container</span> <span class="o">|</span> <span class="n">ContainerRequest</span><span class="p">,</span>
<span class="n">success</span><span class="p">:</span> <span class="n">typing</span><span class="p">.</span><span class="n">Container</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span><span class="o">=</span><span class="nb">frozenset</span><span class="p">([</span><span class="mi">0</span><span class="p">]),</span>
<span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="n">container_obj</span> <span class="o">=</span> <span class="n">lookup</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">container</span><span class="p">)</span>
<span class="k">return</span> <span class="p">(</span>
<span class="n">container_obj</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span>
<span class="ow">and</span> <span class="n">container_obj</span><span class="p">[</span><span class="s">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s">'Complete'</span>
<span class="ow">and</span> <span class="n">container_obj</span><span class="p">[</span><span class="s">'exit_code'</span><span class="p">]</span> <span class="ow">in</span> <span class="n">success</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">child_requests</span><span class="p">(</span>
<span class="n">client</span><span class="p">:</span> <span class="n">ArvadosAPIClient</span><span class="p">,</span>
<span class="n">container</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">Container</span> <span class="o">|</span> <span class="n">ContainerRequest</span><span class="p">,</span>
<span class="n">filters</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span><span class="o">=</span><span class="p">[],</span>
<span class="n">select</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span><span class="o">=</span><span class="p">[],</span>
<span class="n">depth</span><span class="p">:</span> <span class="nb">int</span> <span class="o">|</span> <span class="bp">None</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-></span> <span class="n">Iterator</span><span class="p">[</span><span class="n">ContainerRequest</span><span class="p">]:</span> <span class="p">...</span>
<span class="c1"># lookup container, yield from `keyset_list_all` calls, down to depth if given
</span>
<span class="k">def</span> <span class="nf">child_containers</span><span class="p">(</span>
<span class="n">client</span><span class="p">:</span> <span class="n">ArvadosAPIClient</span><span class="p">,</span>
<span class="n">container</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">Container</span> <span class="o">|</span> <span class="n">ContainerRequest</span><span class="p">,</span>
<span class="n">request_filters</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span><span class="o">=</span><span class="p">[],</span>
<span class="n">container_filters</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span><span class="o">=</span><span class="p">[],</span>
<span class="n">select</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span><span class="o">=</span><span class="p">[],</span>
<span class="n">depth</span><span class="p">:</span> <span class="nb">int</span> <span class="o">|</span> <span class="bp">None</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-></span> <span class="n">Iterator</span><span class="p">[</span><span class="n">Container</span><span class="p">]:</span> <span class="p">...</span>
<span class="c1"># Same dance as above, call `child_requests` to get container_uuids, then
# yield from `keyset_list_all` calls with those UUIDs, down to depth if given.
# Note that `child_requests` can be called with a very narrow `select` for
# optimization, and can add the filter `container_uuid is not null`.
</span></code></pre> Arvados - Idea #20435 (In Progress): CWL user guide releasehttps://dev.arvados.org/issues/204352023-04-26T16:03:36ZPeter Amstutzpeter.amstutz@curii.comArvados - Idea #20237 (New): fair scheduling in cloud dispatcherhttps://dev.arvados.org/issues/202372023-03-13T19:06:02ZPeter Amstutzpeter.amstutz@curii.com
<p>Customer expressed desire that when a workflow is submitted and then a new workflow is submitted later while the 1st workflow is still running, the dispatcher/scheduler should allocate scare resources between both workflows instead of the current priority system which is mostly first-come-first-served.</p>
<p>The scenario, as portrayed, is if there is a 5000 sample batch running, and another user submits a 10 sample test run, the 2nd user should not have to wait for the entire 5000 sample batch to complete before their run can make any progress.</p> Arvados - Idea #20107 (New): Research retry strategies when SDK API calls return 5xx errorshttps://dev.arvados.org/issues/201072023-02-15T16:49:37ZPeter Amstutzpeter.amstutz@curii.comArvados - Idea #19676 (In Progress): Turn data organization patterns deck into documentation page...https://dev.arvados.org/issues/196762022-10-26T20:29:17ZPeter Amstutzpeter.amstutz@curii.com
<p><a class="external" href="https://docs.google.com/presentation/d/1jIvMfDGmOwtAJXp7TuujUmazyp-1VZX9Qn78Tx0Hz5c/edit#slide=id.g10e5fa06e9b_0_458">https://docs.google.com/presentation/d/1jIvMfDGmOwtAJXp7TuujUmazyp-1VZX9Qn78Tx0Hz5c/edit#slide=id.g10e5fa06e9b_0_458</a></p> Arvados - Idea #19403 (New): Review eager requirement eval PRhttps://dev.arvados.org/issues/194032022-08-17T16:02:26ZPeter Amstutzpeter.amstutz@curii.com
<p><a class="external" href="https://github.com/common-workflow-language/cwltool/pull/1703">https://github.com/common-workflow-language/cwltool/pull/1703</a></p> Arvados - Idea #17274 (New): Add "viewer.commonwl.org" DNS aliashttps://dev.arvados.org/issues/172742021-01-20T16:27:08ZPeter Amstutzpeter.amstutz@curii.com
<p>Figure out who has access to commonwl.org domain</p> Arvados - Idea #16055 (In Progress): User guide update planninghttps://dev.arvados.org/issues/160552020-01-22T17:08:03ZPeter Amstutzpeter.amstutz@curii.com
<p>Make a specific plan for what we are going to do to update the documentation.</p>
<p>From discussion -- initial plan should be the minimal things that need to be updated 1:1 from workbench 1 documentation to workbench 2 documentation.</p>
<p>Follow up plan for a more complete overhaul:</p>
<p><a class="external" href="https://docs.google.com/document/d/1yOU2gTV21MhQ6uieKrRJgJDGZq1epFPojJomS2Tg6WI/edit?usp=sharing">https://docs.google.com/document/d/1yOU2gTV21MhQ6uieKrRJgJDGZq1epFPojJomS2Tg6WI/edit?usp=sharing</a></p>
<p>I think the biggest short term improvement would come from splitting the user guide into separate toplevel walkthrough/tutorial and reference sections and mainly reorganizing what we already have.</p> Tapestry - Idea #6257 (In Progress): Add public facing database dump, cron job and Arvados uploadhttps://dev.arvados.org/issues/62572015-06-05T20:56:01ZAbram Connellyabram.connelly@gmail.com
<p>There is code to do a dump of the public facing data. Merge it into Tapestry, create a cron job to automatically run on a daily basis and have it upload to an Arvados project.</p> Lightning - Idea #5001 (New): Create a API interface to the public tile library including a "beacon"https://dev.arvados.org/issues/50012015-01-16T18:29:36ZSarah Guthriesallyeguthrie@gmail.comLightning - Idea #4827 (New): Automatic Deployment Implementationhttps://dev.arvados.org/issues/48272014-12-15T22:59:30ZSarah Guthriesallyeguthrie@gmail.com
<p>Goal: have an automatic process to run tests and setup Lightning that is run when a commit is pushed to master</p>
Steps:
<ol>
<li>Research how to structure the git repository</li>
<li>Research/communicate with Ward how to set up automatic deployment</li>
<li>Start writing tests and deploying</li>
</ol>