Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422023-12-21T16:55:13ZArvados
Redmine Arvados - Bug #21314 (New): a-d-c should cancel a container if it can't be loadedhttps://dev.arvados.org/issues/213142023-12-21T16:55:13ZTom Cleggtom@curii.com
<p>If a container's "mounts" field is invalid, a-d-c logs this, and keeps trying.</p>
<code class="json syntaxhl"><span class="p">{</span><span class="nl">"ClusterID"</span><span class="p">:</span><span class="s2">"irdev"</span><span class="p">,</span><span class="nl">"ContainerUUID"</span><span class="p">:</span><span class="s2">"<a href="https://arvadosapi.com/xxxxx-dz642-xxxxxxxxxxxxxxx">xxxxx-dz642-xxxxxxxxxxxxxxx</a>"</span><span class="p">,</span><span class="nl">"PID"</span><span class="p">:</span><span class="mi">2037423</span><span class="p">,</span><span class="nl">"error"</span><span class="p">:</span><span class="s2">"json: cannot unmarshal array into Go struct field Container.mounts of type arvados.Mount"</span><span class="p">,</span><span class="nl">"level"</span><span class="p">:</span><span class="s2">"warning"</span><span class="p">,</span><span class="nl">"msg"</span><span class="p">:</span><span class="s2">"error getting mounts"</span><span class="p">,</span><span class="nl">"time"</span><span class="p">:</span><span class="s2">"2023-12-13T20:34:41.064140517Z"</span><span class="p">}</span><span class="w">
</span></code>
<p>In this situation, the offending container should be cancelled.</p> Arvados - Bug #21187 (New): a-c-r should detect and warn when arv:IntermediateOutput outputTTL is...https://dev.arvados.org/issues/211872023-11-09T19:31:33ZTom Cleggtom@curii.com
<p>Currently, if outputTTL is set too low and a workflow tries to use intermediate data after it has already been trashed, a-c-r may read a intermediate collection manifest successfully (before trash time) but then fail to save it later (after trash time) in a combined collection. In that case the user ends up getting a python stack trace ending in a 403 error (invalid blob signature).</p>
<p>a-c-r should warn the user when the duration the current workflow has been running exceeds outputTTL (this is probably a good indicator the user should increase outputTTL even if it hasn't actually broken anything yet)</p>
<p>a-c-r should also report a more helpful error message when it fails to create a collection due to expired blob signatures. This could be done by checking for a 403 error from the create call and/or checking the expiry times (given as hexadecimal unix times) on the blob signatures in the manifest text.</p> Arvados - Feature #21133 (New): Add diagnostics checks for container log APIhttps://dev.arvados.org/issues/211332023-10-20T15:18:09ZTom Cleggtom@curii.com
If it runs a container, <code>arvados-server diagnostics</code> should
<ol>
<li>access the container log endpoint while waiting for the container to finish, to make sure it returns a valid response (due to races, it's not necessarily an error if it can't get any actual log data before the container finishes, but it's an error if it receives a 502 error at any point during the container lifecycle, for example).</li>
<li>access the container log endpoint after the container has finished, to make sure the controller→webdav communication works correctly.</li>
</ol>
<p>It is possible for a cluster to be misconfigured such that logs work only for unfinished containers, or only for finished containers, so <code>diagnostics</code> should do its best to check for both problems.</p> Arvados - Feature #20471 (New): Admin CLI for managing dispatcher / cloud VMshttps://dev.arvados.org/issues/204712023-05-02T20:48:54ZTom Cleggtom@curii.com
<p><code>arvados-server instance list</code></p>
<p><code>arvados-server instance kill -reason "optional reason" {instanceID|containerUUID}</code></p>
<p>See <a class="wiki-page" href="https://dev.arvados.org/projects/arvados/wiki/Admin_CLI_for_managing_dispatcher__cloud_VMs">Admin CLI for managing dispatcher / cloud VMs</a></p> Arvados - Feature #20310 (New): Add server-sent events stream at .../containers/{uuid}/log_eventshttps://dev.arvados.org/issues/203102023-04-06T03:23:06ZTom Cleggtom@curii.com
<p>See <a class="wiki-page" href="https://dev.arvados.org/projects/arvados/wiki/Efficient_live_access_to_container_logs">Efficient live access to container logs</a></p> Arvados - Feature #19196 (New): Allow API select parameter to add/remove fields from the default sethttps://dev.arvados.org/issues/191962022-06-14T14:35:55ZTom Cleggtom@curii.com
<p>Currently, if the select parameter is used, the caller must specify the full fields they want to receive. Some use cases would be more convenient and easier to maintain if the caller could specify "defaults, plus field X, minus field Y".</p>
<p>Possible syntax:</p>
<pre><code>{"select":["-manifest_text","+writable_by"]}</code></pre>
<p>("send all default fields except manifest_text, and also send writable_by")</p> Arvados - Bug #18334 (New): Accept release info changes in docker recipeshttps://dev.arvados.org/issues/183342021-11-04T15:02:18ZTom Cleggtom@curii.com
<p>In some circumstances, "apt-get update" stops working due to existence of a future debian version.</p>
<p>This can break cmd/arvados-package tests.</p>
<pre>
$ docker run --rm -it arvados-package-deps-debian:10 bash
root@7d1560822db7:/# apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Reading package lists... Done
E: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.11'
E: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
</pre>
<p>Proposed fix: "apt-get --allow-releaseinfo-change update" in scripts.</p> Arvados - Bug #18114 (New): [a-d-c] slow down retries when CreateInstance returns non-quota/non-t...https://dev.arvados.org/issues/181142021-09-07T18:04:45ZTom Cleggtom@curii.com
<p>If we get an error from the cloud provider when trying to create an instance, but the error isn't recognized as a quota or rate-limiting error, we retry very quickly, which would be unhelpful for an error like "invalid instance type". We should consider other options, like a per-instance-type quiet period for unrecognized errors (which would probably be a better way to respond to InsufficientInstanceCapacity as well).</p> Tapestry - Bug #6412 (New): Admin cannot use UserFilesController#reprocess unless admin account i...https://dev.arvados.org/issues/64122015-06-24T20:09:53ZTom Cleggtom@curii.comArvados - Task #5861 (New): Allow editing pipeline parameters in workbenchhttps://dev.arvados.org/issues/58612015-04-30T15:39:10ZTom Cleggtom@curii.comGET-Evidence - Feature #714 (New): Add spam-control measureshttps://dev.arvados.org/issues/7142011-09-22T18:59:34ZTom Cleggtom@curii.com
Approaches to minimize spam:
<ul>
<li>Max new openids per IP address per day</li>
<li>Max edits per day per openid</li>
<li>Max edits per day by "new users" </li>
<li>Notify admins and stop accepting <em>any</em> new edits when certain rate limits are reached ("under attack")</li>
<li>Put new OpenIDs in quarantine until a "known good" user has seen their edits</li>
<li>Mark quarantined edits/content</li>
</ul>
Approaches to make it easier to clean up spam:
<ul>
<li>Revert page to earlier version</li>
<li>Mark edit as spam (automatically revert the edit if it's the latest one; otherwise force flagging user to revert it, or do a "git rebase"-like process)</li>
<li>Mark OpenID as spam (mark/flag all edits by that OpenID; revert automatically if possible; otherwise flag the edited content so other readers know it's spam)</li>
<li>Show all edits by given OpenID</li>
<li>Show all edits by quarantined OpenIDs</li>
</ul>
More things to consider:
<ul>
<li>Compromised "known good" OpenID</li>
</ul> GET-Evidence - Feature #696 (New): Add Google Search APIhttps://dev.arvados.org/issues/6962011-09-15T13:53:33ZTom Cleggtom@curii.com
<p>Yahoo BOSS no longer offers any free searches. Pricing is currently $0.80 per 1000 queries.</p>
<p>Google pricing is currently $5.00 per 1000 queries but the first 100 queries per day are free. This model is more development/testing-friendly, although considerably more expensive for bootstrapping and keeping the production site up-to-date (36500 queries per year => ~3.5 years to look up every variant in current GET-Evidence DB).</p>
<p>Requirements:</p>
<ul>
<li>Implement Google Custom Search API<sup><a href="#fn1">1</a></sup>, store result xml and # hits locally (similar to what <code>import_yahoo_search.php</code> does now)</li>
<li>Support "queue variants that haven't been looked up with any search engine" and "queue variants that haven't been looked up in Google" modes</li>
<li>Show Yahoo and/or Google search results in the "web search results" section, whichever is available.</li>
<li>Allow installer/admin to limit periodic job to X queries per script invocation</li>
</ul>
<p id="fn1" class="footnote"><sup>1</sup> <a class="external" href="http://code.google.com/apis/customsearch/v1/overview.html">http://code.google.com/apis/customsearch/v1/overview.html</a></p> GET-Evidence - Feature #488 (New): Allow non-nsSNP variantshttps://dev.arvados.org/issues/4882010-05-08T16:02:23ZTom Cleggtom@curii.com
Need to stop restricting database to nsSNP variants.
<ul>
<li>Support lookup/create by dbSNP id</li>
<li>Support lookup/create by {chromosome, position, ref_seq, variant_seq}</li>
</ul>
Considerations:
<ul>
<li>Is "strand" helpful? Or should "chr1:1234:-:A:C" just be described as "chr1:1234:T:G"?</li>
<li>How to deal with different references?
<ul>
<li>Include a reference identifier in database key -- "hg18:chr:pos:refseq:varseq" </li>
<li>Extend database schema to include multiple identifiers per locus, so "hg18:chr1:1234" and "hg19:chr1:1235" can map to the same variant ID</li>
</ul>
</li>
<li>Ensure only one page per variant. E.g., if rs1234 maps to chr1:1234:T:G and causes ABCD-A1T, then there should only be one GET-Evidence page regardless of which order those identifiers are looked up / used to create a page.</li>
</ul> GET-Evidence - Feature #482 (New): add “affects self (hom or dominant)” checkbox (vs “affects off...https://dev.arvados.org/issues/4822010-05-07T18:51:38ZTom Cleggtom@curii.com
<p>add “affects self (hom or dominant)” checkbox (vs “affects offspring”) on result page</p> GET-Evidence - Feature #476 (New): allow “lab members” group to upload/view publication PDFshttps://dev.arvados.org/issues/4762010-05-07T17:31:32ZTom Cleggtom@curii.com
<p>allow “lab members” group to upload/view publication PDFs</p>