Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422019-12-03T15:32:00ZArvados
Redmine Arvados - Bug #15902 (Resolved): Very slow query for large workflowshttps://dev.arvados.org/issues/159022019-12-03T15:32:00ZTom Morristfmorris@veritasgenetics.com
<p>A customer reports that the query below takes 32 seconds for a ~3,000 container workflow:</p>
<pre><code>duration: 32570.970 ms execute &lt;unnamed&gt;: SELECT container_requests."uuid", container_requests."owner_uuid", container_requests."created_at", container_requests."modified_at", container_requests."modified_by_client_uuid", container_requests."modified_by_user_uuid", container_requests."name", container_requests."description", container_requests."properties", container_requests."state", container_requests."requesting_container_uuid", container_requests."container_uuid", container_requests."container_count_max", container_requests."runtime_constraints", container_requests."container_image", container_requests."environment", container_requests."cwd", container_requests."command", container_requests."output_path", container_requests."priority", container_requests."expires_at", container_requests."filters", container_requests."container_count", container_requests."use_existing", container_requests."scheduling_parameters", container_requests."output_uuid", container_requests."log_uuid", container_requests."output_name", container_requests."output_ttl" <br />FROM "container_requests" <br />WHERE (container_requests.owner_uuid NOT IN (SELECT target_uuid FROM materialized_permission_view WHERE trashed = 1) ) AND (container_requests.requesting_container_uuid in ('<a href="https://arvadosapi.com/zzzzz-dz642-r9n91uyk20xowyf">zzzzz-dz642-r9n91uyk20xowyf</a>',...[~3000 UUIDs omitted]<br />) ORDER BY container_requests.modified_at desc, container_requests.uuid LIMIT $1 OFFSET $2<br />< 2019-11-26 10:50:27.162 EST >DETAIL: parameters: $1 = '1000', $2 = '0'</code></pre>
<p>Here's the EXPLAIN ANALYZE for the original query:</p>
<pre><code>Limit (cost=2385.09..16840.05 rows=1000 width=791) (actual time=32385.658..32385.658 rows=0 loops=1)<br /> -> Index Scan using index_container_requests_on_modified_at_uuid on container_requests (cost=2385.09..1995593.21 rows=137891 width=791) (actual time=32385.656..32385.656 rows=0 loops=1)<br /> Filter: ((NOT (hashed SubPlan 1)) AND ((requesting_container_uuid)::text = ANY ('{<a href="https://arvadosapi.com/zzzzz-dz642-r9n91uyk20xowyf">zzzzz-dz642-r9n91uyk20xowyf</a>,...}'::text[])))<br /> Rows Removed by Filter: 376597<br /> SubPlan 1<br /> -> Index Only Scan using permission_target_trashed on materialized_permission_view (cost=0.42..2380.40 rows=1709 width=28) (actual time=0.050..0.351 rows=95 loops=1)<br /> Index Cond: (trashed = 1)<br /> Heap Fetches: 95<br /> Planning time: 29.160 ms<br /> Execution time: 32385.751 ms</code></pre>
<p>Deleting the index that the query planner incorrectly chose, index_container_requests_on_modified_at_uuid, dropped the query time from 30+ seconds to ~50 msec.</p>
<pre><code>Limit (cost=83535.57..83538.07 rows=1000 width=780) (actual time=54.419..54.419 rows=0 loops=1)<br /> -> Sort (cost=83535.57..83877.22 rows=136657 width=780) (actual time=54.417..54.417 rows=0 loops=1)<br /> Sort Key: container_requests.modified_at DESC, container_requests.uuid<br /> Sort Method: quicksort Memory: 25kB<br /> -> Index Scan using index_container_requests_on_requesting_container_uuid on container_requests (cost=635.83..76042.82 rows=136657 width=780) (actual time=54.323..54.323 rows=0 loops=1)</code></pre>
<p>Before this was done, a full VACUUM ANALYZE had no affect on performance, but dropping the ORDER BY clause from the query also dramatically improved the query times.</p>
<p>One thing that seems strange is that the query is ordering by not only <code>modified_at</code>, but also UUID which doesn't really seem to add any value.</p>
<p>We should see if we can adjust the indexes and/or query in a way which the query planner can optimize correctly.</p> Arvados - Bug #15836 (Resolved): Escape / convert forward slashes in collection names accessed vi...https://dev.arvados.org/issues/158362019-11-18T21:00:15ZTom Morristfmorris@veritasgenetics.com
<p>We have a customer report that the Mountain Goat WebDAV client doesn't display collections which have slashes (/) in their names. That's not terribly surprising considering it's use as a path delimiter, but it's not great to not be able to access the collection at all. We should consider a configurable option to replace the slash with another character such as underscore (_) or hyphen (-).</p> Arvados Workbench 2 - Bug #15832 (New): [WB2] Add create data collection to context menu for proj...https://dev.arvados.org/issues/158322019-11-18T15:12:59ZTom Morristfmorris@veritasgenetics.comArvados Workbench 2 - Bug #15767 (Duplicate): "Load more" button on search results page always showshttps://dev.arvados.org/issues/157672019-10-25T20:09:53ZTom Morristfmorris@veritasgenetics.com
<p>The "Load More" button should only be displayed on the search results page if there are (potentially) more results available. It should never be displayed when less than the requested number of results was returned. In the edge case where the number of results exactly matches the page size, it's acceptable to display the button even if it'll not return any additional results if the performance cost of getting the exact count up front is too high (likely).</p> Arvados Workbench 2 - Bug #15766 (Resolved): [Workbench 2] Property tag/value text is not copyablehttps://dev.arvados.org/issues/157662019-10-25T20:00:41ZTom Morristfmorris@veritasgenetics.com
<p>After property tag/value pairs are added to a collection, it's impossible to select the text of either the tag or value.</p>
<p>The element has focusable=false (and also aria-hidden=true which might not be correct either)</p> Arvados Workbench 2 - Bug #15765 (Rejected): [Workbench 2] Shared projects show up in the wrong p...https://dev.arvados.org/issues/157652019-10-25T19:52:52ZTom Morristfmorris@veritasgenetics.com
<p>A project which I created and then shared shows up under the "Shared with me" portion of the navigation tree instead of the "Projects" subtree. Things that I own/created should show up in the Projects portion of the hierarchy.</p>
<p>The breadcrumbs are also wrong (but match the position in the tree).</p> Arvados - Bug #15764 (Closed): Right clicking on a node in the navigation tree should select ithttps://dev.arvados.org/issues/157642019-10-25T19:50:23ZTom Morristfmorris@veritasgenetics.com
<p>If nothing is selected in the navigation tree and and right click on a subproject and select "View Details", the system currently brings up a blanks Details pane. This action should select the node which is under the right click action.</p> Arvados Workbench 2 - Bug #15763 (Resolved): Owner field in Details pane should contain name and ...https://dev.arvados.org/issues/157632019-10-25T19:47:13ZTom Morristfmorris@veritasgenetics.com
<p>The current link is of the form <a class="external" href="https://workbench2.e51c5.arvadosapi.com/projects/e51c5-tpzed-zbu1spr7ibvy9t5">https://workbench2.e51c5.arvadosapi.com/projects/e51c5-tpzed-zbu1spr7ibvy9t5</a> which is invalid since it's a User, not a Project.</p>
<p>Changing it to <a class="external" href="https://workbench2.e51c5.arvadosapi.com/users/e51c5-tpzed-zbu1spr7ibvy9t5">https://workbench2.e51c5.arvadosapi.com/users/e51c5-tpzed-zbu1spr7ibvy9t5</a> shows all Users instead of just the indicated user.</p>
<p>The text of the field should be the name of the user/project and it should link to a page which shows the details for that single user (or link to the parent project if the "owner" is a project).</p> Arvados - Bug #15695 (Closed): [a-d-c] Long delay before cloud dispatcher starts jobs on playgroundhttps://dev.arvados.org/issues/156952019-10-07T18:00:02ZTom Morristfmorris@veritasgenetics.com
<p>This workflow wants 100 parallel jobs running the same code over different date.</p>
<p>There are two separate runs shown in the Prometheus graph below:<br /><a class="external" href="https://prometheus.curoverse.com/consoles/qr1hi/index.html#pctc%7B%22duration%22%3A7200%2C%22endTime%22%3A1570223940%7D">https://prometheus.curoverse.com/consoles/qr1hi/index.html#pctc%7B%22duration%22%3A7200%2C%22endTime%22%3A1570223940%7D</a></p>
<p>The timeline is (all times UTC):<br />19:24 First run submitted with requirements for 100 x 4 core nodes - <a class="external" href="https://workbench.qr1hi.arvadosapi.com/container_requests/qr1hi-xvhdp-ctw1t6m8z718emc">https://workbench.qr1hi.arvadosapi.com/container_requests/qr1hi-xvhdp-ctw1t6m8z718emc</a><br />19:41 17 nodes with 4 cores each started<br />19:57 Workflow canceled<br />19:57 75 nodes idle<br />19:58 Second run submitted with edited run time requirements for 100 x 2 core nodes - <a class="external" href="https://workbench.qr1hi.arvadosapi.com/container_requests/qr1hi-xvhdp-2ry6g3l031wlygu">https://workbench.qr1hi.arvadosapi.com/container_requests/qr1hi-xvhdp-2ry6g3l031wlygu</a><br />20:00 71 nodes idle from 1st cancelled workflow<br />20:03 1 node busy, 0 nodes idle<br />20:26 1st node child container started<br />20:31 2nd node for child container start<br />20:36 39 nodes booting for child containers<br />20:40 another 22 nodes start booting<br />20:46 final 7 nodes start booting<br />20:55 All 100 containers finally running</p> Arvados Workbench 2 - Bug #15609 (Resolved): Clicking outside popup dialog dismisses ithttps://dev.arvados.org/issues/156092019-09-09T14:38:17ZTom Morristfmorris@veritasgenetics.com
<p>When creating a new project or doing any other action which uses a popup dialog to collection information, clicking outside of the popup modal (on the greyed out background), dismisses the popup, losing all information which has been collected.</p>
<p>The popup should only be dismissed when the Cancel button is clicked explicitly (and even then, it should probably ask for confirmation if some of the fields have already been filled in).</p> Arvados - Bug #15554 (Resolved): [crunchstat-summary] Fix live log reporting for crunch 2 v1.4+https://dev.arvados.org/issues/155542019-08-14T18:55:52ZTom Morristfmorris@veritasgenetics.com
<p>Currently crunchstat-summary assumes that if the log collection is available, that the container is no longer running, but this is no longer true, so it should be fixed to instead explicitly check the container state and use the live crunchstat data from the log table if the container is still running.</p> Arvados - Bug #15537 (New): Clarify that Arvados CLI tools are Linux onlyhttps://dev.arvados.org/issues/155372019-08-12T15:30:22ZTom Morristfmorris@veritasgenetics.com
<p>The installation docs <a class="external" href="https://doc.arvados.org/sdk/cli/install.html">https://doc.arvados.org/sdk/cli/install.html</a> should mention that a supported Linux distro is required so that people don't attempt to install via gem and pip on non-Linux systems.</p> Arvados Workbench 2 - Bug #15525 (New): [WB2] Copying a collection with files in subdirectories f...https://dev.arvados.org/issues/155252019-08-06T16:33:25ZTom Morristfmorris@veritasgenetics.com
<p>From a customer:</p>
<blockquote>
<p>In workbench 2, when files generated by the workflow are located in a folder, I am unable of copying them to another collection (the attached error massage appears in the bottom right end corner, and yes the files are incorrect :) ). In contrast, this functionality is working fine when files are not embedded in a folder or when using workbench 1.</p>
</blockquote> Arvados Workbench 2 - Bug #15524 (Resolved): [WB2] No human readable name for anonymous sharinghttps://dev.arvados.org/issues/155242019-08-06T16:29:53ZTom Morristfmorris@veritasgenetics.com
<p>If you make a project public and shared with anonymous users, the resulting permission link is missing a human readable name.</p>
<p>Secondary bug reported: adding permission to a group seems to add permissions to each individual user instead (!).</p> Arvados - Bug #15486 (Resolved): [crunchstat-summary] error fetching container log collection for...https://dev.arvados.org/issues/154862019-07-18T14:30:49ZTom Morristfmorris@veritasgenetics.com
<p>When attempting to analyze Arvados v1.4 container requests, crunchstat-summary attempts to load the collection specified by the log PDH in the Container record, but that collection never exists because it is deleted immediately. Also there is no longer a `log` field for the container_request record.</p>
For Arvados v1.4 and up, crunchstat-summary needs to be modified so that:
<ul>
<li>for Containers, it first looks up the associated Container Request and uses the log from there</li>
<li>for Container Requests, remove the current logic which looks up the associated Container log and instead load the CR log by UUID from `log_uuid` (there is no `log` field)</li>
</ul>