Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422019-12-04T16:34:42ZArvados
Redmine Arvados - Task #15913 (Resolved): Reviewhttps://dev.arvados.org/issues/159132019-12-04T16:34:42ZTom Morristfmorris@veritasgenetics.comArvados - Bug #15902 (Resolved): Very slow query for large workflowshttps://dev.arvados.org/issues/159022019-12-03T15:32:00ZTom Morristfmorris@veritasgenetics.com
<p>A customer reports that the query below takes 32 seconds for a ~3,000 container workflow:</p>
<pre><code>duration: 32570.970 ms execute &lt;unnamed&gt;: SELECT container_requests."uuid", container_requests."owner_uuid", container_requests."created_at", container_requests."modified_at", container_requests."modified_by_client_uuid", container_requests."modified_by_user_uuid", container_requests."name", container_requests."description", container_requests."properties", container_requests."state", container_requests."requesting_container_uuid", container_requests."container_uuid", container_requests."container_count_max", container_requests."runtime_constraints", container_requests."container_image", container_requests."environment", container_requests."cwd", container_requests."command", container_requests."output_path", container_requests."priority", container_requests."expires_at", container_requests."filters", container_requests."container_count", container_requests."use_existing", container_requests."scheduling_parameters", container_requests."output_uuid", container_requests."log_uuid", container_requests."output_name", container_requests."output_ttl" <br />FROM "container_requests" <br />WHERE (container_requests.owner_uuid NOT IN (SELECT target_uuid FROM materialized_permission_view WHERE trashed = 1) ) AND (container_requests.requesting_container_uuid in ('<a href="https://arvadosapi.com/zzzzz-dz642-r9n91uyk20xowyf">zzzzz-dz642-r9n91uyk20xowyf</a>',...[~3000 UUIDs omitted]<br />) ORDER BY container_requests.modified_at desc, container_requests.uuid LIMIT $1 OFFSET $2<br />< 2019-11-26 10:50:27.162 EST >DETAIL: parameters: $1 = '1000', $2 = '0'</code></pre>
<p>Here's the EXPLAIN ANALYZE for the original query:</p>
<pre><code>Limit (cost=2385.09..16840.05 rows=1000 width=791) (actual time=32385.658..32385.658 rows=0 loops=1)<br /> -> Index Scan using index_container_requests_on_modified_at_uuid on container_requests (cost=2385.09..1995593.21 rows=137891 width=791) (actual time=32385.656..32385.656 rows=0 loops=1)<br /> Filter: ((NOT (hashed SubPlan 1)) AND ((requesting_container_uuid)::text = ANY ('{<a href="https://arvadosapi.com/zzzzz-dz642-r9n91uyk20xowyf">zzzzz-dz642-r9n91uyk20xowyf</a>,...}'::text[])))<br /> Rows Removed by Filter: 376597<br /> SubPlan 1<br /> -> Index Only Scan using permission_target_trashed on materialized_permission_view (cost=0.42..2380.40 rows=1709 width=28) (actual time=0.050..0.351 rows=95 loops=1)<br /> Index Cond: (trashed = 1)<br /> Heap Fetches: 95<br /> Planning time: 29.160 ms<br /> Execution time: 32385.751 ms</code></pre>
<p>Deleting the index that the query planner incorrectly chose, index_container_requests_on_modified_at_uuid, dropped the query time from 30+ seconds to ~50 msec.</p>
<pre><code>Limit (cost=83535.57..83538.07 rows=1000 width=780) (actual time=54.419..54.419 rows=0 loops=1)<br /> -> Sort (cost=83535.57..83877.22 rows=136657 width=780) (actual time=54.417..54.417 rows=0 loops=1)<br /> Sort Key: container_requests.modified_at DESC, container_requests.uuid<br /> Sort Method: quicksort Memory: 25kB<br /> -> Index Scan using index_container_requests_on_requesting_container_uuid on container_requests (cost=635.83..76042.82 rows=136657 width=780) (actual time=54.323..54.323 rows=0 loops=1)</code></pre>
<p>Before this was done, a full VACUUM ANALYZE had no affect on performance, but dropping the ORDER BY clause from the query also dramatically improved the query times.</p>
<p>One thing that seems strange is that the query is ordering by not only <code>modified_at</code>, but also UUID which doesn't really seem to add any value.</p>
<p>We should see if we can adjust the indexes and/or query in a way which the query planner can optimize correctly.</p> Arvados - Idea #15885 (New): Integrate status dashboardhttps://dev.arvados.org/issues/158852019-11-27T15:46:44ZTom Morristfmorris@veritasgenetics.com
<p>There are a number of open source dashboards like :</p>
<p><a class="external" href="https://demo.cachethq.io/">https://demo.cachethq.io/</a><br /><a class="external" href="https://staytus.co/">https://staytus.co/</a></p>
<p>It's also possible that we could use Prometheus to fill this need.</p>
<p>It would be useful to integrate one of these to show the health of the Arvados system, its various services, and even perhaps user level (workflow) status.</p> Arvados - Idea #15883 (Duplicate): Support LDAP loginshttps://dev.arvados.org/issues/158832019-11-27T15:32:38ZTom Morristfmorris@veritasgenetics.com
<p>As a system administrator, I'd like to continue to be able to support LDAP logins without having to use the SSO server.</p>
<p>May want to do this through an authentication framework rather than directly.</p> Arvados - Idea #15880 (New): Remove hosted git servicehttps://dev.arvados.org/issues/158802019-11-27T15:12:07ZTom Morristfmorris@veritasgenetics.com
<p>With the retirement of Crunch 1, the built-in git hosting isn't really needed anymore.</p>
<ul>
<li>Remove pages from Workbench 2</li>
<li>Remove documentation</li>
<li>Remove from arvbox/installer</li>
<li>Remove arv-git-http</li>
<li>Remove any other references / packages</li>
</ul> Arvados - Idea #15879 (New): Out of the box routing (NGINX) standard as part of basic installhttps://dev.arvados.org/issues/158792019-11-27T15:01:55ZTom Morristfmorris@veritasgenetics.comArvados - Idea #15856 (Resolved): [WB2] Warn user when viewing a project / collection containing ...https://dev.arvados.org/issues/158562019-11-20T17:31:28ZTom Morristfmorris@veritasgenetics.com
<p>If a project name or collection name contains a slash (/) or is the string "." or ".." warn the user that this is incompatible with WebDAV clients.</p>
<p>UI affordance is a small red "!" icon with hover text containing the error message.</p> Arvados - Idea #15849 (Resolved): Migrate collection & project properties to new vocabulary formathttps://dev.arvados.org/issues/158492019-11-20T14:55:03ZTom Morristfmorris@veritasgenetics.com
<p>Customer databases which use literal text labels for property/value pairs need to be converted to the new ID format in the database.</p>
<p>Input to this will be a new format vocabulary file which matches the original vocabulary file used in creating the database.</p>
<p>This will be an example script which accesses the database through the API.</p> Arvados - Bug #15836 (Resolved): Escape / convert forward slashes in collection names accessed vi...https://dev.arvados.org/issues/158362019-11-18T21:00:15ZTom Morristfmorris@veritasgenetics.com
<p>We have a customer report that the Mountain Goat WebDAV client doesn't display collections which have slashes (/) in their names. That's not terribly surprising considering it's use as a path delimiter, but it's not great to not be able to access the collection at all. We should consider a configurable option to replace the slash with another character such as underscore (_) or hyphen (-).</p> Arvados Workbench 2 - Bug #15832 (New): [WB2] Add create data collection to context menu for proj...https://dev.arvados.org/issues/158322019-11-18T15:12:59ZTom Morristfmorris@veritasgenetics.comArvados - Idea #15822 (Rejected): [WB2] Update to React 16.9https://dev.arvados.org/issues/158222019-11-13T14:53:44ZTom Morristfmorris@veritasgenetics.comArvados - Task #15811 (In Progress): Review 15573-deprecate-atat-filterhttps://dev.arvados.org/issues/158112019-11-06T17:11:19ZTom Morristfmorris@veritasgenetics.comArvados - Idea #15800 (Resolved): Update 1.4-dev branch with cherry picked bugs for 1.4.2https://dev.arvados.org/issues/158002019-11-06T16:31:02ZTom Morristfmorris@veritasgenetics.com
<p>Tickets here: <a class="external" href="https://dev.arvados.org/rb/release/27">https://dev.arvados.org/rb/release/27</a></p> Arvados - Bug #15183 (New): [Workbench2] No errors reported when file is missing a blockhttps://dev.arvados.org/issues/151832019-05-01T21:39:26ZTom Morristfmorris@veritasgenetics.com
<p>When looking at collection <a href="https://arvadosapi.com/su92l-4zz18-44qy35pg0lxshkf">su92l-4zz18-44qy35pg0lxshkf</a>/cwl.input.yml the file is cwl.output.json is listed as containing 361 bytes, but when downloaded from Workbench, it produces a zero byte file, but without giving any error or warning.</p>
<pre>
$ md5sum keep/by_id/su92l-4zz18-44qy35pg0lxshkf/*
a890894f5a748d2a93440fd046c73069 keep/by_id/su92l-4zz18-44qy35pg0lxshkf/cwl.input.yml
md5sum: keep/by_id/su92l-4zz18-44qy35pg0lxshkf/cwl.output.json: Input/output error
d41d8cd98f00b204e9800998ecf8427e keep/by_id/su92l-4zz18-44qy35pg0lxshkf/hu27FD1F_var-GS000034212-ASM.bed
b02868b9ac1264bd1f04b61d04518f57 keep/by_id/su92l-4zz18-44qy35pg0lxshkf/workflow.cwl
</pre> Arvados - Idea #14382 (Resolved): Review Java SDK PRhttps://dev.arvados.org/issues/143822018-10-19T13:29:03ZTom Morristfmorris@veritasgenetics.com
<p>Please do a design and code review of the PR for the new Java SDK in the arvados-sdk-java repo</p>