Arvados

Story

Resolved

Closed

Sprint Impediments

5623 5623 Tom Clegg Tom Clegg 3 Crunch 52 Crunch [Crunch] Specify Crunch2 features and APIs Resolved Resolved 3 1.0 1.0 1.0 5623 36 3 96 6 1.0	5628 5628 Meeting about crunch strategy Peter Amstutz 47 0 5623 3 36 -c-a 5 5711 5711 Requirements / API for Crunch v2 Tom Clegg 3 0 5623 3 36 -c-a 5 6104 6104 Review Peter Amstutz 47 0 5623 3 36 -c-a 5
5790 5790 Brett Smith Brett Smith 79 SDKs 54 SDKs [SDKs] arv-copy does not select the correct Docker image to copy Resolved Resolved 3 1.0 1.0 1.0 5790 36 79 96 1 1.0	6103 6103 Review 5790-copy-most-recent-docker-image-wip Brett Smith 79 0 5790 3 36 -c-a 5 6107 6107 Fix Docker image search/list Brett Smith 79 0 5790 3 36 -c-a 5
6091 6091 Radhika Chippada Radhika Chippada 72 Support users and triage their bugs (second half of sprint) Resolved Resolved 3 3.0 3.0 3.0 6091 36 72 96 6 3.0
6092 6092 Radhika Chippada Radhika Chippada 72 [TBD] Improve the performance of the worst-performing collections component Resolved Resolved 3 2.0 2.0 2.0 6092 36 72 96 6 2.0
6203 6203 Radhika Chippada Radhika Chippada 72 Performance 70 Performance [API] [Performance] Optimize time spent on the API server side during a large collection creation. Resolved Resolved 3 6203 36 72 96 6	6113 6113 Review branch: 6203-collection-perf-api Radhika Chippada 72 0 6203 3 36 -c-a 5
6088 6088 Peter Amstutz Peter Amstutz 47 Contribute to CWL specification Resolved Resolved 3 2.0 2.0 2.0 6088 36 47 96 6 2.0	6189 6189 Fix command line input binding ordering Peter Amstutz 47 0 6088 3 36 -c-a 5 6100 6100 Improve draft 2 spec docs Peter Amstutz 47 0 6088 3 36 -c-a 5 6099 6099 Finish fixing tests for draft2-updates Peter Amstutz 47 0 6088 3 36 -c-a 5 6190 6190 Improve schema Peter Amstutz 47 0 6088 3 36 -c-a 5 6097 6097 Implement "id" loading Peter Amstutz 47 0 6088 3 36 -c-a 5
6152 6152 Brett Smith Brett Smith 79 Documentation 58 Documentation [Docs] Eliminate `arv` dependency in compute node installer Resolved Resolved 3 0.5 0.5 0.5 6152 36 79 96 1 0.5	6256 6256 Review 6152-compute-node-no-arv-wip Ward Vandewege 1 0 6152 3 36 -c-a 5
6187 6187 Peter Amstutz Peter Amstutz 47 Crunch 52 Crunch [Crunch] Flag to disable job reuse with GATK Queue plugin Resolved Resolved 3 6187 36 47 96 2		6242 6242 Test to make sure it works Bryan Cosca 189 0 6187 5 36 -c-a 5
6090 6090 Peter Amstutz Peter Amstutz 47 Support users and triage their bugs (first half of sprint) Resolved Resolved 3 3.0 3.0 3.0 6090 36 47 96 6 3.0	6102 6102 IRC support Peter Amstutz 47 0 6090 3 36 -c-a 5
6094 6094 Tom Clegg Tom Clegg 3 Crunch 52 Crunch [Crunch] Install script exited 141 Resolved Resolved 3 1.0 1.0 1.0 6094 36 3 96 1 1.0	6111 6111 Review 6094-install-script-sigpipe Tom Clegg 3 0 6094 3 36 -c-a 5 6144 6144 Review 6094-install-script-sigpipe @ e010ab7 Tom Clegg 3 0 6094 3 36 -c-a 5
6074 6074 Tom Clegg Tom Clegg 3 API 57 API [API] collections are being returned too slowly + NoMemoryError Resolved Resolved 3 1.0 1.0 1.0 6074 36 3 96 1 1.0	6123 6123 Determine why still NoMemoryError even with API memory-limiting feature Tom Clegg 3 0 6074 3 36 -c-a 5 6223 6223 Add index to fix slow query Data manager generates queries like this (log from qr1hi): <pre> 2015-06-02 17:09:06 UTC LOG: duration: 12715.661 ms statement: SELECT collections."manifest_text", collections."owner_uuid", collections."uuid", collections."modified_at" FROM "collections" WHERE (expires_at IS NULL or expires_at > CURRENT_TIMESTAMP) AND ((collections.modified_at >= '2014-11-06 16:12:59.000000')) ORDER BY collections.modified_at asc, collections.modified_at desc, collections.uuid LIMIT 50 OFFSET 0 2015-06-02 17:10:08 UTC LOG: duration: 28088.273 ms statement: SELECT collections."manifest_text", collections."owner_uuid", collections."uuid", collections."modified_at", (length(manifest_text)) as read_length FROM "collections" WHERE (expires_at IS NULL or expires_at > CURRENT_TIMESTAMP) AND ((collections.modified_at >= '2014-11-06 23:46:04.000000')) ORDER BY "collections"."id" ASC LIMIT 1000 OFFSET 0 2015-06-02 17:10:17 UTC LOG: duration: 4362.355 ms statement: SELECT collections."manifest_text", collections."owner_uuid", collections."uuid", collections."modified_at" FROM "collections" WHERE (expires_at IS NULL or expires_at > CURRENT_TIMESTAMP) AND ((collections.modified_at >= '2014-11-06 23:46:04.000000')) ORDER BY collections.modified_at asc, collections.modified_at desc, collections.uuid LIMIT 50 OFFSET 0 2015-06-02 17:10:51 UTC LOG: duration: 22759.479 ms statement: SELECT collections."manifest_text", collections."owner_uuid", collections."uuid", collections."modified_at", (length(manifest_text)) as read_length FROM "collections" WHERE (expires_at IS NULL or expires_at > CURRENT_TIMESTAMP) AND ((collections.modified_at >= '2014-11-07 20:08:25.000000')) ORDER BY "collections"."id" ASC LIMIT 1000 OFFSET 0 2015-06-02 17:11:21 UTC LOG: duration: 26131.538 ms statement: SELECT collections."manifest_text", collections."owner_uuid", collections."uuid", collections."modified_at", (length(manifest_text)) as read_length FROM "collections" WHERE (expires_at IS NULL or expires_at > CURRENT_TIMESTAMP) AND ((collections.modified_at >= '2014-11-07 22:52:48.000000')) ORDER BY "collections"."id" ASC LIMIT 1000 OFFSET 0 2015-06-02 17:11:51 UTC LOG: duration: 25701.136 ms statement: SELECT collections."manifest_text", collections."owner_uuid", collections."uuid", collections."modified_at", (length(manifest_text)) as read_length FROM "collections" WHERE (expires_at IS NULL or expires_at > CURRENT_TIMESTAMP) AND ((collections.modified_at >= '2014-11-08 07:17:22.000000')) ORDER BY "collections"."id" ASC LIMIT 1000 OFFSET 0 </pre> They are extremely slow. Explain analyze on 4xphq shows that we're missing an index on modified_at: <pre> QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------ Limit (cost=561.89..564.39 rows=1000 width=212) (actual time=5230.714..5233.244 rows=1000 loops=1) -> Sort (cost=561.89..568.55 rows=2664 width=212) (actual time=5230.707..5231.627 rows=1000 loops=1) Sort Key: id Sort Method: top-N heapsort Memory: 352kB -> Seq Scan on collections (cost=0.00..415.83 rows=2664 width=212) (actual time=0.018..5225.941 rows=3633 loops=1) Filter: ((modified_at >= '2014-11-08 21:04:18'::timestamp without time zone) AND ((expires_at IS NULL) OR (expires_at > now()))) Total runtime: 5234.403 ms (7 rows) </pre> Tom Clegg 3 0 6074 3 35 -c-a 5 6110 6110 Review 6074-collections-index @ 22c8b63 Tom Clegg 3 0 6074 3 36 -c-a 5
5930 5930 Nancy Ouyang Nancy Ouyang 235 Documentation 58 Documentation [Documentation] Misc. documentation fixes Resolved Resolved 3 5930 36 235 96 1		6052 6052 Review branch: 5930-smalldocfix Nancy Ouyang 235 0 5930 5 36 -c-a 5
6194 6194 Peter Amstutz Peter Amstutz 47 SDKs 54 SDKs [SDK] Support writing more than 64MiB at a time in Python SDK Resolved Resolved 3 6194 36 47 96 1	6210 6210 Review 6194-python-arvfile-large-write Tom Clegg 3 0 6194 3 36 -c-a 5
5548 5548 Nico César Nico César 288 [CWL/OPS] Jenkins runs cwltool tests Resolved Resolved 3 0.5 0.5 0.5 5548 36 288 96 6 0.5	5631 5631 create Jenkins pipeline Ward Vandewege 1 0 5548 3 36 -c-a 5 5629 5629 Write test script Peter Amstutz 47 0 5548 3 36 -c-a 5 6158 6158 Review 5548-test-cwltool in arvados-dev Peter Amstutz 47 0 5548 3 36 -c-a 5
6098 6098 Tom Clegg Tom Clegg 3 API 57 API [API] slow search? SQL query Resolved Resolved 3 1.0 1.0 1.0 6098 36 3 96 1 1.0	6112 6112 Review 6098-full-text-index 64cfaed Tom Clegg 3 0 6098 3 36 -c-a 5 6122 6122 Improve query performance (or prevent clients from doing it?) Tom Clegg 3 0 6098 3 36 -c-a 5	6106 6106 Add a performance test case that exposes the slow query Tom Clegg 3 0 6098 5 36 -c-a 5 6150 6150 Test correct index usage with "explain analyze" Tom Clegg 3 0 6098 5 36 -c-a 5
6141 6141 Peter Amstutz Peter Amstutz 47 [Docs] Don't hardcode https:// in links to workbench Resolved Resolved 3 6141 36 47 96 1	6137 6137 Fix documentation links in docker build Peter Amstutz 47 0 6141 3 36 -c-a 5 6155 6155 Review 6141-doc-workbench-links Peter Amstutz 47 0 6141 3 36 -c-a 5
6089 6089 Brett Smith Brett Smith 79 Write development update blog (first three days of sprint) Resolved Resolved 3 0.5 0.5 0.5 6089 36 79 96 6 0.5
6057 6057 Radhika Chippada Radhika Chippada 72 Workbench 56 Workbench [Workbench] Provide an attractive index of public projects Resolved Resolved 3 2.0 2.0 2.0 6057 36 72 96 6 2.0	6115 6115 Review branch: 6057-public-projects-page Tom Clegg 3 0 6057 3 36 -c-a 5
6138 6138 Peter Amstutz Peter Amstutz 47 [Docker] Docker install uses standalone SSO server Resolved Resolved 3 6138 36 47 96 6	6128 6128 Test Docker install Peter Amstutz 47 0 6138 3 36 -c-a 5 6133 6133 Review 6090-docker-use-local-sso Peter Amstutz 47 0 6138 3 36 -c-a 5
6095 6095 Brett Smith Brett Smith 79 SDKs 54 SDKs [SDKs] Make arv-copy preserve pipeline template component order Resolved Resolved 3 0.5 0.5 0.5 6095 36 79 96 1 0.5	6117 6117 Review 6095-arv-copy-preserve-object-order-wip Brett Smith 79 0 6095 3 36 -c-a 5