Activity

From 01/23/2017 to 02/21/2017

02/20/2017

10:04 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
7995-keep-balance-docs @ commit:14304c7af0b0dd7bc9345b6c5aeb61a3bdc1d3b0 Tom Clegg
10:03 pm Task #11134 (Resolved): Explain limitations re shared volumes and service_type
Tom Clegg
08:50 pm Task #11134 (In Progress): Explain limitations re shared volumes and service_type
Tom Clegg
10:03 pm Revision 14304c7a (arvados): 7995: Add "dry run" note.
Tom Clegg
09:32 pm Task #11141 (New): Review 6520-nodemanager-docs
Peter Amstutz
09:18 pm Revision 8b75947e (arvados): Merge branch '6520-pending-reason' refs #6520
Peter Amstutz
09:17 pm Revision 4bd54c4c (arvados): 6520: Add ReqNodeNotAvail to list of reasons (along with "Resources"...
Peter Amstutz
09:13 pm Revision 53887579 (arvados): 6520: Add information about setting up SLURM to crunchv2 documentation.
Peter Amstutz
08:49 pm Task #11140 (Resolved): Update tests
Tom Clegg
08:34 pm Task #11140 (In Progress): Update tests
Tom Clegg
08:34 pm Task #11140 (Resolved): Update tests
Tom Clegg
08:46 pm Bug #11097: [API] Reuse containers even when multiple matching containers exist with differing ou...
commit:4f0e07d462b7860bb10686c27fac16970220377f with updated test case. Tom Clegg
08:45 pm Revision 4f0e07d4 (arvados): 11097: Update test to match new behavior.
Tom Clegg
08:34 pm Revision 4cd89bd1 (arvados): 11097: Merge branch 'master' into 11097-reuse-impure
Tom Clegg
07:47 pm Bug #11139: Different node size calculations.
This can be fixed entirely through configuration by overriding the "ram" field for each size. However, we need deter... Peter Amstutz
07:07 pm Bug #11139: Different node size calculations.
GCP... Peter Amstutz
06:20 pm Bug #11139 (New): Different node size calculations.
There's a discrepancy between the RAM of a VM used to choose what size node to boot for a job, and the actual amount ... Peter Amstutz
04:45 pm Task #11133 (Resolved): Set -enable-trash in keepstore docs
Tom Clegg
03:12 pm Task #11133 (In Progress): Set -enable-trash in keepstore docs
Tom Clegg
03:41 pm Revision 84bd6883 (arvados): 7995: Add keep-balance to install guide.
Tom Clegg
12:10 pm Bug #11138 (New): Protection against running new style docker 1.10+ image on old docker host
I upgraded docker to 1.13.1 over the weekend (or so I thought). Since then, I've been struggling to get the system wo... Joshua Randall

02/19/2017

12:06 pm Bug #11137 (New): apt-get upgrade arvados-api-server on Ubuntu 12.04 did not update crunch-job
My jobs were failing since I upgraded docker to 1.13.1 and installed v1.10+ versions of the docker images in keep.
...
Joshua Randall

02/17/2017

10:32 pm Revision d069de03 (arvados): Merge branch '11127-delete-trash-with-links'
refs #11127 Tom Clegg
10:30 pm Bug #11136 (New): Nginx config should speak JSON when returning its own response for unproxyable ...
Tom Clegg
10:27 pm Story #7995 (In Progress): [Documentation] Document Keep Balance setup in the Install Guide
Tom Clegg
10:26 pm Task #11132 (Resolved): Add page to install guide
Tom Clegg
10:23 pm Task #11132 (In Progress): Add page to install guide
Tom Clegg
10:23 pm Task #11132 (Resolved): Add page to install guide
Tom Clegg
10:26 pm Task #11134 (Resolved): Explain limitations re shared volumes and service_type
Tom Clegg
10:26 pm Task #11133 (Resolved): Set -enable-trash in keepstore docs
Tom Clegg
10:25 pm Bug #11121 (Resolved): [Crunch] Set output collection owner_uuid to match job
Applied in changeset arvados|commit:d0ba6eaac1387ad817de1c2df2d7f4f00800aaa5. Lucas Di Pentima
09:45 pm Bug #11121: [Crunch] Set output collection owner_uuid to match job
LGTM, thanks! Tom Clegg
10:23 pm Revision ead2ecb2 (arvados): 7995: Add keep-balance to install guide.
Tom Clegg
10:20 pm Task #11125 (Resolved): Review 11121-crunch-output-collection-owner
Lucas Di Pentima
10:18 pm Revision d0ba6eaa (arvados): Merge branch '11121-crunch-output-collection-owner'
Closes #11121 Lucas Di Pentima
10:17 pm Revision b1fffbeb (arvados): 11121: Merge branch 'master' into 11121-crunch-output-collection-owner
Lucas Di Pentima
09:17 pm Revision 484370cb (arvados): 11127: Delete dependent links too when emptying trash.
Tom Clegg
08:28 pm Feature #10218: [Crunch2] Gather and record cloud/physical node information for each container
Things to consider logging:
$ uname -a
$ cat /proc/cpuinfo
$ cat /proc/meminfo
Peter Amstutz
08:15 pm Revision f29e958f (arvados): 10979: TrackContainer implementation and test
Radhika Chippada
07:49 pm Revision 85887cd7 (arvados): 11127: Do not crash server if SweepTrashedCollections thread has an ...
Tom Clegg
12:30 pm Bug #9998: Improve performance of API server when keep-balance retrieves collections
Using the arv cli, I timed the performance of collection retrieval on our system using a query very similar to what k... Joshua Randall

02/16/2017

11:22 pm Revision f6071ef7 (arvados): Merge branch 'master' into 10979-cancelled-job-nodes
Radhika Chippada
10:20 pm Revision 6f93ea89 (arvados): 6520: Node manager docs WIP
Peter Amstutz
09:46 pm Task #11031 (Resolved): Review 6520-nodemanager-crunchv2
Peter Amstutz
09:08 pm Task #11031 (In Progress): Review 6520-nodemanager-crunchv2
Tom Clegg
09:46 pm Task #11106 (Resolved): Review 6520-skip-compute0
Peter Amstutz
09:08 pm Task #11106 (In Progress): Review 6520-skip-compute0
Tom Clegg
09:44 pm Revision 3fa4a2b6 (arvados): Merge branch '6520-nodemanager-crunchv2' refs #6520
Peter Amstutz
09:43 pm Revision ed1acd6f (arvados): 6520: Crunch v1 recognizes 'mix' and 'drng' as busy states for
consistency (won't affect node manager behavior going forward, however.) Peter Amstutz
09:35 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Tom Clegg wrote:
> in ArvadosNodeListMonitorActor it looks like "alloc*" will be reported as "down" -- instead of "a...
Peter Amstutz
09:18 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
... Peter Amstutz
09:07 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
in ArvadosNodeListMonitorActor it looks like "alloc*" will be reported as "down" -- instead of "alloc", which is how ... Tom Clegg
09:34 pm Revision 24b1aecb (arvados): Merge branch '6520-skip-compute0' refs #6520
Peter Amstutz
09:31 pm Revision d5ddfd9d (arvados): 6520: Expand recognized 'busy' states.
Peter Amstutz
09:17 pm Task #11125 (Resolved): Review 11121-crunch-output-collection-owner
Lucas Di Pentima
09:17 pm Bug #11121: [Crunch] Set output collection owner_uuid to match job
Test added at commit:3a6966e
Test run: https://ci.curoverse.com/job/developer-run-tests/170/
Lucas Di Pentima
05:03 pm Bug #11121: [Crunch] Set output collection owner_uuid to match job
Branch @11121-crunch-output-collection-owner@ - Update at: commit:0403a0c
Running tests: https://ci.curoverse.com/jo...
Lucas Di Pentima
04:04 pm Bug #11121 (In Progress): [Crunch] Set output collection owner_uuid to match job
Lucas Di Pentima
03:50 pm Bug #11121 (Resolved): [Crunch] Set output collection owner_uuid to match job
Currently, if a job is started with owner_uuid set to a shared project, the job and its log are shared, but its outpu... Tom Clegg
09:11 pm Revision 3a6966e5 (arvados): 11121: Added test.
Lucas Di Pentima
09:07 pm Revision f6bdb550 (arvados): Merge branch 'master' into 10979-cancelled-job-nodes
Radhika Chippada
08:16 pm Task #11107 (Resolved): Review 9277-trash-container-outputs
Radhika Chippada
07:45 pm Task #11107 (In Progress): Review 9277-trash-container-outputs
Radhika Chippada
08:15 pm Story #9277 (Resolved): [Crunch2] System-owned container outputs should be garbage-collected
Applied in changeset arvados|commit:21598295f38998d8028aaa117f192de6b5758808. Peter Amstutz
08:02 pm Story #9277: [Crunch2] System-owned container outputs should be garbage-collected
LGTM Radhika Chippada
07:16 pm Story #9277 (In Progress): [Crunch2] System-owned container outputs should be garbage-collected
Peter Amstutz
06:15 pm Story #9277: [Crunch2] System-owned container outputs should be garbage-collected
Even better than @trash_at=now@, it looks like we can just set @is_trashed: true@
9277-trash-container-outputs @ c...
Peter Amstutz
08:13 pm Revision 21598295 (arvados): Merge branch '9277-trash-container-outputs' closes #9277
Peter Amstutz
08:10 pm Revision 425f9a75 (arvados): 6520: Explicitly mark node as 'down' before shutting down.
Peter Amstutz
08:04 pm Revision be33e85d (arvados): closes #11104
Merge branch '11104-work-unit-expand-on-caret' Radhika Chippada
07:08 pm Revision 73c2bd55 (arvados): 11104: add fa-caret-down class to hyperlink rather than adding a spa...
Radhika Chippada
06:37 pm Task #11123 (New): Add node manager to install guide
Peter Amstutz
05:51 pm Revision 72873aff (arvados): 9277: Include trashed collections when creating copy collection for ...
request. Peter Amstutz
05:49 pm Revision dc086bed (arvados): closes #11103
Merge branch '11103-fts-crs' Radhika Chippada
05:48 pm Revision 197dd558 (arvados): 11103: ensure the index exists before attempting to drop it.
Radhika Chippada
05:28 pm Revision 2d80bb7d (arvados): 9277: crunch-run Log and output collections set is_trashed to true i...
setting trash_at. Peter Amstutz
05:27 pm Revision 1aa3f436 (arvados): 11103: Omit mounts from container_requests and definition from workf...
Radhika Chippada
05:09 pm Task #11122 (New): log partial job output to keep while job is running
Joshua Randall
05:00 pm Revision 0403a0c8 (arvados): 11121: Add owner_uuid to output collection on crunch-job
Lucas Di Pentima
03:42 pm Bug #9232 (Resolved): datamanager outputs replication level distribution histogram in unsorted order
(incorporated into keep-balance) Tom Clegg
03:25 pm Story #11017 (Resolved): [API Server] Implement Docker version compatibility fallback support
Applied in changeset arvados|commit:5624fec61db977d386ce03ca333241c74ca251b5. Tom Clegg
03:17 pm Revision 5624fec6 (arvados): Merge branch '11017-docker-migration'
closes #11017 Tom Clegg

02/15/2017

10:41 pm Revision 4edf67ae (arvados): Merge branch '10797-ruby-2.3'
closes #10797 Tom Clegg
10:40 pm Revision 31d76600 (arvados): 10797: Merge branch 'master' into 10797-ruby-2.3
Tom Clegg
10:40 pm Revision 03306e21 (arvados): 10797: Ruby 2.3.3
Tom Clegg
10:35 pm Revision 6873eca0 (arvados): Update golang version to 1.7.5. Also make the equivalent changes to the
(new) ubuntu1604 build scripts.
refs #10797
Ward Vandewege
10:25 pm Revision 6b775ec4 (arvados): Merge branch 'master' into 10797-ruby-2.3
refs #10797 Ward Vandewege
09:38 pm Task #11119 (New): Review 7995-keep-balance-docs
Tom Morris
09:37 pm Task #11118 (New): Review
Tom Clegg
09:35 pm Task #11115 (New): Review
Peter Amstutz
09:35 pm Task #11114 (New): Review
Peter Amstutz
09:35 pm Task #11112 (New): unit tests
Tom Clegg
09:35 pm Task #11111 (New): Review 11097-reuse-impure
Tom Clegg
09:30 pm Task #11108 (New): Review
Radhika Chippada
09:28 pm Bug #7853 (Closed): [Data Manager] Behave appropriately when multiple keepstore nodes share a sin...
Tom Clegg
08:33 pm Task #11107 (Resolved): Review 9277-trash-container-outputs
Peter Amstutz
08:29 pm Feature #10267 (Rejected): Permission-monitoring service keeps a flattened permissions table up t...
Tom Clegg
08:29 pm Feature #10683 (Rejected): [Crunch2] Separate service to poll sinfo and update nodes table
Node manager checks sinfo directly in #6520 Peter Amstutz
08:17 pm Task #11106 (Resolved): Review 6520-skip-compute0
Peter Amstutz
08:15 pm Revision fb60dbc9 (arvados): 6520: Don't ever assign compute0, start at slot 1 when assigning hos...
Peter Amstutz
08:08 pm Revision 314f3c34 (arvados): 6520: Check that hostname is set before setting weight.
Peter Amstutz
07:56 pm Revision 15c13298 (arvados): 6520: Apply node size fixup in CloudNodeListMonitorActor
Peter Amstutz
07:47 pm Revision 264ffa31 (arvados): 11097: Drop check for conflicting outputs when reusing completed con...
Tom Clegg
07:31 pm Revision 47476fa1 (arvados): 9849: Fix expensive "all readable collections" query.
Tom Clegg
05:57 pm Task #11035 (Resolved): Review 11017-docker-migration
Radhika Chippada
04:50 pm Task #11035 (In Progress): Review 11017-docker-migration
Radhika Chippada
05:56 pm Story #11017: [API Server] Implement Docker version compatibility fallback support
LGTM Radhika Chippada
05:28 pm Revision 0e6c1a28 (arvados): 6520: Update cloud_node.size to match server calculator configuration.
Peter Amstutz
05:14 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Skipping compute0 ensures that even if every other node compute1-compute255 in slurm are updated to smaller actual no... Peter Amstutz
05:10 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
It turns out you can't reconfigure hardware from "scontrol". You can only set the hardware configuration in slurm.co... Peter Amstutz
04:30 pm Revision e86743ca (arvados): 11017: Add arv-migrate-docker19 to package.
Tom Clegg
04:29 pm Revision 908959ea (arvados): 6520: Assign weight to node based on price to preferentially schedul...
cheaper nodes. Peter Amstutz

02/14/2017

09:26 pm Revision 83c8f168 (arvados): 10979: trackContainers func in crunch-dispatch-slurm.go
Radhika Chippada
09:22 pm Revision f14ce113 (arvados): 6520: Add testcase using sinfo to set crunch_worker_state
Peter Amstutz
09:01 pm Revision bf9af31f (arvados): 6520: Add testcase getting wishlist from squeue.
Peter Amstutz
08:21 pm Revision 4440f049 (arvados): 6520: Call squeue to include slurm jobs in wishlist. Call sinfo to ...
crunch_worker_state in arvados node records. Peter Amstutz
08:20 pm Task #11086 (Resolved): Review 10701-refactor-dispatch
Tom Clegg
08:20 pm Task #11090 (Resolved): Add arv-migrate-docker19 program
Tom Clegg
08:19 pm Story #11017: [API Server] Implement Docker version compatibility fallback support
11017-docker-migration @ commit:f782a2505422ad9c853c4c416640c41f3b1e7e79 Tom Clegg
08:16 pm Revision f782a250 (arvados): 11017: When compute nodes use image format v2, prefer migrated docke...
Tom Clegg
08:06 pm Feature #2660 (Resolved): [Tests] [Draft] Make entire test suite run faster by eliminating excess...
Tom Clegg
08:03 pm Feature #3807 (Closed): [DRAFT] [Crunch] Tasks should be reused if marked "pure"
Tom Clegg
08:02 pm Revision 3f303422 (arvados): Merge branch 'arvbox-docker-imgfmt' no issue #
Peter Amstutz
08:02 pm Revision c90f6525 (arvados): Update test_with_arvbox script.
Peter Amstutz
08:00 pm Story #3090 (Closed): [Crunch] Pipeline template has a "parameters" section whose values can be r...
Tom Clegg
07:11 pm Revision 6e8c06b8 (arvados): Set docker_image_formats to v1.
Peter Amstutz
06:30 pm Revision 347160e3 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
06:27 pm Revision e62a76da (arvados-dev): Add a new component, arvados-version-server.
No issue # Ward Vandewege
06:25 pm Revision 9bd113ee (arvados-dev): Add copyright notices to (almost) all files in this repository.
No issue # Ward Vandewege
06:19 pm Revision fd4b4bb1 (arvados-dev): Remove unused easy-docker-install.sh script.
No issue # Ward Vandewege
05:09 pm Revision 72b6a299 (arvados): Merge branch 'fix-arvbox' no issue #
Peter Amstutz
05:08 pm Revision 38bf86b7 (arvados): Fix arvbox build.
* Set up fuse groups separately.
* Separate generating superuser token from writing file so as not to write file on f...
Peter Amstutz
04:16 pm Revision 713a9030 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
04:15 pm Revision 739ade8f (arvados-dev): Docker 1.13 doesn't support tag --force anymore. Detect that.
No issue # Ward Vandewege
04:15 pm Bug #10701 (Resolved): [Crunch2] crunch-dispatch-slurm leaks file descriptors
Applied in changeset arvados|commit:c86cbaa6f286e50900dae3203a42044449e042f7. Tom Clegg
04:00 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
LGTM. Thanks. Radhika Chippada
03:46 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
Radhika Chippada wrote:
> * can we rename Dispatcher.running as “trackers” ?
Good call -- done.
> * In running...
Tom Clegg
12:12 am Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
In dispatch.go:
* can we rename Dispatcher.running as “trackers” ?
* In runningUUIDs, this comment is confusing...
Radhika Chippada
04:08 pm Revision c86cbaa6 (arvados): Merge branch '10701-refactor-dispatch'
closes #10701 Tom Clegg
03:50 pm Revision 0030acee (arvados): 10701: Update error text in test.
Tom Clegg
03:42 pm Revision 70f47bfd (arvados): 10701: Improve comments and identifiers.
Tom Clegg
03:36 pm Revision cf91723a (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
03:35 pm Revision 1680a689 (arvados-dev): The docker tag -f flag is now --force.
No issue # Ward Vandewege

02/13/2017

10:18 pm Revision 7c667d89 (arvados): 11017: Add arv-migrate-docker19
Tom Clegg
09:25 pm Story #9397 (Resolved): [Crunch2] Support prepopulating the output directory - CWL InitialWorkDir...
Applied in changeset arvados|commit:5763c67176e8e34656cd96881074777b14b2dc4a. Peter Amstutz
09:23 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Radhika Chippada wrote:
> Branch 9397-cwl-initialworkdir-crunchv2 @ commit:8c286d1
>
> * In test_container.py, s...
Peter Amstutz
09:24 pm Task #11018 (Resolved): Add support for InitialWorkDirRequirement to arvados-cwl-runner
Peter Amstutz
09:24 pm Task #11028 (Resolved): Update documentation
Peter Amstutz
09:24 pm Task #11051 (Resolved): Review 9397-cwl-initialworkdir-crunchv2
Peter Amstutz
09:19 pm Revision 5763c671 (arvados): Merge branch '9397-cwl-initialworkdir-crunchv2' closes #9397
Peter Amstutz
08:20 pm Revision 53b24cf4 (arvados): 9397: Improve container InitialWorkDir test.
Peter Amstutz
07:53 pm Revision 232a69ce (arvados): 9397: Fix keep URI splitting. Fix error when container has error log
collection with single file. Peter Amstutz
07:53 pm Revision 10b3fe2a (arvados): 9397: arvados-cwl-runner implementation of InitialWorkDir on contain...
Peter Amstutz
07:34 pm Revision 90982bd4 (arvados): Update schema-salad pin. refs #10993
Peter Amstutz
06:45 pm Bug #11097 (In Progress): [API] Reuse containers even when multiple matching containers exist wit...
Tom Clegg
06:45 pm Bug #11097: [API] Reuse containers even when multiple matching containers exist with differing ou...
11097-reuse-impure @ commit:264ffa31bae106bb6c36643e13186289b6cd0e18
...fails a few tests -- but perhaps only beca...
Tom Clegg
04:39 pm Bug #11097 (In Progress): [API] Reuse containers even when multiple matching containers exist wit...
h2. Background
Sometimes, running the same container twice on the same inputs can result in two successes with two...
Tom Clegg
06:40 pm Revision adf03c32 (arvados): 11097: Reuse containers even if we can detect they are impure. Fix e...
Tom Clegg
06:03 pm Revision 01779cbb (arvados): Update cwltool pin. closes #10993
Peter Amstutz
05:29 pm Revision 2a469c48 (arvados): Merge branch '9397-prepopulate-output-directory' refs #9397
Peter Amstutz
05:03 pm Task #10817 (Resolved): Review 9397-prepopulate-output-directory-pa
Peter Amstutz
05:01 pm Revision 091c92ae (arvados): Merge branch '10846-nodemanager-errors' closes #10846
Peter Amstutz
04:25 pm Feature #11095 (New): [CWL] Hint to control reuse of individual workflow steps
Through the use of a CWL hint, enable the user to enable/disable job reuse for individual tools or workflow steps.
...
Peter Amstutz

02/12/2017

05:19 pm Bug #11093 (New): "arv keep_service create" should go to an editor.
reading the docs, http://doc.arvados.org/install/install-keepstore.html at the very end:... Nico César

02/11/2017

07:25 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Branch 9397-cwl-initialworkdir-crunchv2 @ commit:8c286d1
* In test_container.py, should the path be “bar”, not “f...
Radhika Chippada

02/10/2017

11:23 pm Task #11072 (Resolved): Review 9397-go-manifest
Radhika Chippada
11:23 pm Task #11051 (In Progress): Review 9397-cwl-initialworkdir-crunchv2
Radhika Chippada
11:23 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Peter: branch 9397-prepopulate-output-directory-pa LGTM. Please make sure to review and make any amendments to the do... Radhika Chippada
07:43 pm Story #9397 (In Progress): [Crunch2] Support prepopulating the output directory - CWL InitialWork...
Peter Amstutz
11:10 pm Story #11017 (In Progress): [API Server] Implement Docker version compatibility fallback support
Tom Clegg
11:10 pm Task #11090 (In Progress): Add arv-migrate-docker19 program
Tom Clegg
11:10 pm Task #11090 (Resolved): Add arv-migrate-docker19 program
Tom Clegg
11:03 pm Revision e2f3f8e7 (arvados): Merge branch '11088-keep-cache-units'
closes #11088 Tom Clegg
10:15 pm Bug #11070 (Resolved): [arvados-ws] job logs are not being updated in real time
Applied in changeset arvados|commit:08c06bc813d96634b362e8c5736341fb2d874f59. Tom Clegg
10:08 pm Revision 08c06bc8 (arvados): Merge branch '11070-ws-listener-problem'
closes #11070 Tom Clegg
09:52 pm Task #10817 (In Progress): Review 9397-prepopulate-output-directory-pa
Radhika Chippada
06:54 pm Task #10817 (New): Review 9397-prepopulate-output-directory-pa
Peter Amstutz
09:42 pm Task #11086 (In Progress): Review 10701-refactor-dispatch
Radhika Chippada
07:19 pm Task #11086 (Resolved): Review 10701-refactor-dispatch
Tom Clegg
09:40 pm Task #11036 (Resolved): Review 10903-api-cancel-cascade
Radhika Chippada
09:40 pm Bug #10903 (Resolved): Cancel button on jobs & pipeline instances should cancel all child jobs too
Applied in changeset arvados|commit:d6aab18f9688d46bc1f86f021d68e02e5601cfe7. Radhika Chippada
08:20 pm Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
LGTM! Lucas Di Pentima
07:10 pm Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
Branch 10903-wb-cancel-cascade at commit:2e9cb97 addresses the suggestions from note 9 (need_transaction added).
T...
Radhika Chippada
09:39 pm Task #11044 (Resolved): Review 3900-trash-collections-delete-api
Radhika Chippada
09:36 pm Revision d6aab18f (arvados): closes #10903
Merge branch '10903-wb-cancel-cascade' Radhika Chippada
09:29 pm Revision 8c286d1b (arvados): 9397: Fix keep URI splitting. Fix error when container has error log
collection with single file. Peter Amstutz
09:21 pm Revision 5dfc48c1 (arvados): 11088: Fix keep_cache_ram units.
Tom Clegg
09:15 pm Revision 6e09079c (arvados): 11088: Fix keep_cache_ram units.
Tom Clegg
09:00 pm Revision dc82c1c1 (arvados): Merge branch '10956-arvput-pdh-fix'
Closes #10956 Lucas Di Pentima
08:31 pm Task #11073 (Resolved): Review 10903-wb-cancel-cascade
Lucas Di Pentima
07:26 pm Task #11073 (In Progress): Review 10903-wb-cancel-cascade
Lucas Di Pentima
07:44 pm Feature #6520 (In Progress): [Node Manager] [Crunch2] Take queued containers into account when co...
Peter Amstutz
07:25 pm Feature #3900 (Resolved): [Workbench] Trash button on collection uses "delete" API instead of set...
Applied in changeset arvados|commit:d3719c6ba29daec66eb1dd4003f2580f97da8e41. Lucas Di Pentima
04:24 pm Feature #3900: [Workbench] Trash button on collection uses "delete" API instead of setting expire...
* In the following, you might want to invoke "item.destroy" before updating "@removed_uuids" just for semantics
<pre...
Radhika Chippada
07:22 pm Revision f119109e (arvados): 9397: arvados-cwl-runner implementation of InitialWorkDir on contain...
Peter Amstutz
07:20 pm Bug #10945 (Resolved): [Workbench] No way to navigate from CWL job to its parent
Applied in changeset arvados|commit:35ae0e8d87dde84dd3a0ab41cef9567ab93b042a. Radhika Chippada
06:51 pm Bug #10945: [Workbench] No way to navigate from CWL job to its parent
LGTM, please merge. Lucas Di Pentima
05:19 pm Bug #10945: [Workbench] No way to navigate from CWL job to its parent
> In _show_status.html.erb Line 27: Maybe the word “the” should be eliminated from this title.
Updated
> In _sh...
Radhika Chippada
04:36 pm Bug #10945: [Workbench] No way to navigate from CWL job to its parent
Some small details:
* File @apps/workbench/app/views/jobs/_show_status.html.erb@
** Line 27: Maybe the word “the”...
Lucas Di Pentima
04:04 pm Bug #10945 (In Progress): [Workbench] No way to navigate from CWL job to its parent
Lucas Di Pentima
12:13 am Bug #10945: [Workbench] No way to navigate from CWL job to its parent
Branch 10945-used-in-section-for-cwl-jobs @ commit:78b94fd adds a "used in jobs" section to job#show page. I noticed ... Radhika Chippada
12:08 am Bug #10945: [Workbench] No way to navigate from CWL job to its parent
From IRC:
radhika
3:51 tom: just one quick question about 10945. looking at the code, the resolution for this iss...
Radhika Chippada
07:19 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
10701-refactor-dispatch @ commit:a2fe6f9367de3ee93064fbee3f2df78ce84aa318 Tom Clegg
07:18 pm Revision d3719c6b (arvados): Merge branch '3900-trash-collections-delete-api'
Closes #3900 Lucas Di Pentima
07:15 pm Revision 35ae0e8d (arvados): closes #10945
Merge branch '10945-used-in-section-for-cwl-jobs' Radhika Chippada
07:08 pm Revision 2e9cb970 (arvados): 10903: support need_transaction for job and pipeline_instance cancel...
Radhika Chippada
06:51 pm Task #11039 (Resolved): Review branch 10945-used-in-section-for-cwl-jobs
Lucas Di Pentima
04:07 pm Task #11039 (In Progress): Review branch 10945-used-in-section-for-cwl-jobs
Lucas Di Pentima
06:51 pm Revision fbc6c71e (arvados): 9397: Fix bind behavior to include Path. Update tests to check cont...
as well as output manifest. Peter Amstutz
06:40 pm Revision 397a7476 (arvados): 9397: Adjust stream normalization to use just digest (no hints) to d...
block locators. Peter Amstutz
06:31 pm Revision a2fe6f93 (arvados): 10701: Get multiple pages of queue if necessary.
Tom Clegg
06:16 pm Revision 9333c9f6 (arvados): 10701: Fix chan type.
Tom Clegg
06:11 pm Revision cff6cd57 (arvados): 10701: Add back MinRetryPeriod throttle. Update comments/identifiers.
Tom Clegg
06:04 pm Revision 6eded76b (arvados): 3900: Merge branch 'master' into 3900-trash-collections-delete-api
Lucas Di Pentima
06:01 pm Revision d6cc6f74 (arvados): 3900: Flipped item.destroy call for semantics reasons
Lucas Di Pentima
05:13 pm Revision ef9bbef0 (arvados): 10945: "Used in jobs" panel placement
Radhika Chippada
04:04 pm Revision ad464d41 (arvados): Merge branch 'master' into origin-9397-prepopulate-output-directory
Peter Amstutz
04:03 pm Revision 80689aac (arvados): Merge branch '9397-go-manifest' refs #9397
Peter Amstutz
04:02 pm Revision 4ee34d21 (arvados): 9397: Update comment & tests for CollectionFileReader to reflect it ...
lenient in the paths it accepts as a result of updates to implementation of
manifest.FileSegmentIterByName.
Peter Amstutz
02:31 pm Bug #11002 (In Progress): [arv-put] crash in arvfile on upload NoneType object has no attribute '...
Lucas Di Pentima
09:56 am Revision 6b1bf777 (arvados): 10701: cleanup
Tom Clegg
06:25 am Revision ddee3839 (arvados): 10701: Refactor dispatch library.
Tom Clegg
12:06 am Revision 78b94fd0 (arvados): 10945: show jobs in which this job is used in job#show page.
Radhika Chippada

02/09/2017

10:00 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
commit:d95e6df LGTM thanks! Tom Clegg
09:50 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Tom Clegg wrote:
> I think this would be an easy change, and it would make it possible to fix the "ignored error" ...
Peter Amstutz
08:25 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
If we have Extract() then we don't especially need NormalizedText() at all -- @m.Extract(...).Text@ would give us wha... Tom Clegg
08:19 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
ManifestTextForPath does two things at once, one of which we already have a separate function for. How about:
<pre...
Tom Clegg
06:22 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Tom Clegg wrote:
> If there's an edge case where it's expected to happen, can we check for that case explicitly be...
Peter Amstutz
09:58 pm Revision d95e6df5 (arvados): 9397: Tweak method documentation for Extract()
Peter Amstutz
09:55 pm Revision 7b4c1b70 (arvados): 9397: Update method documentation for Extract()
Peter Amstutz
09:47 pm Revision 4fa9baf6 (arvados): 10956: Fixed cwl test mocks to match behaviour change on collection ...
Lucas Di Pentima
09:46 pm Revision 4c081dd0 (arvados): 9397: Fix major bug in firstBlock(). Refactor ManifestTextForPath() to
Extract(). Test returning errors from Extract(). Peter Amstutz
09:14 pm Task #11036 (In Progress): Review 10903-api-cancel-cascade
Tom Clegg
08:39 pm Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
10903-api-cancel-cascade @ commit:3f83368
I think this needs to be protected by a transaction. Suggest new signatu...
Tom Clegg
12:59 am Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
Tom: branch 10903-wb-cancel-cascade @ commit:210af1cc7ede88914026fde078e45ef84c187a0c has both API and workbench upda... Radhika Chippada
07:30 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
Problems:
* sdk/go/dispatch assumes the dispatcher's RunContainer method is done as soon as the channel closes while...
Tom Clegg
05:15 pm Bug #10701 (In Progress): [Crunch2] crunch-dispatch-slurm leaks file descriptors
Tom Clegg
07:28 pm Revision 2b297df8 (arvados): 9397: Fix collection.go to use blockdigest.LocatorPattern
Peter Amstutz
07:28 pm Revision a0195a50 (arvados): 9397: Fix collection.go to use blockdigest.LocatorPattern
Peter Amstutz
07:25 pm Revision 85c68412 (arvados): 9397: Use ManifestTextForPath
Peter Amstutz
07:11 pm Revision 1c434a9f (arvados): Merge branch '9397-go-manifest' into origin-9397-prepopulate-output-...
Peter Amstutz
07:11 pm Revision 310f7b02 (arvados): 9397: Remove conflicting code for easier merge
Peter Amstutz
06:46 pm Revision 15b49783 (arvados): 10846: Specify whether to try to resume the node when cancelling shu...
Peter Amstutz
06:25 pm Revision 5bf9312a (arvados): 9397: Use idomatic loop form.
Peter Amstutz
06:21 pm Revision 2b783402 (arvados): 9397: Move most functions into private API.
* Document ManifestTextForPath().
* Check for invalid file segments, and fix tests that were using invalid file segme...
Peter Amstutz
05:24 pm Revision 4fd89ed7 (arvados): 10701: Tidy up error handling.
Tom Clegg
05:15 pm Revision 8ee2c7c4 (arvados): 10701: Fix edge case: scheduling_parameters.partitions=[]
Tom Clegg
05:11 pm Revision d5effac0 (arvados): 11070: Fix startup race: clients can safely connect before/during pq...
Tom Clegg
09:25 am Bug #11070 (In Progress): [arvados-ws] job logs are not being updated in real time
Tom Clegg
09:24 am Bug #11070: [arvados-ws] job logs are not being updated in real time
11070-ws-listener-problem @ commit:0fc6eaead0bf7691e99d19e74ec33636909001a7
testing on 9tee4 now.
to test behav...
Tom Clegg
09:04 am Revision 0fc6eaea (arvados): 11070: Add some logs, ensure shutdown after a pq connection problem.
Tom Clegg
08:35 am Revision 8e7c3b36 (arvados): Merge branch '10700-dispatch'
refs #10700 Tom Clegg
07:28 am Revision 4c1ca6b3 (arvados): 10700: Fix error message.
Tom Clegg
12:16 am Revision 210af1cc (arvados): 10903: use cancel with cascade to cancel jobs and pause pipelines.
Radhika Chippada

02/08/2017

11:03 pm Revision bf0945d1 (arvados): 10956: When asked for the recently uploaded collection's pdh, arv-pu...
Lucas Di Pentima
10:37 pm Revision ca566236 (arvados): 10956: Get PDH from API server's response when saving a collection s...
Updated tests to reflect this change. Lucas Di Pentima
09:52 pm Bug #11070: [arvados-ws] job logs are not being updated in real time
I think I have it: when we hit a connection problem, we stopped our own event loop but
* didn't tell the pq library'...
Tom Clegg
06:30 pm Bug #11070: [arvados-ws] job logs are not being updated in real time
I ran tcpdump on the active connection, and I could see the once-per-minute "ping" between arvados-ws and postgresql.... Tom Clegg
05:55 am Bug #11070: [arvados-ws] job logs are not being updated in real time
Lots of connections from arvados-ws to postgresql sitting in CLOSE_WAIT, which doesn't seem right:... Tom Clegg
09:34 pm Revision 2565719b (arvados): 11070: Add some logs, ensure shutdown after a pq connection problem.
Tom Clegg
08:56 pm Revision c99fd72a (arvados): 11070: Add some logs, ensure shutdown after a pq connection problem.
Tom Clegg
07:33 pm Feature #3900: [Workbench] Trash button on collection uses "delete" API instead of setting expire...
Updates: commit:4dbaf8c
Test run: https://ci.curoverse.com/job/developer-run-tests/159/
Reverted changes so that ...
Lucas Di Pentima
04:30 pm Feature #3900: [Workbench] Trash button on collection uses "delete" API instead of setting expire...
Branch @3900-trash-collections-delete-api@ at commit:108467e
Test run: https://ci.curoverse.com/job/developer-run-te...
Lucas Di Pentima
07:28 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Peter Amstutz wrote:
> It can happen if @wantPos@ refers to a location past the end of the stream. @sendFileSegment...
Tom Clegg
06:44 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
On further consultation, the Python SDK does turn it into @0:0:zero@9@ so that does count as a Go SDK bug. Either @s... Peter Amstutz
06:38 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Tom Clegg wrote:
> 9397-go-manifest @ commit:ab6a70e86dd041f3b4da167c59e3e91309f14365
>
> Functional
> * Is this...
Peter Amstutz
04:43 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
9397-go-manifest @ commit:ab6a70e86dd041f3b4da167c59e3e91309f14365
Functional
* Is this a panic (i.e., could only...
Tom Clegg
04:12 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
arvados-cwl-runner support in 9397-cwl-initialworkdir-crunchv2 Peter Amstutz
03:08 am Story #9397 (New): [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequi...
Okay, I think this will unblock you:
# Merge 9397-go-manifest (it is a separate branch, I'm going to ask Tom for a...
Peter Amstutz
06:59 pm Revision 4dbaf8c3 (arvados): 3900: Reverted changes so that only Collection deletion from a proje...
Lucas Di Pentima
04:43 pm Bug #11075: Trying to copy a pipeline template to 9tee4 fails
9tee4 is a docker 1.13 version only cluster, see #10901
maybe this ticket will be relevant once #11017 gets resolv...
Nico César
03:33 pm Bug #11075: Trying to copy a pipeline template to 9tee4 fails
... Bryan Cosca
03:29 pm Bug #11075 (New): Trying to copy a pipeline template to 9tee4 fails
trying to copy a pipeline template to 9tee4... Bryan Cosca
04:18 pm Revision 108467e2 (arvados): 3900: When deleting items inside a project, use the delete API. In t...
Lucas Di Pentima
04:14 pm Task #11018 (In Progress): Add support for InitialWorkDirRequirement to arvados-cwl-runner
Peter Amstutz
04:11 pm Revision 8a1a96b6 (arvados): 9397: arvados-cwl-runner implementation of InitialWorkDir on contain...
Peter Amstutz
03:28 pm Task #11072 (In Progress): Review 9397-go-manifest
Tom Clegg
12:14 am Task #11072 (Resolved): Review 9397-go-manifest
Peter Amstutz
02:00 am Task #11073 (Resolved): Review 10903-wb-cancel-cascade
Radhika Chippada
02:00 am Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
Branch 10903-api-cancel-cascade @ commit:3f833687 addresses the API server updates: (1) enhance job.cancel method to ... Radhika Chippada
12:14 am Revision ab6a70e8 (arvados): 9397: Add manifest normalization and sub-manifest extraction by path.
Introduces "SegmentedManifest" which stores streams -> files -> file segments.
Enables reexport of manifest in normal...
Peter Amstutz

02/07/2017

11:47 pm Revision b3fb8f09 (arvados): 9397: Make FileSegmentIterByName channel buffered so it isn't necess...
goroutine context switch on every segment. Peter Amstutz
10:51 pm Revision 3f833687 (arvados): 10903: Add cancel method with cascade to pipeline_instance and updat...
Radhika Chippada
10:47 pm Revision da3ed065 (arvados): 9397: Use binary search to find the first block. Store BlockOffsets...
Peter Amstutz
10:46 pm Bug #11071 (New): [Workbench] qr1hi performance tests are broken
Oldest 'OK' run: https://ci.curoverse.com/job/qr1hi-automated-performance-suite/3162/
Since then they have been br...
Ward Vandewege
09:59 pm Bug #11070 (Resolved): [arvados-ws] job logs are not being updated in real time
In workbench, job logs are updated in real time via websockets. On our test installations with arvados-ws, that is no... Ward Vandewege
09:44 pm Revision 559f3e22 (arvados): 9397: Improve manifest normalization efficiency
Peter Amstutz
07:50 pm Bug #9154 (Closed): data manager sometimes refuses to continue when collections are being written...
Tom Clegg
07:30 pm Feature #11064 (Duplicate): [Crunch2] crunch-run expires output collection immediately
Peter Amstutz
06:47 pm Revision e79722b0 (arvados): 9397: Syntax/logic fixes
Peter Amstutz
05:51 pm Bug #11068 (New): [Arvados-CWL-runner] need better error message when there are issues getting th...

The 9tee4 diagnostics job (e.g. 9tee4-xvhdp-yuv7hvb80fbb3lk) failed with this message:...
Ward Vandewege
02:26 pm Bug #10903 (In Progress): Cancel button on jobs & pipeline instances should cancel all child jobs...
Radhika Chippada

02/06/2017

10:24 pm Story #9397 (In Progress): [Crunch2] Support prepopulating the output directory - CWL InitialWork...
Radhika Chippada
05:37 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
> You can add "/" to the end of OutputPath, eliminates the need for the check on line 677
Updated
> Line 681 -...
Radhika Chippada
10:16 pm Revision 6045f3a6 (arvados): 9397: Add manifest normalization and sub-manifest extraction by path.
Peter Amstutz
09:56 pm Revision ef934e66 (arvados): 9397: add pre-population of output directory documentation
Radhika Chippada
07:41 pm Bug #11065: [API] Delete rows from logs table when they exceed a configured threshold
TBD:
* What should the default be? 2w?
* Any special considerations for existing sites that _want_ to keep logs for...
Tom Clegg
06:16 pm Bug #11065 (New): [API] Delete rows from logs table when they exceed a configured threshold
h2. Background
The logs table currently serves three purposes:
* an audit log, permitting admins and users to loo...
Tom Clegg
07:00 pm Revision d146a62d (arvados): 6520: Call squeue to include slurm jobs in wishlist. Call sinfo to ...
crunch_worker_state in arvados node records. Peter Amstutz
06:51 pm Task #11028 (In Progress): Update documentation
Radhika Chippada
04:15 pm Feature #11064 (Duplicate): [Crunch2] crunch-run expires output collection immediately
The collection only needs to exist long enough for the container request to make a copy which is furnished to the use... Peter Amstutz
02:06 pm Task #11042 (Resolved): Review 10968-arvput-resume-progress
Lucas Di Pentima
03:41 am Revision 8734a739 (arvados): Merge branch 'master' into 9397-prepopulate-output-directory
Radhika Chippada
03:40 am Revision 8ee173e8 (arvados): 9397: add normalized manifest test
Radhika Chippada
02:03 am Revision 837c509a (arvados): 9397: add handling of denormalized manifest
Radhika Chippada
12:14 am Revision 8adad211 (arvados): 9397: NormalizeManifest
Radhika Chippada

02/03/2017

10:40 pm Story #10968 (Resolved): [arv-put] Notify user that resume cache is being used & provide progress...
Applied in changeset arvados|commit:aaa45b09de0e9437743fce53d7c0bf8165074b5e. Lucas Di Pentima
10:00 pm Story #10968: [arv-put] Notify user that resume cache is being used & provide progress indication
LGTM, thanks! (Might as well fix errant space at EOL in comment, though... [[Coding Standards#Source-code-formatting]]) Tom Clegg
07:45 pm Story #10968: [arv-put] Notify user that resume cache is being used & provide progress indication
Branch @10968-arvput-resume-progress@ at: commit:ef328d9
Test run: https://ci.curoverse.com/job/developer-run-tests/...
Lucas Di Pentima
10:34 pm Revision aaa45b09 (arvados): Merge branch '10968-arvput-resume-progress'
Closes #10968 Lucas Di Pentima
10:34 pm Revision efee0049 (arvados): 10968: Deleted space char at the end of a comment
Lucas Di Pentima
09:52 pm Task #11042 (In Progress): Review 10968-arvput-resume-progress
Tom Clegg
09:19 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Ideally nodemanager will
* ensure the slurm config for the fake node matches the characteristics of the largest node...
Tom Clegg
09:15 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Another idea from chat:
Set aside one or more node entries which are sized to the biggest machines available (migh...
Peter Amstutz
08:55 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
One drawback of using the --hold workaround is that it also means we can't detect situations where the container is g... Peter Amstutz
04:57 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Problem: sbatch will fail "Requested node configuration is not available" if you try to add a job to the queue which ... Peter Amstutz
04:13 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
job cpus, memory, tempory disk space, reason:... Peter Amstutz
09:13 pm Revision 925a1526 (arvados): Merge branch 'master' into 10797-ruby-2.3
Conflicts:
build/package-build-dockerfiles/Makefile
Tom Clegg
09:00 pm Revision 785c967e (arvados): 9397: get manifest segment for a subdir or file using manifest from ...
Radhika Chippada
07:53 pm Feature #3900 (In Progress): [Workbench] Trash button on collection uses "delete" API instead of ...
Lucas Di Pentima
07:21 pm Revision ef328d91 (arvados): 10968: Added a notification when uploading at least one directory, j...
Lucas Di Pentima
06:20 pm Bug #10969 (Resolved): arv keep docker to check for valid Docker image and version
Applied in changeset arvados|commit:74a9decf66adc216d3d8bbb7ee8f6e3704d9590d. Tom Clegg
04:30 pm Bug #10969: arv keep docker to check for valid Docker image and version
This lgtm, please merge. Thanks! Lucas Di Pentima
06:15 pm Revision 74a9decf (arvados): Merge branch '10969-check-docker-format'
closes #10969 Tom Clegg
05:52 pm Revision 211884f4 (arvados): 10968: Changed the periodic update thread to run every 1 second whil...
When starts uploading the rest of the files, the update thread returns to the previous behaviour, running once every ... Lucas Di Pentima
05:04 pm Task #11057 (Resolved): Review 10932-arvput-slow-resuming
Lucas Di Pentima
02:31 pm Task #11057 (In Progress): Review 10932-arvput-slow-resuming
Peter Amstutz
04:33 pm Task #11061 (Resolved): crunch-dispatch-slurm running on cloud clusters
It's installed in every cluster in API server (but not running)... Nico César
03:32 pm Task #11061 (In Progress): crunch-dispatch-slurm running on cloud clusters
Nico César
03:29 pm Task #11061 (Resolved): crunch-dispatch-slurm running on cloud clusters
Peter Amstutz
04:31 pm Task #11033 (Resolved): Review 10969-check-docker-format
Lucas Di Pentima
03:45 pm Bug #10932 (Resolved): [arv-put] job resume too slow & uninformative
Applied in changeset arvados|commit:a8b210d2bb5b10e6583abf295b99788b3dff7479. Lucas Di Pentima
03:24 pm Bug #10932: [arv-put] job resume too slow & uninformative
Peter Amstutz wrote:
> Is there any reason @self._file_paths@ can't be a @set()@? It looks like the only thing that...
Lucas Di Pentima
02:30 pm Bug #10932: [arv-put] job resume too slow & uninformative
Is there any reason @self._file_paths@ can't be a @set()@? It looks like the only thing that would need to be change... Peter Amstutz
03:38 pm Revision a8b210d2 (arvados): Merge branch '10932-arvput-slow-resuming'
Closes #10932 Lucas Di Pentima
03:09 pm Revision e41dc06c (arvados): 10932: Changed _file_paths from being a list to a set so we're not g...
Added comments on cache saving explaining why is better to use json.dumps() instead of copy.deepcopy(). Lucas Di Pentima
02:50 pm Bug #11060 (New): [Workbench] "repositories" page does not show repositories shared with "All users"
Peter Amstutz
12:16 am Revision 471270bc (arvados): 9397: Use manifest.FileSegmentForPath to get manifest segment for a ...
the same collection repeatedly. If no manifest segment found for a mounted path, log that fact. Radhika Chippada

02/02/2017

10:36 pm Revision 6a7d7a2f (arvados): 10846: Revise shutdown behavior.
* Remove node blacklisting. Because of arvados node record reuse and assigning
compute node ids based on record uu...
Peter Amstutz
10:34 pm Task #11033 (In Progress): Review 10969-check-docker-format
Tom Clegg
10:34 pm Task #11033 (New): Review 10969-check-docker-format
Tom Clegg
10:34 pm Task #11033 (In Progress): Review 10969-check-docker-format
Tom Clegg
10:33 pm Bug #10969 (In Progress): arv keep docker to check for valid Docker image and version
10969-check-docker-format @ commit:d02bf4d817e50c3c0ee9f5e2dd901c512ea30943 Tom Clegg
07:45 pm Bug #10969: arv keep docker to check for valid Docker image and version
Hm. Looks like "registry v1" to "registry v2" is the change we're dealing with here, not manifest format. New version... Tom Clegg
07:17 pm Bug #10969: arv keep docker to check for valid Docker image and version
The addition looks fine, and the specification of format as well.
I think the default should be v2s2, not empty. A...
Ward Vandewege
07:11 pm Bug #10969: arv keep docker to check for valid Docker image and version
The two relevant formats are "image manifest V 2, schema 1" and "image manifest V 2, schema 2". We could label those ... Tom Clegg
10:32 pm Revision d02bf4d8 (arvados): 10969: Add docker_image_formats server config, and corresponding che...
Tom Clegg
10:19 pm Bug #10932: [arv-put] job resume too slow & uninformative
Updates at: commit:9c9b520
Tests: https://ci.curoverse.com/job/developer-run-tests/151/
A couple of optimizations...
Lucas Di Pentima
08:28 pm Bug #10932 (In Progress): [arv-put] job resume too slow & uninformative
Lucas Di Pentima
10:10 pm Revision 9c9b5203 (arvados): 10932: Replaced the use of a list with a set to check if files on lo...
Also, the save_state method was taking too much time on two operations: deepcopy() and json.dump(). Replaced both wit... Lucas Di Pentima
08:09 pm Task #11057 (Resolved): Review 10932-arvput-slow-resuming
Lucas Di Pentima
03:53 pm Bug #10978: [CWL] Consider changing keep mounts to use "-" instead of "+" in containers API
Just a thought: if it's too much work under the hood to use symbolic names like @"bamFile"@, we could consider @"/mnt... Tom Clegg
02:02 pm Story #10968 (In Progress): [arv-put] Notify user that resume cache is being used & provide progr...
Lucas Di Pentima

02/01/2017

09:44 pm Task #10997 (Resolved): More unit tests
Tom Clegg
09:35 pm Feature #10990 (Resolved): [keep-web] Add support for HTTP Range requests
Applied in changeset arvados|commit:519ab4c83aeb44ce91da941ecc191d00b6c6c72b. Tom Clegg
09:31 pm Revision 519ab4c8 (arvados): Merge branch '10990-keep-web-ranges'
closes #10990 Tom Clegg
09:30 pm Revision 1bba7f8f (arvados): 10990: Add Seek tests.
Tom Clegg
09:29 pm Revision 929b5059 (arvados): 10990: Add Seek tests.
Tom Clegg
08:58 pm Task #11051 (Resolved): Review 9397-cwl-initialworkdir-crunchv2
Radhika Chippada
08:55 pm Task #11044 (Resolved): Review 3900-trash-collections-delete-api
Lucas Di Pentima
08:55 pm Task #11042 (Resolved): Review 10968-arvput-resume-progress
Lucas Di Pentima
08:55 pm Task #11041 (New): Review
Lucas Di Pentima
08:53 pm Task #11039 (Resolved): Review branch 10945-used-in-section-for-cwl-jobs
Radhika Chippada
08:53 pm Task #11036 (Resolved): Review 10903-api-cancel-cascade
Radhika Chippada
08:51 pm Task #11035 (Resolved): Review 11017-docker-migration
Tom Clegg
08:51 pm Task #11033 (Resolved): Review 10969-check-docker-format
Tom Clegg
08:48 pm Task #11031 (Resolved): Review 6520-nodemanager-crunchv2
Peter Amstutz
08:42 pm Task #11028 (Resolved): Update documentation
Radhika Chippada
08:32 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Disregard #note-5 Peter Amstutz
08:30 pm Story #3150 (Rejected): [API] Add "trash" behavior for all object types, instead of literal rdbms...
Tom Clegg
08:30 pm Bug #4409 (Rejected): [Workbench] Want to remove projects from Home
Tom Clegg
08:15 pm Bug #11027 (New): [Crunchv2] python thread error after job success
A diagnostics cwl container request on 9tee4 (9tee4-xvhdp-hg279yhtfwcliv9) logged this:... Ward Vandewege
08:06 pm Bug #10701 (Closed): [Crunch2] crunch-dispatch-slurm leaks file descriptors
Closing because there's a good chance it got fixed in #10700 by getting rid of a bunch of stdin/stdout/stderr pipes. Tom Clegg
08:02 pm Task #10742 (Resolved): Review
Peter Amstutz
07:58 pm Bug #11026 (New): "Run a example test" Button.
I want to be able to ask someone to do an example pipeline run over the phone.
Without going into multistep exampl...
Nico César
07:57 pm Bug #10292 (Resolved): [CWL] Sometimes deadlocks on completion of subworkflow
Peter Amstutz
07:57 pm Task #10919 (Resolved): Wait and see if problem reoccurs
Peter Amstutz
07:56 pm Feature #10812 (Resolved): [CWL] Upload tool dependencies to separate collections
Peter Amstutz
05:40 pm Revision 04a8e2c7 (arvados): 10846: Remove duplicate log message suppression by ComputeNodeMonito...
Peter Amstutz
04:50 pm Revision e6986ed8 (arvados): 9397: Add ExcludeFromOutput to Mount struct.
Radhika Chippada
04:46 pm Revision f927cd24 (arvados): Merge branch '10705-dedup-log-message'
refs #10705 Tom Clegg
04:38 pm Revision 7d295784 (arvados): 10705: Fix double-logged error.
Tom Clegg
04:06 pm Task #10817 (In Progress): Review 9397-prepopulate-output-directory-pa
Peter Amstutz
03:54 pm Story #9397 (New): [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequi...
Notes:
crunchrun.go:670
You can add "/" to the end of OutputPath, eliminates the need for the check on line 6...
Peter Amstutz
03:00 pm Revision f24ddd34 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
02:59 pm Revision f5c8b138 (arvados-dev): Merge branch '9050-script-to-delete-merged-branches'
refs #9050 Javier Bértoli

01/31/2017

11:57 pm Story #10241 (Resolved): [Crunch2][Tests] Add a diagnostic test to run a container_request based ...
Nico needed to make sure the token used in configuration can read workflows. Radhika Chippada
10:20 pm Task #10769 (Resolved): Add a jenkins job to run container_requests based diagnostics test for 9t...
merged Nico César
10:18 pm Revision 46fa5309 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
10:16 pm Revision 51d325fa (arvados-dev): Merge branch '10769-diagnostics-9tee4'
refs #10769 Nico César
10:03 pm Bug #10813: Improve performance of arv-put
Just to record some of the final performance figures, I did three 600+ GB uploads with the following results:
- 2 ...
Tom Morris
03:15 pm Bug #10813 (Resolved): Improve performance of arv-put
Applied in changeset arvados|commit:7a53cfc92d4bca452a687db0a6f338e6deb1564a. Lucas Di Pentima
03:07 pm Bug #10813: Improve performance of arv-put
Lucas Di Pentima wrote:
> Updates: commit:e502060
> Tests: https://ci.curoverse.com/job/developer-run-tests/150/
>...
Peter Amstutz
09:40 pm Bug #10704 (Resolved): [Crunch2] sbatch submit failures not reported to use, loop forever
Applied in changeset arvados|commit:c3d4f8a585202ec58df5506934b698039c200b68. Tom Clegg
09:40 pm Bug #10703 (Resolved): [Crunch2] crunch-dispatch-slurm deadlocks instead of graceful shutdown
Applied in changeset arvados|commit:c3d4f8a585202ec58df5506934b698039c200b68. Tom Clegg
09:40 pm Bug #10702 (Resolved): [Crunch2] crunch-dispatch-slurm buggy error handling
Applied in changeset arvados|commit:c3d4f8a585202ec58df5506934b698039c200b68. Tom Clegg
09:40 pm Bug #10700 (Resolved): [Crunch2] crunch-dispatch-slurm pileup
Applied in changeset arvados|commit:c3d4f8a585202ec58df5506934b698039c200b68. Tom Clegg
09:24 pm Bug #10700: [Crunch2] crunch-dispatch-slurm pileup
LGTM. Peter Amstutz
08:31 pm Bug #10700: [Crunch2] crunch-dispatch-slurm pileup
Peter Amstutz wrote:
> Yes, I meant sbatch. I have to review the logs, but I believe it was getting into a situatio...
Tom Clegg
04:33 pm Bug #10700: [Crunch2] crunch-dispatch-slurm pileup
Tom Clegg wrote:
> Peter Amstutz wrote:
> > This seems to be the primary bug. This seems to explain the flip-flopp...
Peter Amstutz
12:24 am Bug #10700: [Crunch2] crunch-dispatch-slurm pileup
Refactored the squeue stuff, (I think) it's much simpler now.
I didn't see any leaking FDs in the old code, but I ...
Tom Clegg
09:38 pm Task #11018 (Resolved): Add support for InitialWorkDirRequirement to arvados-cwl-runner
Tom Morris
09:32 pm Revision c3d4f8a5 (arvados): Merge branch '10700-dispatch'
closes #10700
refs #10701
closes #10702
closes #10703
closes #10704
Conflicts:
services/crunch-dispatch-slurm/cru...
Tom Clegg
09:26 pm Task #11008 (Resolved): Review 10813-arv-put-six-threads
Peter Amstutz
09:26 pm Task #10741 (Resolved): Review
Peter Amstutz
09:26 pm Task #10739 (Resolved): Review
Peter Amstutz
09:26 pm Task #10749 (Resolved): Review
Peter Amstutz
09:26 pm Task #10992 (Resolved): Review 10990-keep-web-ranges
Peter Amstutz
09:26 pm Task #10991 (Resolved): Review 10700-dispatch
Peter Amstutz
09:23 pm Revision c90a8dac (arvados): 5439: Run tests in a publishable docker image
Tom Clegg
09:21 pm Bug #4401 (Resolved): [Docker] Check dependencies at top of build.sh
Tom Clegg
08:34 pm Bug #10721: [Docs] Missing docker_image link classes from API docs
Hey Peter can you describe this a little more? Nico César
08:24 pm Story #11017 (Resolved): [API Server] Implement Docker version compatibility fallback support
1. provide a python program that runs arv-keepdocker, looks for pairs like the ones below, and creates a tag link for... Tom Morris
08:03 pm Story #11016 (New): Document how to choose a suitable blob signature TTL
Tom Clegg
07:42 pm Feature #11015 (New): Improve throughput of crunch-run output-uploading stage using multi-threade...
To improve throughput of crunch-run job output uploading, add support for multi-threaded asynchronous transfers to hi... Tom Clegg
07:36 pm Bug #11014 (New): [Workbench] Hide the "idle/busy nodes" part of the dashboard if crunch1 APIs ar...
Tom Clegg
07:18 pm Feature #10980 (Resolved): [OPS] add ubuntu1604 packages
Ward Vandewege
07:10 pm Task #10987 (Resolved): [OPS] add ubuntu1604 build steps to jenkins
Ward Vandewege
07:10 pm Task #10987 (In Progress): [OPS] add ubuntu1604 build steps to jenkins
Ward Vandewege
07:10 pm Task #10986 (Resolved): [OPS] update build scripts for ubuntu1604
Ward Vandewege
05:27 pm Revision 36360d1f (arvados): Merge branch '10694-provenance-graph-consistency'
Closes #10694 Lucas Di Pentima
05:25 pm Bug #11013 (New): [WB] 'Used by' tab empty when no graph is available
On collection's 'Used by' tab, it would be useful to show some message (or make the tab greyed out, unclickable?) whe... Lucas Di Pentima
03:52 pm Revision 3860042a (arvados): Merge branch '10868-ip-addr-conflict'
refs #10868 Tom Clegg
03:13 pm Revision 7a53cfc9 (arvados): Merge branch '10813-arv-put-six-threads'
Closes #10813 Lucas Di Pentima
02:39 pm Revision 5caa73a3 (arvados): 9397: update comments
Radhika Chippada
12:19 am Revision a19ffad9 (arvados): 10700: Rename squeue identifiers (sqCheck = SqueueChecker{})
Tom Clegg

01/30/2017

11:32 pm Revision 1bc602ad (arvados): 10700: Simplify squeue checker.
Tom Clegg
11:29 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Successfully tested the deployed version of this. Tom Morris
09:37 pm Task #11008 (Resolved): Review 10813-arv-put-six-threads
Lucas Di Pentima
09:32 pm Bug #10813: Improve performance of arv-put
Updates: commit:e502060
Tests: https://ci.curoverse.com/job/developer-run-tests/150/
Addressed both comments and ...
Lucas Di Pentima
09:30 pm Bug #10700: [Crunch2] crunch-dispatch-slurm pileup
Peter Amstutz wrote:
> This seems to be the primary bug. This seems to explain the flip-flopping between monitored ...
Tom Clegg
09:29 pm Revision e502060f (arvados): 10813: Merge branch 'master' into 10813-arv-put-six-threads
Lucas Di Pentima
09:26 pm Revision dcf5452f (arvados): 10813: Added only_committed argument to ArvFile.manifest_text() meth...
Lucas Di Pentima
09:25 pm Revision 4f485c13 (arvados): 10813: Removed unnecessary argument
Lucas Di Pentima
09:20 pm Revision da428616 (arvados): 9397: if mount.Path ends with "/", trim it.
Radhika Chippada
08:59 pm Revision dc7d01f4 (arvados): Merge branch 'master' into 9397-prepopulate-output-directory
Radhika Chippada
08:58 pm Revision a5e7c0c8 (arvados): 9397: more testing
Radhika Chippada
08:17 pm Revision ecd83812 (arvados): 10868: Remove old DNS entry immediately when a new node re-uses an o...
Tom Clegg
07:39 pm Revision edd5811c (arvados): 10694: Flipped graph orientations so they are all (collections, jobs...
Lucas Di Pentima
06:34 pm Revision 28062130 (arvados): 9397: test with multiple mount points under subdirectories
Radhika Chippada
06:31 pm Revision 804d23f3 (arvados): 10694: Changed Pipeline Instance's Graph orientation to match Collec...
Lucas Di Pentima
06:05 pm Revision 49aa7305 (arvados): 10694: Changed the Provenance Graph orientation of Jobs to be the sa...
Lucas Di Pentima
04:33 pm Revision 65123c5a (arvados): 10700: Rephrase "should cancel" condition to be less unclear.
Tom Clegg
03:35 pm Bug #10690 (Resolved): "dump config yaml" for Go programs and Rails projects
Applied in changeset arvados|commit:aa05b5d5ee2e42f2e456cca0c2f3cd835ab7cef4. Tom Clegg
02:13 pm Bug #10690: "dump config yaml" for Go programs and Rails projects
As far as I can understand the code, it LGTM. I'd just do some minor changes, for consistency, ie:
in **services/k...
Javier Bértoli
03:33 pm Task #10925 (Resolved): Review 10690-config-dump
Tom Clegg
02:23 pm Task #10925 (In Progress): Review 10690-config-dump
Javier Bértoli
03:32 pm Story #10241: [Crunch2][Tests] Add a diagnostic test to run a container_request based test
radika take a look at
apps/workbench/app/controllers/work_units_controller.rb: attrs['command'] = ["arvados-...
Nico César
03:31 pm Revision aa05b5d5 (arvados): Merge branch '10690-config-dump'
closes #10690 Tom Clegg
03:24 pm Revision 658183e2 (arvados): enable consul webgui on :18500
Tom Clegg

01/29/2017

07:01 pm Revision 315fb56f (arvados): fix templated services
Tom Clegg
10:54 am Revision 55ce7e1b (arvados): unseal vault
Tom Clegg
07:15 am Revision eb459d42 (arvados): add vault, consul-template, arvados pkgs
Tom Clegg
01:15 am Revision 42ae3ec7 (arvados): add runit support
Tom Clegg
01:15 am Revision a31038eb (arvados): install+start arvados services (wip)
Tom Clegg
01:15 am Revision 0dda0005 (arvados): split webgui,main from server.go
Tom Clegg

01/28/2017

10:14 pm Revision fa3bb426 (arvados): add runit support
Tom Clegg
08:23 pm Bug #11002 (In Progress): [arv-put] crash in arvfile on upload NoneType object has no attribute '...
8995M / 387184M 2.3% Traceback (most recent call last):
File "/home/tfmorris/venv/bin/arv-put", line 4, in <module...
Tom Morris
04:58 pm Revision 31b9a86d (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
04:56 pm Revision 97d589fa (arvados-dev): Remove centos6 from our package upload script.
refs #10786 Ward Vandewege
04:56 pm Revision 3fb4f8f3 (arvados-dev): Remove debian7 from our package upload script.
refs #10785 Ward Vandewege
06:56 am Revision aede26e6 (arvados): refactor as procedural
Tom Clegg

01/27/2017

10:04 pm Revision 436f5c76 (arvados): 10690: De-duplicate dumpConfig.
Tom Clegg
10:04 pm Revision 71229e23 (arvados): 10690: Add -dump-config to arv-git-httpd, crunch-dispatch-slurm, kee...
Tom Clegg
10:04 pm Revision 28aea3c6 (arvados): 10690: Fix missing return after exit().
Tom Clegg
10:04 pm Revision fdecbc70 (arvados): 10690: Fix conflicting uses of "config" name.
Tom Clegg
09:42 pm Bug #10700: [Crunch2] crunch-dispatch-slurm pileup
Tom Clegg wrote:
> 10700-dispatch @ commit:e34a5060cfc1cc4821b431e8aa6778a31898e0eb
> * 10701 (leaking fds) &rarr; ...
Peter Amstutz
08:57 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Peter Amstutz wrote:
> I can't think of a way that an attacker would easily trick keep-web into requesting an arbitr...
Tom Clegg
08:44 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Peter Amstutz wrote:
> Peter Amstutz wrote:
>
> > * We could stream results through instead of adding another sto...
Peter Amstutz
08:42 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Tom Clegg wrote:
> Peter Amstutz wrote:
> > keep-web:
> >
> > * keep-web should have an option to adjust the cac...
Peter Amstutz
08:11 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Peter Amstutz wrote:
> keep-web:
>
> * keep-web should have an option to adjust the cache size
Agreed
> * c...
Tom Clegg
06:26 pm Feature #10990: [keep-web] Add support for HTTP Range requests
keep-web:
* keep-web should have an option to adjust the cache size
* cache should be (API token)+(hash)
Peter Amstutz
04:57 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Peter Amstutz wrote:
> * We could stream results through instead of adding another store-and-forward stage by usin...
Peter Amstutz
04:10 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Tom Clegg wrote:
> 10990-keep-web-ranges @ commit:24b137a5b3313778e2db7f5d1e0c82daf0634a9c
>
> Known issue: Seek(...
Peter Amstutz
07:15 am Feature #10990: [keep-web] Add support for HTTP Range requests
10990-keep-web-ranges @ commit:24b137a5b3313778e2db7f5d1e0c82daf0634a9c
Known issue: Seek() should get more unit t...
Tom Clegg
08:45 pm Revision 28a9d054 (arvados): 10690: Fix missing return after exit().
Tom Clegg
08:38 pm Task #10924 (Resolved): add dump option to remaining Go programs
Tom Clegg
08:38 pm Bug #10690: "dump config yaml" for Go programs and Rails projects
10690-config-dump @ -commit:b9aef91effaa4ddeb115579cff4c1a47b09b606c- commit:fdecbc701188e64dc9e539da2dddbbeded709b42 Tom Clegg
08:36 pm Revision a61bee32 (arvados): 10690: Add -dump-config to arv-git-httpd, crunch-dispatch-slurm, kee...
Tom Clegg
08:36 pm Revision b9aef91e (arvados): 10690: De-duplicate dumpConfig.
Tom Clegg
08:16 pm Feature #10998 (New): [keep-web] configurable block cache size
Block cache size is configurable in the SDK, but keep-web doesn't have a config entry, so keep-web's data cache is ef... Tom Clegg
08:16 pm Task #10997 (Resolved): More unit tests
Tom Clegg
07:35 pm Task #10991 (In Progress): Review 10700-dispatch
Peter Amstutz
07:35 pm Task #10992 (In Progress): Review 10990-keep-web-ranges
Peter Amstutz
07:17 am Task #10992 (Resolved): Review 10990-keep-web-ranges
Tom Clegg
07:32 pm Revision 1b16fe46 (arvados): 10990: Use AsyncStream to minimize store-and-forward latency
Peter Amstutz
07:06 pm Revision d6579d95 (arvados): Doc update: add trademark symbol next to the word 'Arvados' in 2 pla...
No issue # Ward Vandewege
06:31 pm Story #10996 (New): [SDK] Improve caching & data pipelining in Go SDK
Ideas
* BlockCache could use AsyncStream, e.g. 10990-keep-web-ranges-pa
* Could cache smaller portions of blocks
Peter Amstutz
06:30 pm Revision 827879be (arvados): Merge branch '10990-keep-web-ranges'
refs #10990 Tom Clegg
02:15 pm Task #10769 (Feedback): Add a jenkins job to run container_requests based diagnostics test for 9t...
Nico César
02:15 pm Task #10769: Add a jenkins job to run container_requests based diagnostics test for 9tee4 environ...
Radhika, can you check
https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-l9fj6nb5eqbpwv8#Stat...
Nico César
01:57 pm Task #10769: Add a jenkins job to run container_requests based diagnostics test for 9tee4 environ...
created user https://workbench.qr1hi.arvadosapi.com/users/qr1hi-tpzed-w68kvgin92qp9gs and copied 9tee4-7fd4e-tyhlr6qi... Nico César
02:13 pm Story #10241: [Crunch2][Tests] Add a diagnostic test to run a container_request based test
I couldn't find a way to copy to other people's projects from the workbench so I ended up doing:
arv-copy --src se...
Nico César
01:51 pm Revision 4d22bec8 (arvados-dev): 10769: added diagnostics for container requests in 9tee4
refs #10769 Nico César
07:27 am Revision 24b137a5 (arvados): 10990: Remove superfluous code for headers now handled by http.Serve...
Tom Clegg
07:10 am Revision b0014257 (arvados): 10990: Adjust readahead to work better on small segments. Improve co...
Tom Clegg
04:22 am Revision 2b37f7a8 (arvados): 10990: Fix collectionreader tests.
Tom Clegg
03:23 am Revision 1b8106da (arvados): 10990: Support Range headers with non-zero start offset.
Tom Clegg
12:46 am Revision e4fadb61 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
12:43 am Revision f3f7e6aa (arvados-dev): Add support for ubuntu1604 to run_upload_packages.py.
refs #10980 Ward Vandewege

01/26/2017

10:44 pm Feature #10980 (In Progress): [OPS] add ubuntu1604 packages
The build scripts have been updated; I merged that change into master. I tested the package build and test locally an... Ward Vandewege
06:23 pm Feature #10980 (Resolved): [OPS] add ubuntu1604 packages
Ward Vandewege
10:40 pm Revision dcb7f718 (arvados): 10990: Add tests for byte range requests that start at byte >0.
Tom Clegg
10:38 pm Revision 95d89844 (arvados): Merge branch '10980-add-ubuntu1604-support'
refs #10980 Ward Vandewege
10:33 pm Revision c1a46a8a (arvados): Document our Ubuntu 16.04 support.
refs #10988 Ward Vandewege
10:30 pm Revision 8675eec8 (arvados): Add support for Ubuntu 1604 ("xenial") to our build scripts.
refs #10980 Ward Vandewege
10:24 pm Revision 799b5333 (arvados): Remove old debian7 package testing file.
refs #10785 Ward Vandewege
10:23 pm Revision 9dd66f4d (arvados): Remove deprecated --force-yes flag during package testing, replace with
--allow-unauthorized
refs #10980
Ward Vandewege
09:04 pm Feature #10990 (In Progress): [keep-web] Add support for HTTP Range requests
Tom Clegg
07:55 pm Feature #10990: [keep-web] Add support for HTTP Range requests
Implementation plan:
Make CollectionFileReader seekable.
Enable keep-web to handle range requests at nonzero of...
Peter Amstutz
07:25 pm Feature #10990 (Resolved): [keep-web] Add support for HTTP Range requests
Currently keep-web only supports range requests which begin at byte 0. It's desirable for things like genome viewers ... Tom Morris
08:16 pm Bug #10700 (In Progress): [Crunch2] crunch-dispatch-slurm pileup
Tom Clegg
07:50 pm Bug #10700: [Crunch2] crunch-dispatch-slurm pileup
10700-dispatch @ commit:e34a5060cfc1cc4821b431e8aa6778a31898e0eb
* 10701 (leaking fds) &rarr; 318dd88 10701: Remove ...
Tom Clegg
08:06 pm Bug #10704 (In Progress): [Crunch2] sbatch submit failures not reported to use, loop forever
Tom Clegg
08:05 pm Bug #10703 (In Progress): [Crunch2] crunch-dispatch-slurm deadlocks instead of graceful shutdown
Tom Clegg
08:05 pm Bug #10701 (In Progress): [Crunch2] crunch-dispatch-slurm leaks file descriptors
Tom Clegg
03:32 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
Tried running the old version of submit() many times with a command that fails or can't exec, but this doesn't reprod... Tom Clegg
08:03 pm Task #10991 (Resolved): Review 10700-dispatch
Tom Clegg
07:40 pm Revision e34a5060 (arvados): 10700: Rename PollInterval to PollPeriod in library to match command...
Tom Clegg
07:37 pm Revision ae5eb12d (arvados): 10704: Rate-limit startup attempts per container.
Tom Clegg
06:46 pm Bug #10978: [CWL] Consider changing keep mounts to use "-" instead of "+" in containers API
Tom Morris wrote:
> [Tom replied after I'd composed this reply, but before I submitted it. I like this idea, but sti...
Peter Amstutz
06:33 pm Bug #10978: [CWL] Consider changing keep mounts to use "-" instead of "+" in containers API
Tom Clegg wrote:
> We could also consider naming the mount dirs according to purpose instead of content -- e.g., mou...
Peter Amstutz
07:31 am Bug #10978: [CWL] Consider changing keep mounts to use "-" instead of "+" in containers API
[Tom replied after I'd composed this reply, but before I submitted it. I like this idea, but still think the question... Tom Morris
06:28 pm Feature #10988 (New): [DOC] update documentation to list ubuntu1604
Ward Vandewege
06:27 pm Task #10987 (Resolved): [OPS] add ubuntu1604 build steps to jenkins
Ward Vandewege
06:27 pm Task #10986 (Resolved): [OPS] update build scripts for ubuntu1604
Ward Vandewege
04:10 pm Revision 7dd717ca (arvados): Bump schema-salad version for schema loading performance fix. no is...
Peter Amstutz
06:57 am Revision cdc28f63 (arvados): fade out when stale
Tom Clegg

01/25/2017

10:39 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
Just before reaching max fds, server logs show many attempts to run sbatch. Some failed ("Error submitting container ... Tom Clegg
09:52 pm Revision 683f5374 (arvados): 10703: Un-pyramid run().
Tom Clegg
09:50 pm Revision 318dd887 (arvados): 10701: Remove unneeded complexity in squeue invocation.
Tom Clegg
08:53 pm Bug #10978: [CWL] Consider changing keep mounts to use "-" instead of "+" in containers API
We could also consider naming the mount dirs according to purpose instead of content -- e.g., mount at @"/mnt/bamFile... Tom Clegg
06:42 pm Bug #10978 (New): [CWL] Consider changing keep mounts to use "-" instead of "+" in containers API
The "+" character is misinterpreted by some tools such as older versions of Picard. Consider changing it to "-" in k... Peter Amstutz
07:31 pm Revision a7222f4d (arvados): arv_docker_get_image should not modify dockerRequirement parameter. ...
Peter Amstutz
06:51 pm Revision a3ca6693 (arvados): Merge branch '10812-docker-upload-fix' refs #10812
Peter Amstutz
06:20 pm Revision 13c27ef9 (arvados): 10812: Fix bug and add test that docker dependencies are found and u...
Peter Amstutz
03:59 pm Revision e356309e (arvados): 10703: Do not catch signals in crunch-dispatch-slurm. Simplify "stop...
Tom Clegg
03:09 pm Revision 5ba9471d (arvados): Merge branch '10895-run-in-single-container-reuse' refs #10895
Peter Amstutz
02:53 pm Revision 0d7ceb1e (arvados): 10895: Don't report "reused" for Failed/Cancelled containers. Tweak...
message. Tweak behavior of --debug to also enable arvados SDK debugging. Peter Amstutz
01:24 pm Bug #8787 (Resolved): [API] repositories/get_all_permissions method is too slow
This got resolved with the last deploy no more Timeout alerts in my inbox Nico César
06:00 am Revision cf2e2be2 (arvados): add consul task
Tom Clegg
02:04 am Story #9397 (In Progress): [Crunch2] Support prepopulating the output directory - CWL InitialWork...
Radhika Chippada

01/24/2017

08:43 pm Task #10950 (Resolved): Review 9998-no-count-items-available
Tom Clegg
03:34 pm Task #10950 (In Progress): Review 9998-no-count-items-available
Radhika Chippada
08:35 pm Bug #9998 (Resolved): Improve performance of API server when keep-balance retrieves collections
Applied in changeset arvados|commit:da0aee751d8cb039c9b6b85a03e7d62cb973e3b3. Tom Clegg
08:14 pm Bug #9998: Improve performance of API server when keep-balance retrieves collections
* “exact” versus “exclude_count”: I do not want to belabor it if there is a future goal of supporting other choices s... Radhika Chippada
06:54 pm Bug #9998: Improve performance of API server when keep-balance retrieves collections
Radhika Chippada wrote:
> * The count option with potential values of ‘exact’ and ‘none’ seems non-intuitive from AP...
Tom Clegg
03:34 pm Bug #9998: Improve performance of API server when keep-balance retrieves collections
* The count option with potential values of ‘exact’ and ‘none’ seems non-intuitive from API perspective. A boolean va... Radhika Chippada
08:28 pm Revision da0aee75 (arvados): Merge branch '9998-no-count-items-available'
closes #9998 Tom Clegg
08:28 pm Revision a20c7702 (arvados): 9998: Update CLI usage.
Tom Clegg
08:27 pm Feature #6034 (Rejected): [Deployment] Add sysvinit script to the keepproxy package
Tom Morris
08:18 pm Revision 1edd524e (arvados): 9998: Remove superfluous word.
Tom Clegg
08:17 pm Revision b172bbd9 (arvados): Merge branch '10868-job-log-node-info'
refs #10868 Tom Clegg
08:13 pm Task #10818 (Resolved): Review 10813-arv-put-six-threads
Peter Amstutz
08:01 pm Bug #8787: [API] repositories/get_all_permissions method is too slow
see #10917 looks like we've go an order of magnitude of improvement Nico César
07:37 pm Bug #10969 (Resolved): arv keep docker to check for valid Docker image and version
Add list of supported Docker image format versions to API server discovery document.
arv keep docker to check API...
Tom Morris
07:33 pm Story #10968 (Resolved): [arv-put] Notify user that resume cache is being used & provide progress...
Inform the user that resume cache is being used & calcluations.
If possible, provide period progress updates to the ...
Tom Morris
07:02 pm Task #10924 (In Progress): add dump option to remaining Go programs
Tom Clegg
06:51 pm Revision 8e569c16 (arvados): 9998: Use links instead of specimens to test count param.
Tom Clegg
06:42 pm Revision 38ac8f3e (arvados): 9998: Update API docs.
Tom Clegg
03:35 pm Bug #10955 (New): arvados-cli gem dependency error
When installing the arvados-cli gem on my local machine, it installs OK, but get dependency errors when running it:
...
Javier Bértoli
03:13 pm Story #6958 (Resolved): [Deployment] Arrange to run tests on CentOS 6
This was done a long time ago, and we have since deprecated CentOS 6. Ward Vandewege
03:47 am Revision 1e7d93d8 (arvados): 10868: Include node UUIDs and properties in job logs.
Tom Clegg

01/23/2017

10:24 pm Revision 7fd60cc8 (arvados): 10895: Don't add uploaded files to pathmap if they are not referenced.
Peter Amstutz
10:14 pm Revision 1b4e2717 (arvados): 10895: Don't add uploaded files to pathmap if they are not referenced.
Peter Amstutz
10:04 pm Task #10950 (Resolved): Review 9998-no-count-items-available
Tom Clegg
10:03 pm Revision a01dc8ec (arvados): 10895: Don't add uploaded files to pathmap if they are not referenced.
Peter Amstutz
09:21 pm Revision aea835bc (arvados): 10895: Improve reuse for RunInSingleContainer jobs
* Consolodate and de-duplicate requirements and hints
* Use json.dumps() with sorted_keys=True to avoid issues with r...
Peter Amstutz
08:59 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Radhika Chippada wrote:
> 1. Is the *kind* going to be *collection* always for these mount points underneath the out...
Tom Clegg
07:01 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
1. Is the *kind* going to be *collection* always for these mount points underneath the output_dir?
2. What would b...
Radhika Chippada
08:33 pm Bug #9998: Improve performance of API server when keep-balance retrieves collections
Now at commit:16fe80b0e93ed8c8416b2dcbc0e2ad49bc850738 Tom Clegg
03:42 pm Bug #9998: Improve performance of API server when keep-balance retrieves collections
9998-no-count-items-available @ commit:b8de9b3e62e82b806576b237be5f317bf378169f https://ci.curoverse.com/job/develope... Tom Clegg
08:31 pm Revision 16fe80b0 (arvados): 9998: Handle timestamp collisions in collection index.
Tom Clegg
06:56 pm Revision 655d3e0a (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
06:55 pm Revision caee5f60 (arvados-dev): Abort diagnostics suite with prejudice if the git checkout fails.
No issue # Ward Vandewege
06:36 pm Revision 5100b47f (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
06:35 pm Revision fc7fe0e6 (arvados-dev): Abort performance suite with prejudice if the git checkout fails.
No issue # Ward Vandewege
04:33 pm Bug #10948 (New): [Workbench] Live stats should use consistent coloring for graphs
The colors assigned to different metrics aren't stable and can change from job to job. We should always use the same ... Tom Morris
03:44 pm Bug #10946 (New): [Docs] [SDKs] Mention in API docs and discovery doc that destroy=delete and ind...
Tom Clegg
03:41 pm Revision b8de9b3e (arvados): Merge branch 'master' into 9998-no-count-items-available
Tom Clegg
03:35 pm Bug #10935 (Resolved): [API] Empty list turned to null
Applied in changeset commit:80f042bec0b25966498324cb6aafb7fc24a36e12. Peter Amstutz
03:33 pm Task #10939 (Resolved): Review 10812-log-merge
Peter Amstutz
03:33 pm Task #10938 (Resolved): Review 10812-submit-runner-image
Peter Amstutz
03:33 pm Task #10889 (Resolved): Review 10812-cwl-separate-dependencies
Peter Amstutz
03:32 pm Revision 39189b90 (arvados): Merge branch '10812-cwl-separate-dependencies' refs #10812
Peter Amstutz
03:31 pm Revision 6143bba5 (arvados): 10812: Check for dockerOutputDirectory and raise UnsupportedRequirem...
Peter Amstutz
03:31 pm Revision dc5a33cb (arvados): 10812: Handle expected NotFoundError error when checking if file exi...
Peter Amstutz
03:31 pm Revision 01007e0a (arvados): 10812: Don't use cStringIO which doesn't like unicode strings.
Peter Amstutz
03:31 pm Revision 036c59ea (arvados): 10812: Fix imports, reference #main in cwl:tool
Peter Amstutz
03:31 pm Revision 7fa95f2d (arvados): 10812: Improve check that already packed workflow collection exists ...
PDH and similar name. Don't crash when registering pipeline template that has
array inputs.
Peter Amstutz
03:31 pm Revision f40364c4 (arvados): 10812: Don't try to upload embedded tools separately unless they hav...
identifier. Peter Amstutz
03:31 pm Revision 685af7fb (arvados): 10812: Update cwltool and schema-salad dependencies.
Peter Amstutz
03:31 pm Revision cd383b71 (arvados): 10812: Add num_retries to CollectionFetcher and collectionResolver.
Peter Amstutz
03:31 pm Revision 2f953026 (arvados): 10812: Handle workflow keep references.
Peter Amstutz
03:31 pm Revision ae9f7123 (arvados): 10812: Handle $schema references.
Peter Amstutz
03:31 pm Revision ddcaafee (arvados): 10812: Fix indentation typo.
Peter Amstutz
03:31 pm Revision 60d986b8 (arvados): 10812: Use packed workflows for all run modes.
Conflicts:
sdk/cwl/arvados_cwl/__init__.py
sdk/cwl/arvados_cwl/runner.py
Peter Amstutz
03:29 pm Revision c40389e0 (arvados): Merge branch '10812-log-merge' refs #10812
Peter Amstutz
03:28 pm Revision b6e15a58 (arvados): Merge branch '10812-submit-runner-image' refs #10812
Peter Amstutz
03:28 pm Revision 80f042be (arvados): Merge branch '10935-noop-munge' closes #10935
Peter Amstutz
03:20 pm Revision 223dfd6e (arvados): 10812: Handle expected NotFoundError error when checking if file exi...
Peter Amstutz
02:56 pm Feature #10812: [CWL] Upload tool dependencies to separate collections
> In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retri... Peter Amstutz
02:52 pm Revision dc17bad9 (arvados): 10812: Add num_retries to CollectionFetcher and collectionResolver.
Peter Amstutz
05:56 am Revision e95856da (arvados): report state of all tasks
Tom Clegg
01:42 am Revision 5e8fb0fa (arvados): add "download consul binary" task
Tom Clegg
 

Also available in: Atom