Activity

From 04/30/2017 to 05/29/2017

05/26/2017

08:39 pm Bug #11769: crunch-dispatch.rb tries to scancel crunchv2 containers
revision at commit:3ef1803
LGTM
Nico César
08:02 pm Bug #11769 (In Progress): crunch-dispatch.rb tries to scancel crunchv2 containers
Peter Amstutz
07:44 pm Bug #11769 (In Progress): crunch-dispatch.rb tries to scancel crunchv2 containers
When running both crunch-dispatch.rb and crunch-dispatch-slurm, crunch-dispatch.rb will attempt to cancel the contain... Peter Amstutz
08:23 pm Revision e3da56d9 (arvados): 11766: Bump cwltool version for deadlock fix.
Peter Amstutz
08:22 pm Revision 7c03ff7b (arvados): Fix crunch script to set trash_intermediate and intermediate_output_...
Peter Amstutz
08:01 pm Task #11771 (New): Review 11769-scancel-jobs-only
Peter Amstutz
08:00 pm Revision 9d6fb27b (arvados): 11769: HasUuid::UUID_REGEX matches anything that looks like an Arvad...
Peter Amstutz
07:51 pm Story #11770 (New): [Python SDK] Implement support for universal newline mode in Collections API
Implementation should match that of the Python 3 file open method as described at: https://docs.python.org/release/3.... Tom Morris
07:49 pm Revision 0c0a0d85 (arvados): 11769: HasUuid::UUID_REGEX matches anything that looks like an Arvad...
As a result, if crunchv1 and crunchv2 dispatchers are on the same cluster,
crunch-dispatch.rb will try to scancel con...
Peter Amstutz
07:32 pm Revision f7a665dc (arvados): Merge branch '11767-slurm-units' refs #11767
Peter Amstutz
07:31 pm Revision b436b04a (arvados): 11767: Test cases that it will convert fractional values from squeue.
Peter Amstutz
07:25 pm Revision 3cd9054a (arvados): 11767: Slurm apparently will print out half values (like 2.5G).
Peter Amstutz
07:14 pm Revision dbd91c31 (arvados): 11767: Recognize lowercase suffixes, just in case future versions of...
change the format again. Peter Amstutz
07:03 pm Revision 8d76f6ca (arvados): 11766: Bump cwltool version for deadlock fix.
Peter Amstutz
06:56 pm Task #11389 (Resolved): Review 11100-cwl-set-output-ttl
Peter Amstutz
06:53 pm Revision d0811968 (arvados): 11767: Correctly parse values with unit suffixes printed by squeue.
Peter Amstutz
06:48 pm Revision 2f2cf8be (arvados): 11767: Correctly parse values with unit suffixes printed by squeue.
Peter Amstutz
06:25 pm Feature #11100 (Resolved): [CWL] Intermediary collection handling can be specified
Applied in changeset arvados|commit:0da65aef99133cb76661f56f2adac83a77ad8ac9. Peter Amstutz
05:58 pm Feature #11100: [CWL] Intermediary collection handling can be specified
LGTM, thanks! Lucas Di Pentima
03:44 pm Feature #11100: [CWL] Intermediary collection handling can be specified
Please check the tests, there's one failing on me, at least when running locally:... Lucas Di Pentima
03:08 pm Feature #11100: [CWL] Intermediary collection handling can be specified
Lucas Di Pentima wrote:
> * File @sdk/cwl/arvados_cwl/arvcontainer.py:L178@ - Should that check be also on the @arva...
Peter Amstutz
06:21 pm Revision 0da65aef (arvados): Merge branch '11100-cwl-set-output-ttl' closes #11100
Peter Amstutz
06:21 pm Revision 4dac499d (arvados): Merge branch 'master' into 11100-cwl-set-output-ttl
Peter Amstutz
05:29 pm Revision 537861a6 (arvados): 11766: Bump cwltool version for deadlock fix.
Peter Amstutz
04:13 pm Revision d0b58511 (arvados): 11100: Fix test
Peter Amstutz
03:06 pm Revision 5223e68f (arvados): 11100: Add ciso8601 dependency
Peter Amstutz
02:42 pm Revision 79b826be (arvados): 11100: Add test for --trash-intermediate. Add log message when inte...
outputs are scheduled for trash. Peter Amstutz
01:42 pm Bug #11557 (In Progress): [CWL] arvados-cwl-runner with job reuse doesn't put the jobs/output col...
Lucas Di Pentima
02:34 am Revision d094ae4e (arvados): 11684: When packing small blocks into one, save references of the files
included on the block when committing it asynchronously, so that the
segment's locators can be updated at the put thr...
Lucas Di Pentima
01:49 am Revision 8ffcfd15 (arvados): closes #11362 Merge branch '11362-missing-input-sdk/cwl'
Jiayong Li
01:47 am Revision 9a7ea560 (arvados): Change visit in pathmapper.py to raise OSError if input file is not ...
Jiayong Li
01:38 am Revision 932a3186 (arvados): refs #11362 Merge branch '11362-missing-input-sdk/python'
Jiayong Li
01:36 am Revision e578f043 (arvados): Change statfile in run.py to be able to raise OSError, and change th...
Jiayong Li

05/24/2017

07:21 pm Task #11761 (New): Review
Tom Morris
07:18 pm Task #11759 (New): Review
Lucas Di Pentima
07:18 pm Task #11758 (New): Review
Lucas Di Pentima
07:17 pm Task #11757 (New): Review
Peter Amstutz
07:05 pm Task #11754 (New): Review
Radhika Chippada
06:59 pm Task #11753 (New): Review
Radhika Chippada
06:56 pm Task #11752 (New): Review
Radhika Chippada
06:53 pm Bug #11751 (New): [Workbench] In /container_requests page, display the column title as "Name" ins...
Radhika Chippada
06:27 pm Task #11750 (New): Rollback API server updates from 9587-include-trash-in-group-contents branch m...
Radhika Chippada
06:26 pm Task #11749 (New): Review
Radhika Chippada
06:26 pm Task #11587 (Resolved): Review 11546-fast-lock
Radhika Chippada
06:10 pm Revision 1d77dec0 (arvados): Merge branch '11501-job-stats-discrepancy'
Closes #11501 Lucas Di Pentima
06:09 pm Story #11748 (New): Standard convention for Nagios-compatible health check scripts kept alongside...
Establish a convention for health check scripts such as "keep-web-health-check.sh" which check cluster functionality ... Peter Amstutz
04:51 pm Revision d451eabb (arvados): 11501: Simplified helper method call. Changed wording of run time de...
Lucas Di Pentima
04:46 pm Feature #11100: [CWL] Intermediary collection handling can be specified
Side note: I found a typo (Aravdos) on @doc/user/cwl/cwl-runner.html.textile.liquid:129@, maybe you can add that fix ... Lucas Di Pentima
04:12 pm Feature #11100: [CWL] Intermediary collection handling can be specified
* File @sdk/cwl/arvados_cwl/arvcontainer.py:L178@ - Should that check be also on the @arvados-cwl-runner@ side?
* Do...
Lucas Di Pentima
02:55 pm Feature #11100: [CWL] Intermediary collection handling can be specified
I decided to add --trash-intermediate/--no-trash-intermediate as distinct options from --intermediate-output-ttl.
...
Peter Amstutz
02:56 pm Revision 456dde00 (arvados): Merge branch '11543-collection-per-tool' closes #11543
Peter Amstutz
02:38 pm Task #11370 (Resolved): Update cwl runner after API feature is merged
Peter Amstutz
02:36 pm Revision bee4fa82 (arvados): 11100: Separate "trash intermediate on success" behavior from "outpu...
Peter Amstutz
01:47 pm Revision 18007467 (arvados): 11501: Don't filter out reused children when calculating running time.
Use always wall time when saying how much time passed after completion/failure. Lucas Di Pentima
05:39 am Feature #11745 (New): Support schema.org/Google structured markup for datasets
https://developers.google.com/search/docs/data-types/datasets Tom Morris

05/23/2017

11:11 pm Revision 6eae7635 (arvados): 11501: When calculating a work unit's running time, only include 'le...
Lucas Di Pentima
07:53 pm Revision 72a4000d (arvados): 11543: Upload tool dependencies into single collection. Add test fo...
Peter Amstutz
07:53 pm Revision f60885ad (arvados): 11543: Bump version dependency on arvados-python-client.
Peter Amstutz
07:52 pm Revision df6578e9 (arvados): Merge branch '11543-uploadfile-collection' refs #11543
Peter Amstutz
07:28 pm Task #11727 (Resolved): Review branch 11710-container-request-show-perf
Radhika Chippada
07:25 pm Revision 8316594b (arvados): 11543: Upload tool dependencies into single collection. Add test fo...
Peter Amstutz
07:22 pm Revision 4a1df4f3 (arvados): 11543: arvados.command.run.uploadfiles takes optional Collection to ...
Peter Amstutz
07:08 pm Revision 81b0d721 (arvados): Merge branch 'master' into 9587-wb-trash-tab
Radhika Chippada
07:06 pm Revision 6898695e (arvados): 9587: trash tab
Radhika Chippada
06:54 pm Story #11742 (New): [API Server] Support for pulling Docker images and uploading them to Keep
Tom Morris
06:47 pm Story #11741 (New): [Crunch2] Create workflow object for CWL runs which are initiated from the co...
When submitting a workflow on the command line, CWL runner should create a corresponding Workflow record for the job ... Tom Morris
06:27 pm Story #11740 (New): Support display & reseting of expiration dates for collections
Tom Morris
06:02 pm Revision 381f8ef7 (arvados): 11543: Add test for collection per tool
Peter Amstutz
05:47 pm Revision c4d82e42 (arvados): 11543: Upload tool dependencies into single collection. Fix tests.
Peter Amstutz
02:26 pm Revision b310fc04 (arvados): 11501: Merge branch 'master' into 11501-job-stats-discrepancy
Lucas Di Pentima
02:21 pm Bug #11739 (New): [Workbench] dashboard is fetching each container for the container_requests bei...
Radhika Chippada
02:19 pm Feature #11100: [CWL] Intermediary collection handling can be specified
Reviewed commit:8645888f1 - branch @11100-cwl-set-output-ttl@
Some minor comments:
* File @doc/user/cwl/cwl-ext...
Lucas Di Pentima
01:21 pm Story #9587: [Workbench] Interface to list and untrash trashed collections
I had a similar thought. Once an object is trashed (not just future trash) it's going to be very hard to find in ord... Peter Amstutz
12:04 am Story #9587: [Workbench] Interface to list and untrash trashed collections
TomM and Peter, the more I look into it, the more this UI design is not making sense to me. Are we sure what we want ... Radhika Chippada

05/22/2017

08:46 pm Revision 26c14294 (arvados): Change statfile in run.py to raise exception when a reference file i...
Jiayong Li
08:16 pm Task #11389 (In Progress): Review 11100-cwl-set-output-ttl
Lucas Di Pentima
08:13 pm Revision 8645888f (arvados): 11100: Update/add tests for --intermediate-output-ttl
Peter Amstutz
08:05 pm Revision 3b12ef6b (arvados): 11684: Reverted easy fix to expose the bug: when there's a delay wri...
produced by packing smaller blocks into one, its locator doesn't get updated with the
correct access token, so it wil...
Lucas Di Pentima
07:15 pm Story #9587: [Workbench] Interface to list and untrash trashed collections
(The description, as written, does not define precisely the criteria for a collection to appear under the "Trash" tab) Peter Amstutz
06:53 pm Story #9587: [Workbench] Interface to list and untrash trashed collections
Should the Trash tab show collections with "trash_at" in the future? Because if not, there won't be any way in the U... Peter Amstutz
06:37 pm Story #9587: [Workbench] Interface to list and untrash trashed collections
This is conflating two different things - 1) something which has been deleted/trashed and 2) something which is expir... Tom Morris
05:25 pm Story #9587: [Workbench] Interface to list and untrash trashed collections
@is_trashed@ is set once trash_at is in the past. In the case of temporary files, trash_at is in the future.
I th...
Peter Amstutz
06:32 pm Revision 43dbdec9 (arvados): 11100: Implement & document arv:IntermediateOutput hint.
Peter Amstutz
05:29 pm Task #11370 (In Progress): Update cwl runner after API feature is merged
Peter Amstutz
05:06 pm Revision dd584702 (arvados): 11100: Propagate through to runner. Use intermediate_output_ttl con...
Peter Amstutz
04:51 pm Revision 67fbf00f (arvados): 11100: a-c-r sets output_ttl and deletes intermediate collections on...
Peter Amstutz
03:41 pm Revision 2e83bfa0 (arvados): 11501: Fix some tests when trying to use the walltime method on show...
Lucas Di Pentima
03:40 pm Revision ccc2ef47 (arvados): 11501: Improved test name
Lucas Di Pentima
03:10 pm Revision 8045ed8a (arvados): Merge branch '11369-crunchv2-notes' refs #11369
Peter Amstutz
02:57 pm Revision 89bf2019 (arvados): 11369: Add migration notes about crunchv1-to-crunchv2
Peter Amstutz

05/21/2017

09:31 pm Task #11661 (Resolved): Review 11644-mounts-api
Tom Clegg
09:31 pm Task #11728 (Resolved): Review 11590-log-reuse
Tom Clegg
09:30 pm Task #11723 (Resolved): Review
Tom Clegg
06:45 pm Story #11720 (Resolved): Add dependency management for Go packages
Applied in changeset arvados|commit:b22ff6242148a9b37baef6c74701fa85a8765adf. Tom Clegg
06:33 pm Revision b22ff624 (arvados): Merge branch '11720-govendor'
closes #11720 Tom Clegg

05/20/2017

04:25 pm Bug #11590 (Resolved): job reuse / re-use logic does not log decisions
Applied in changeset arvados|commit:a9b15a2f041744c8fac8c0b12fa8b61d1a7292f8. Tom Clegg
04:18 pm Bug #11590: job reuse / re-use logic does not log decisions
Radhika Chippada wrote:
> * Do we want to log the entire command and mounts? Would the log messages become too big a...
Tom Clegg

05/19/2017

11:36 pm Revision a9b15a2f (arvados): Merge branch '11590-log-reuse'
closes #11590 Tom Clegg
11:20 pm Revision 57fa190e (arvados): 9005: Add connection leak test.
Tom Clegg
11:20 pm Revision 25fb9f53 (arvados): 9005: Skip starting unused http server in handler tests.
Tom Clegg
11:17 pm Revision 138eedbe (arvados): Merge branch '9005-disable-keepalive'
refs #9005
refs #11726
refs #11729
Tom Clegg
11:16 pm Revision 83b73a47 (arvados): 9005: 11726: 11729: Disable http keepalive.
The previous workaround for #9005 did not account for the prefetch
feature: if a goroutine is using the HTTP client t...
Tom Clegg
10:07 pm Revision cee8f888 (arvados): 11501: A work unit running time with children was only computed calc...
childs, so if a work unit child has more children, its running time could differ. Lucas Di Pentima
09:55 pm Revision 820cd602 (arvados): 9587: project#Trash tab displays trashed collections
Radhika Chippada
08:32 pm Revision 695dee19 (arvados): 9005: Skip starting unused http server in handler tests.
Tom Clegg
08:23 pm Revision 303424c3 (arvados): Merge branch '9005-conn-leak'
refs #9005 Tom Clegg
08:06 pm Revision 1e7d0d5f (arvados): 9005: Fix missing Close().
Tom Clegg
08:06 pm Revision cad24bba (arvados): 9005: Fix missing error checks.
Tom Clegg
07:58 pm Story #9005: [SDKs] Go SDK's arvadosclient and keepclient should share http.Client objects by def...
Some related fixes:
9005-conn-leak @ commit:cad24bba2240b47f59bc5719a035e85ff5eb60ef
Tom Clegg
06:10 pm Story #9005: [SDKs] Go SDK's arvadosclient and keepclient should share http.Client objects by def...
Peter Amstutz wrote:
> I don't see how we can share the same HTTP client for both API requests and keep blocks when ...
Tom Clegg
04:01 pm Story #9005 (New): [SDKs] Go SDK's arvadosclient and keepclient should share http.Client objects ...
Tom Clegg wrote:
> Peter Amstutz wrote:
> > API client and keepstore have different socket settings, which probably...
Peter Amstutz
02:15 pm Story #9005 (In Progress): [SDKs] Go SDK's arvadosclient and keepclient should share http.Client ...
Tom Clegg
07:55 pm Revision 1eb8881a (arvados): 9005: Fix missing error checks.
Tom Clegg
07:55 pm Revision ccf1048d (arvados): 9005: Fix missing Close().
Tom Clegg
07:40 pm Revision 9b949e31 (arvados): Merge branch 'master' into 9587-wb-trash-tab
Radhika Chippada
07:40 pm Story #11710 (Resolved): [Workbench][Performance] container / container_request show page taking ...
Applied in changeset arvados|commit:49a9c33ea6119c42ad3d2d585ed7b2f9e4339132. Radhika Chippada
06:36 pm Story #11710: [Workbench][Performance] container / container_request show page taking too long to...
@ commit:d7e956d1ed2ffeb5b585896e6568481d474bb095
> Could lines 24 & 26 be unified by using an ‘||’ operator?
Y...
Radhika Chippada
02:42 pm Story #11710: [Workbench][Performance] container / container_request show page taking too long to...
Just a couple of small comments:
* File @apps/workbench/app/models/container_work_unit.rb@
** Could lines 24 & 26...
Lucas Di Pentima
07:34 pm Revision 49a9c33e (arvados): closes #11710
Merge branch '11710-container-request-show-perf' Radhika Chippada
07:34 pm Revision fe746755 (arvados): 11710: fix typo in finding children
Radhika Chippada
07:34 pm Revision 642e5979 (arvados): 11710: fetching requesting containers
Radhika Chippada
07:34 pm Revision d956fd4f (arvados): 11710: preload / batch retrieval of children of a container_work_unit
Radhika Chippada
07:05 pm Revision 23721f23 (arvados): 9005: Skip starting unused http server in handler tests.
Tom Clegg
06:30 pm Revision d7e956d1 (arvados): 11710: fix typo in finding children
Radhika Chippada
05:47 pm Revision dfe5c5a6 (arvados): 9005: Remove debug printf.
Tom Clegg
05:22 pm Bug #11692: CWL Expression Tool Javascript issue when using str.includes
str.include is not part of core Javascript. The standard library is described in the specification:
https://www.e...
Peter Amstutz
03:19 pm Revision 38a5c3e2 (arvados): Merge branch 'master' into 11710-container-request-show-perf
Radhika Chippada
01:24 am Bug #11590: job reuse / re-use logic does not log decisions
* Do we want to log the entire command and mounts? Would the log messages become too big and unreadable? Can we inste... Radhika Chippada
01:17 am Task #11728 (In Progress): Review 11590-log-reuse
Radhika Chippada

05/18/2017

10:20 pm Story #11720: Add dependency management for Go packages
besides that LGTM Nico César
06:19 pm Story #11720: Add dependency management for Go packages
Nico César wrote:
> shouldn't be using GOROOT ? I had the impression GOPATH is a colon/comma separated values of the...
Tom Clegg
06:09 pm Story #11720: Add dependency management for Go packages

in build/run-tests.sh...
Nico César
02:23 pm Story #11720: Add dependency management for Go packages
11720-govendor @ commit:865d1b22bb42a6c89ca9aa9a3513f3c61c6547c7 (missed adding vendor/.gitignore because it said to ... Tom Clegg
09:25 pm Revision d95ccc6b (arvados): 11590: Dry up log_reuse_info() calls.
Tom Clegg
09:01 pm Task #11683 (Resolved): Add logging to container reuse code
Tom Clegg
08:56 pm Task #11683 (In Progress): Add logging to container reuse code
Tom Clegg
09:01 pm Task #11728 (Resolved): Review 11590-log-reuse
Tom Clegg
08:10 pm Task #11727 (In Progress): Review branch 11710-container-request-show-perf
Lucas Di Pentima
08:03 pm Task #11727 (Resolved): Review branch 11710-container-request-show-perf
Radhika Chippada
08:10 pm Revision 35f06a3e (arvados): Merge branch 'master' into 9587-wb-trash-tab
Radhika Chippada
07:46 pm Revision c182087b (arvados): refs #9587
Merge branch '9587-include-trash-in-group-contents' Radhika Chippada
07:45 pm Revision 8ac17750 (arvados): 9587: add support for include_trash in groups_controller -> contents...
Radhika Chippada
07:38 pm Revision abdebdb1 (arvados): 11710: fetching requesting containers
Radhika Chippada
07:37 pm Task #11564 (Resolved): Review branch 9587-include-trash-in-group-contents
Lucas Di Pentima
05:30 pm Task #11564 (In Progress): Review branch 9587-include-trash-in-group-contents
Lucas Di Pentima
07:36 pm Story #9587: [Workbench] Interface to list and untrash trashed collections
Branch @9587-include-trash-in-group-contents@ LGTM, thanks. Lucas Di Pentima
01:50 am Story #9587: [Workbench] Interface to list and untrash trashed collections
Branch 9587-include-trash-in-group-contents @ commit:461b3f5a2edb53adda4f3b703d77e9efc0c262e9
Added support for in...
Radhika Chippada
06:54 pm Bug #11590: job reuse / re-use logic does not log decisions
Container reuse logging samples... Tom Clegg
02:28 pm Bug #11590: job reuse / re-use logic does not log decisions
updated to... Tom Clegg
06:53 pm Revision 02d1074b (arvados): 11590: Add container logging tests.
Tom Clegg
06:50 pm Feature #11644 (Resolved): [keepstore] mount-oriented APIs
Applied in changeset arvados|commit:d121e087ad1b4e91f869dbd57534c6d6ce51d19d. Tom Clegg
06:36 pm Feature #11644: [keepstore] mount-oriented APIs
Thanks for the explanations!
Ran @services/keepstore@ tests locally, all good.
LGTM.
Lucas Di Pentima
05:34 pm Feature #11644: [keepstore] mount-oriented APIs
Lucas Di Pentima wrote:
> * File @services/keepstore/volume.go@
> ** Lines 329-333: Can you please explain to me wh...
Tom Clegg
03:48 pm Feature #11644: [keepstore] mount-oriented APIs
Some questions:
* File @services/keepstore/volume.go@
** Lines 329-333: Can you please explain to me what’s happe...
Lucas Di Pentima
06:45 pm Revision d121e087 (arvados): Merge branch '11644-mounts-api'
closes #11644 Tom Clegg
06:12 pm Story #9005: [SDKs] Go SDK's arvadosclient and keepclient should share http.Client objects by def...
Peter Amstutz wrote:
> API client and keepstore have different socket settings, which probably why they were not com...
Tom Clegg
05:12 pm Story #9005: [SDKs] Go SDK's arvadosclient and keepclient should share http.Client objects by def...
A couple thoughts:
API client and keepstore have different socket settings, which probably why they were not combi...
Peter Amstutz
05:26 pm Revision 37003688 (arvados): 11644: Add DeviceID() to Volume interface.
Tom Clegg
05:22 pm Revision d77b362f (arvados): 11644: Unify block-index handlers. Move prefix arg to query string.
Tom Clegg
05:05 pm Revision 627fbeb0 (arvados): 11590: Log container reuse decisions.
Tom Clegg
03:59 pm Bug #11724 (New): Import Docker images without requiring Docker on client
Some users do not have access to Docker on the client. It should be possible to submit a compute node job which can ... Peter Amstutz
03:57 pm Bug #5426 (Resolved): [Workbench] Large downloads through workbench fail
Now using keep-web and planning to remove workbench download entirely. Peter Amstutz
03:42 pm Revision 55173b4f (arvados): 11710: preload / batch retrieval of children of a container_work_unit
Radhika Chippada
03:39 pm Bug #8784: [Keep-web] Generate directory listings
Related to this, workbench sharing links should link directly to the keep-web index page.
Peter Amstutz
02:41 pm Bug #11717 (Closed): vendor dir for crunch-run dependencies
Tom Clegg
02:26 pm Revision df55660f (arvados): Merge branch '11590-log-reuse'
refs #11590 Tom Clegg
02:26 pm Revision 84906c71 (arvados): 11590: Clarify "job state" condition in log message.
Tom Clegg
02:23 pm Task #11723 (Resolved): Review
Tom Clegg
02:21 pm Revision 865d1b22 (arvados): 11720: Add vendor/.gitignore.
Tom Clegg
03:05 am Story #11710 (In Progress): [Workbench][Performance] container / container_request show page taki...
Radhika Chippada
02:18 am Task #11664 (Resolved): Review
Radhika Chippada
02:18 am Task #11697 (Closed): Review 11580-cr-display-perf
Radhika Chippada
02:18 am Task #11667 (Resolved): Review 11580-container-requests
Radhika Chippada
02:17 am Bug #11278 (Resolved): Trailing space on string pasted into search box should not cause "Oops req...
Radhika Chippada
01:47 am Revision 461b3f5a (arvados): 9587: add support for include_trash in groups_controller -> contents...
Radhika Chippada

05/17/2017

09:08 pm Story #11720: Add dependency management for Go packages
11720-govendor @ commit:b1b7a0be3c789795675e731c8c64bee3cb6d718c
example run https://ci.curoverse.com/job/develope...
Tom Clegg
09:07 pm Story #11720: Add dependency management for Go packages
Procedure for updating dependencies:
# Ensure @$GOPATH@ points somewhere (maybe @~/go@)
# @go get github.com/kardia...
Tom Clegg
07:05 pm Story #11720 (In Progress): Add dependency management for Go packages
Tom Clegg
07:04 pm Story #11720: Add dependency management for Go packages
The first godep problem I ran into was "discussed on github":https://github.com/tools/godep/issues/377:
> FWIW: No...
Tom Clegg
04:20 pm Story #11720 (Resolved): Add dependency management for Go packages
Our Go packages (programs and libraries) need something analogous to bundler, so changes in our dependencies don't br... Tom Clegg
09:07 pm Revision b1b7a0be (arvados): 11720: Update Go dependencies.
Tom Clegg
08:50 pm Bug #11546 (Resolved): Expensive container lock/unlock
Applied in changeset arvados|commit:b3f88934721fc97973b40786cae97e55e84b31d5. Tom Clegg
06:55 pm Bug #11546: Expensive container lock/unlock
LGTM 11546-fast-lock @ 78390940029ab08efecab923615c545b9da922b4
Peter Amstutz
06:41 pm Bug #11546: Expensive container lock/unlock
So an actual idiomatic way of writing this would be:... Peter Amstutz
06:31 pm Bug #11546: Expensive container lock/unlock
indeed
11546-fast-lock @ commit:78390940029ab08efecab923615c545b9da922b4
Tom Clegg
06:24 pm Bug #11546: Expensive container lock/unlock
I think maybe you need a transaction? Looks like normally AR starts its transaction only once you call save and it s... Peter Amstutz
06:03 pm Bug #11546: Expensive container lock/unlock
11546-fast-lock @ commit:fbc576b76977938cf7b742f9770ab90559136dc8
* threw away the "circumvent AR update" code
* us...
Tom Clegg
04:10 pm Bug #11546: Expensive container lock/unlock
Another approach would be to go back to the obvious Rails locking mechanism, but use "select for share nowait".
ht...
Tom Clegg
02:41 pm Bug #11546: Expensive container lock/unlock
11546-fast-lock @ commit:f3d300ce629f1c17ccc83bf1978ff5c7dd41e33d
Still needs some tests, but does this seem like ...
Tom Clegg
08:48 pm Revision 37470b03 (arvados): 11720: Merge branch 'master' into 11720-govendor
Tom Clegg
08:47 pm Revision b3f88934 (arvados): Merge branch '11546-fast-lock'
closes #11546 Tom Clegg
07:26 pm Revision 513b7af1 (arvados): 11720: Control dependencies with govendor.
Tom Clegg
06:40 pm Revision f3c054cb (arvados): 11720: Control dependencies with govendor.
Tom Clegg
06:35 pm Story #11580 (Resolved): [Workbench] Improve format of /container_requests index page
Applied in changeset arvados|commit:6e5079ff9a0349c57ab2cf06398413018a921cc0. Radhika Chippada
02:11 pm Story #11580: [Workbench] Improve format of /container_requests index page
LGTM, thanks! Lucas Di Pentima
01:59 pm Story #11580: [Workbench] Improve format of /container_requests index page
> File apps/workbench/app/views/container_requests/_show_recent.html.erb:L18: Adding a space “ContainerRequest" => "C... Radhika Chippada
01:15 pm Story #11580: [Workbench] Improve format of /container_requests index page
> When scrolling, it seems that the new pages don’t keep the created_at ordering (attached screen capture)
I see t...
Radhika Chippada
06:29 pm Revision a0431de2 (arvados): Merge branch 'master' into 9587-wb-trash-tab
Radhika Chippada
06:28 pm Revision 6e5079ff (arvados): closes #11580
Merge branch '11580-container-requests' Radhika Chippada
06:28 pm Revision 78390940 (arvados): 11546: Wrap lock/unlock in transactions.
Tom Clegg
06:28 pm Revision 80407a6b (arvados): 11580: add container_requests index page
Radhika Chippada
06:28 pm Revision e9481fd0 (arvados): 11580: preload containers
Radhika Chippada
06:26 pm Task #11587 (In Progress): Review 11546-fast-lock
Peter Amstutz
05:51 pm Revision fbc576b7 (arvados): 11546: Avoid loading/saving non-essential fields in /arvados/v1/cont...
Tom Clegg
05:50 pm Revision 867c69d1 (arvados): Merge branch '11714-crunch-run-cgroup-parent' closes #11714
Peter Amstutz
05:15 pm Bug #11718 (Resolved): Update crunch-run for docker client lib update
Applied in changeset arvados|commit:f0e917d2305c66cb7120125aaf6592ce2b9e3855. Peter Amstutz
04:52 pm Bug #11718: Update crunch-run for docker client lib update
Reviewed commit:9d33e0c4f
Jenkins tests ran ok: https://ci.curoverse.com/job/developer-run-tests/288/
There's no ...
Lucas Di Pentima
02:17 pm Bug #11718 (Resolved): Update crunch-run for docker client lib update
The function signature for Docker.ContainerWait has changed, this is breaking crunch-run.
In addition, github.com/...
Peter Amstutz
05:08 pm Revision 7a4bee52 (arvados): 11714: Set CgroupParent under Resources because setting Cgroup in Ho...
doesn't work. Peter Amstutz
05:08 pm Revision f0e917d2 (arvados): Merge branch '11718-crunch-run-docker-wait' closes #11718
Peter Amstutz
04:54 pm Revision 1e5f697d (arvados): 11580: cleanup test to remove unnecessary test
Radhika Chippada
04:14 pm Bug #11137: apt-get upgrade arvados-api-server on Ubuntu 12.04 did not update crunch-job
crunch-job is in the arvados-src package. Perhaps apiserver needs to list "exact same version of arvados-src package"... Tom Clegg
03:07 pm Revision 8347bb8e (arvados): 11714: Set CgroupParent under Resources because setting Cgroup in Ho...
doesn't work. Peter Amstutz
03:04 pm Revision 9d33e0c4 (arvados): 11718: Update crunch-run for docker client API change in ContainerWa...
Peter Amstutz
02:59 pm Bug #11547: [API] Excessive memory and time consumed by API and dispatch when container fields (l...
Instead of setting a column size limit, we can scan the JSON serialization to get an object count:... Peter Amstutz
02:49 pm Revision f3d300ce (arvados): 11546: Avoid loading/saving non-essential fields in /arvados/v1/cont...
Tom Clegg
02:43 pm Revision 7a02915c (arvados): 11546: Avoid loading/saving non-essential fields in /arvados/v1/cont...
Tom Clegg
02:24 pm Revision bbb8b8ee (arvados): 11546: Avoid loading/saving non-essential fields in /arvados/v1/cont...
Tom Clegg
02:12 pm Revision dfc18fd7 (arvados): 11546: Avoid loading/saving non-essential fields in /arvados/v1/cont...
Tom Clegg
02:04 pm Bug #11717 (Closed): vendor dir for crunch-run dependencies
Peter Amstutz
01:49 pm Bug #11716 (New): [Workbench] The sort order of items on /pipeline_instances and /container_reque...
The pipeline instance rows are sorted in each page. However, between pages, we can see the items are out of order. Pl... Radhika Chippada
01:47 pm Revision 68089478 (arvados): 11580: @limit on container_requests index page
Radhika Chippada

05/16/2017

08:33 pm Task #11661 (In Progress): Review 11644-mounts-api
Lucas Di Pentima
08:31 pm Story #11580: [Workbench] Improve format of /container_requests index page
Reviewed commit:701b7fa87
* File @apps/workbench/app/views/container_requests/_show_recent.html.erb:L18@: Adding a...
Lucas Di Pentima
07:40 pm Story #11580: [Workbench] Improve format of /container_requests index page
> It seems to me we should do something like
>> Get a page worth of container request records with one API call
>> ...
Radhika Chippada
02:57 pm Story #11580: [Workbench] Improve format of /container_requests index page
The addition to _show_status.html doesn't seem right to me. It looks like it's trying to preload the children of the ... Tom Clegg
12:05 am Story #11580: [Workbench] Improve format of /container_requests index page
Branch 11580-cr-display-perf @commit:1d45ac3d3
* Preloads containers of the main object being displayed. This now...
Radhika Chippada
07:44 pm Task #11628 (Resolved): Rwview 10111-collection-labels
Peter Amstutz
01:51 pm Task #11628 (In Progress): Rwview 10111-collection-labels
Peter Amstutz
07:38 pm Story #11710 (Resolved): [Workbench][Performance] container / container_request show page taking ...
Loading of the pages /containers/qr1hi-dz642-uypqs2exiy7so7l and /container_requests/qr1hi-xvhdp-6490dha4xorrfwz take... Radhika Chippada
07:33 pm Revision 701b7fa8 (arvados): 11580: preload containers
Radhika Chippada
07:20 pm Bug #11546: Expensive container lock/unlock
Using a direct row update is awkward because it bypasses all the normal ActiveRecord machinery: permission_to_update,... Peter Amstutz
06:06 pm Bug #11546: Expensive container lock/unlock
ActiveRecord @with_lock@ is implemented using "SELECT FOR UPDATE" to acquire a row level write lock for duration of t... Peter Amstutz
06:58 pm Feature #5733 (Closed): Add GNU Guix to Arvados
Peter Amstutz
06:56 pm Story #7928 (Duplicate): [Keep] keepstore identifies underlying volumes to clients
covered by #11644 Tom Clegg
06:56 pm Bug #7343 (Closed): error: reference to undefined identifier ‘sync.Pool’ sync.Pool
Peter Amstutz
06:51 pm Story #7935 (Resolved): [Keep] Test S3 block storage on GCE
Peter Amstutz
06:42 pm Bug #10492 (Rejected): Chatty job logging causes major slowdowns
Design flaw, won't fix. Peter Amstutz
06:40 pm Bug #3259 (Closed): Getting online should be easier
Lucas Di Pentima
06:38 pm Bug #8170 (Closed): Node not doing anything, and cannot tell what happened.
Peter Amstutz
06:38 pm Bug #8205 (Duplicate): Node manager is not spinning up nodes.
Tom Clegg
06:34 pm Feature #10765 (Resolved): [arvados-ws] switch our test infrastructure over to arvados-ws
Tom Clegg
06:34 pm Support #8182 (Closed): Run PCA on Pythonic Tilings Pipeline Fail
Lucas Di Pentima
06:32 pm Bug #8229 (Closed): Node failure on compute0
Peter Amstutz
06:32 pm Story #3299 (Closed): [Workbench] Render textile markup in description fields
We have support for textile markup in description fields now. Radhika Chippada
06:31 pm Bug #8323 (Resolved): [API] Puma hangs forever on a futex, requiring restart
Lucas Di Pentima
06:23 pm Story #8574 (Rejected): Changing architecture for dynamic node spinup
Peter Amstutz
06:23 pm Bug #11209 (Resolved): stuck keep fuse mounts not cleared by crunch-job
Tom Clegg
06:22 pm Story #8964 (In Progress): [SDKs] Design more idiomatic and convenient Go SDK
Peter Amstutz
06:21 pm Story #8647 (Resolved): [Workbench] "Work unit" user interface
Peter Amstutz
06:20 pm Story #11705 (New): [API] Remove async permissions code
We added the option so we could experiment with it, but it's obsolete now that we check permission using a recursive ... Tom Clegg
06:18 pm Story #9186 (Closed): [API] Test client impact of async_permissions_update=True
Obsoleted by current implementation of permission checks. Tom Clegg
06:15 pm Bug #4553 (Closed): [Docker] Compute node records should get created automatically by build/run s...
Peter Amstutz
06:13 pm Story #8180 (Rejected): [Keepstore] Don't trash/delete blocks until they appear on multiple trash...
Tom Clegg
06:11 pm Feature #3402 (Resolved): If a pipeline/job is in a project, it would be nice if the breadcrumbs ...
Peter Amstutz
06:10 pm Bug #3229 (Closed): Keep logging improvements
Tom Clegg
06:09 pm Bug #4977 (Resolved): [Crunch] "stdbuf": executable not found in $PATH
Checks for stdbuf in path. Peter Amstutz
06:08 pm Story #2877 (Rejected): Script looks for CGI data in a specified folder, scrapes header comments ...
Tom Clegg
06:07 pm Feature #4011 (Rejected): [Crunch] port crunch-job to python
Peter Amstutz
06:06 pm Story #8233 (In Progress): [Arvbox] Fix remaining tests & use arvbox on jenkins
Tom Clegg
06:05 pm Bug #6754 (Rejected): [Workbench] Report HTTP error instead of sending 0-byte file when error fet...
Workbench download is deprecated and scheduled for removal, superseded by keep-web. Peter Amstutz
06:03 pm Bug #1668 (Rejected): workbench jobs page does not show who submitted a job
Tom Clegg
05:31 pm Revision 42f5f3a2 (arvados): 11644: Test fields in /mounts response.
Tom Clegg
05:25 pm Feature #11644 (In Progress): [keepstore] mount-oriented APIs
11644-mounts-api @ commit:42f5f3a29761ac9a943530157da376c798a4ac6d
Tom Clegg
05:23 pm Revision 9eff4af0 (arvados): 11644: Ensure generated UUIDs are always 27 chars.
Tom Clegg
05:22 pm Revision c4b5194c (arvados): 11644: Add volume replication level to /mounts response.
Tom Clegg
05:11 pm Revision 8cc51c76 (arvados): 11644: Replace linear search with map for looking up mounts by UUID.
Tom Clegg
05:07 pm Task #11687 (Resolved): Obey MountUUID restrictions in trash list
Tom Clegg
05:07 pm Task #11688 (Resolved): Obey MountUUID restrictions in pull list
Tom Clegg
05:03 pm Revision 74544f43 (arvados): 11644: Test non-empty MountUUID in trash list.
Tom Clegg
04:48 pm Revision 38de76ce (arvados): 11644: Add pull-to-mount-UUID test. Tidy up pull worker and tests.
Tom Clegg
04:47 pm Task #11667 (In Progress): Review 11580-container-requests
Lucas Di Pentima
04:37 pm Revision 027f1112 (arvados): Merge branch 'master' into 11580-container-requests
Radhika Chippada
04:36 pm Revision ac958c15 (arvados): 11580: status label display
Radhika Chippada
04:15 pm Revision 06f2f3f3 (arvados): 11580: test the /container_requests page
Radhika Chippada
02:55 pm Bug #11693 (Resolved): [Crunch2] Correctly handle symlinks to input files in output dir
Applied in changeset arvados|commit:f26f70d0a60798065c5f7a5cb91b95587cc9e9ef. Peter Amstutz
02:51 pm Bug #11693: [Crunch2] Correctly handle symlinks to input files in output dir
LGTM, thanks. Lucas Di Pentima
02:41 pm Bug #11693: [Crunch2] Correctly handle symlinks to input files in output dir
Lucas Di Pentima wrote:
> Reviewed commit:a340487a7
> Did a jenkins test run because my local tests kept failing fo...
Peter Amstutz
02:30 pm Bug #11693: [Crunch2] Correctly handle symlinks to input files in output dir
Reviewed commit:a340487a7
Did a jenkins test run because my local tests kept failing for some reason, all ok: https:...
Lucas Di Pentima
02:52 pm Revision f26f70d0 (arvados): Merge branch '11693-crunch2-output-symlinks' closes #11693
Peter Amstutz
02:51 pm Task #11696 (Resolved): Review 11693-crunch2-output-symlinks
Lucas Di Pentima
02:38 pm Revision aceb2d1e (arvados): 11693: Only check collection mounts when determining whether to call
getCollectionManifestForPath to merge files into output collection. Remove
redundant assignment of outputSuffix.
Peter Amstutz
02:35 pm Story #10111 (Resolved): [Workbench][Crunch2] Provenance graph for Container Request
Applied in changeset arvados|commit:f5fbc48810d1397df9e6244c16cf07c05162d36a. Lucas Di Pentima
02:08 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Thanks. This is a much better separation of concerns.
I'm unhappy with how it behaves with large graphs, but inst...
Peter Amstutz
02:32 pm Revision f5fbc488 (arvados): Merge branch '10111-collection-labels'
Closes #10111 Lucas Di Pentima
02:32 pm Revision b9b4502b (arvados): 10111: Merge branch 'master' into 10111-collection-labels
Lucas Di Pentima
01:52 pm Bug #11681 (Resolved): [SDK] Cannot compare newstr() and str()
Peter Amstutz
01:12 pm Revision a340487a (arvados): 11693: Detect and error on symlinks pointing to locations outside th...
Peter Amstutz
12:09 am Revision a4218e78 (arvados): 11580: add container_requests index page
Radhika Chippada
12:07 am Task #11697 (Closed): Review 11580-cr-display-perf
Radhika Chippada

05/15/2017

11:50 pm Revision 1d45ac3d (arvados): 11580: preload container / container_request objects of the parent c...
Radhika Chippada
09:46 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Updates at commit:795bf007c
Test run: https://ci.curoverse.com/job/developer-run-tests/284/
Moved the code relate...
Lucas Di Pentima
09:36 pm Task #11688 (In Progress): Obey MountUUID restrictions in pull list
Tom Clegg
09:36 pm Task #11687 (In Progress): Obey MountUUID restrictions in trash list
Tom Clegg
09:36 pm Task #11689 (Resolved): Accept index request for a specific mount
Tom Clegg
02:15 pm Task #11689 (In Progress): Accept index request for a specific mount
Tom Clegg
09:35 pm Revision 50c67a3e (arvados): 11644: Accept index/pull/trash requests for a specific mount.
Tom Clegg
08:58 pm Revision 94b92f07 (arvados): 11693: Add test that error capturing output results in cancelled con...
Peter Amstutz
08:25 pm Revision eee2c470 (arvados): 11693: Add test cases for symlinks to directories and symlinks insid...
Peter Amstutz
08:13 pm Task #11696 (In Progress): Review 11693-crunch2-output-symlinks
Lucas Di Pentima
07:53 pm Task #11696 (Resolved): Review 11693-crunch2-output-symlinks
Peter Amstutz
08:08 pm Revision 795bf007 (arvados): 10111: Moved the api request related code to the controller.
Lucas Di Pentima
07:52 pm Revision 34f9129a (arvados): 11693: Add tests.
Peter Amstutz
06:06 pm Revision 1903e0e2 (arvados): 11693: Dereference symlinks to input files and add incorporate them ...
Peter Amstutz
02:19 pm Revision bbe86c4a (arvados): 11693: Mark container as cancelled if there is an error during final...
Peter Amstutz
02:14 pm Bug #11693 (Resolved): [Crunch2] Correctly handle symlinks to input files in output dir
Crunch v1 and cwltool allow the output directory to have symlinks to input files, which are then efficiently propagat... Peter Amstutz
02:14 pm Task #11686 (Resolved): Get filesystem UUIDs from linux APIs
Tom Clegg
02:13 pm Task #11685 (Resolved): Report current mounts using UUIDs assigned at runtime
Tom Clegg
02:11 pm Revision 953a9210 (arvados): 11644: Use storage account and bucket name as DeviceID for Azure vol...
Tom Clegg
02:10 pm Revision 209dbf15 (arvados): 11644: Use endpoint and bucket name as DeviceID for S3 volumes.
Tom Clegg
01:14 pm Bug #11590: job reuse / re-use logic does not log decisions
> have 29 candidates after filtering on job state (either state=Complete, or state=Queued/Running and submitted by cu... Radhika Chippada
05:59 am Revision 61a88734 (arvados): 11644: Use filesystem UUID and path as DeviceID for local disk volumes.
Tom Clegg

05/12/2017

09:21 pm Revision 86907851 (arvados): 11644: Add /mounts endpoint using random mount UUIDs assigned at run...
Tom Clegg
06:46 pm Bug #11692 (New): CWL Expression Tool Javascript issue when using str.includes
Again, not sure if this is a bug, and/or if it is an issue with the arvados cwl runner or cwl in general.
When I t...
Sarah Zaranek
06:44 pm Bug #11691 (New): Using cwl, when you obtain files (file objects) from a directory listing they l...
If I get the files from doing a directory.listing - and then try to use file.nameroot on the individual - it gives me... Sarah Zaranek
06:15 pm Story #11580: [Workbench] Improve format of /container_requests index page
If we move the preload stuff out of ApplicationController then instance variables won't be scoped to the current requ... Tom Clegg
01:23 pm Story #11580: [Workbench] Improve format of /container_requests index page
Tom, in the container_work_unit children method, we are creating CR work_unit objects and (if present) the container ... Radhika Chippada
03:13 pm Bug #11669: [Crunch2] crunch-dispatch-slurm hits scancel too much
We could reduce the number of scancel invocations by ~1 per container by calling scancel() immediately the very first... Tom Clegg
02:56 pm Task #11685 (In Progress): Report current mounts using UUIDs assigned at runtime
Tom Clegg
02:52 pm Task #11685 (Resolved): Report current mounts using UUIDs assigned at runtime
Tom Clegg
02:56 pm Task #11689 (Resolved): Accept index request for a specific mount
Tom Clegg
02:55 pm Task #11688 (Resolved): Obey MountUUID restrictions in pull list
Tom Clegg
02:54 pm Task #11687 (Resolved): Obey MountUUID restrictions in trash list
Tom Clegg
02:53 pm Task #11686 (Resolved): Get filesystem UUIDs from linux APIs
Tom Clegg
02:47 pm Task #11683 (Resolved): Add logging to container reuse code
Tom Clegg
02:46 pm Task #11663 (Resolved): Review
Tom Clegg
02:45 pm Revision ed56c267 (arvados): Merge branch '11590-log-reuse'
refs #11590 Tom Clegg
02:34 pm Revision 0d5962b7 (arvados): 11590: Reword log messages.
Tom Clegg
02:33 pm Bug #11590: job reuse / re-use logic does not log decisions
Updated log messages:... Tom Clegg
02:29 pm Revision 2aaef36d (arvados): Merge branch '11647-no-perm-tokens'
Refs #11647 Lucas Di Pentima
02:28 pm Revision 8794708d (arvados): 11647: Merge branch 'master' into 11647-no-perm-tokens
Lucas Di Pentima
02:00 pm Revision c70fbca1 (arvados): 11647: When repacking small blocks into big ones, commit them synchr...
Lucas Di Pentima
01:57 pm Revision 86c4a20b (arvados): 11647: Test proving that when small blocks get repacked and the resu...
committed asynchronously, it doesn't get the permission token on the manifest. Lucas Di Pentima
01:03 pm Revision 3fbba807 (arvados): 10645: ApplicationController.helpers.object_for_dataclass not access...
Radhika Chippada
12:52 am Bug #11682 (New): [API] Deleting container request doesn't cancel containers
Deleting a container request record should have the same effect as setting priority to 0: it should reevaluate the pr... Peter Amstutz

05/11/2017

11:40 pm Bug #11590: job reuse / re-use logic does not log decisions
* reuselog => log_reuse_decision or log_reuse_plan?
* The following seems like it could be an extremely confusing ...
Radhika Chippada
03:41 pm Bug #11590: job reuse / re-use logic does not log decisions
11590-log-reuse @ commit:fb5d279228a7adcafd858d9e137accd8010bb382 Tom Clegg
03:13 pm Bug #11590: job reuse / re-use logic does not log decisions
... Tom Clegg
03:08 pm Bug #11590 (In Progress): job reuse / re-use logic does not log decisions
Example logs... Tom Clegg
09:27 pm Revision 75953fd3 (arvados): Merge branch '11681-str-newstr' refs #11681
Peter Amstutz
09:25 pm Revision 8c9b3ded (arvados): 11681: Return native str type from Collection.portable_data_hash() ...
Peter Amstutz
09:03 pm Bug #11681: [SDK] Cannot compare newstr() and str()
This is the right line of thinking, except... 11681-str-newstr @ commit:9cd1bd27bada41cd13fb9813144fe694a67c2d00 upda... Peter Amstutz
07:38 pm Bug #11681: [SDK] Cannot compare newstr() and str()
11681-str-newstr @ commit:9cd1bd27bada41cd13fb9813144fe694a67c2d00
Tom Clegg
07:02 pm Bug #11681: [SDK] Cannot compare newstr() and str()
On further examination, it looks like Collection.portable_data_hash() is the most likely culprit. Peter Amstutz
06:17 pm Bug #11681: [SDK] Cannot compare newstr() and str()
Possible solutions:
* Fix the SDK to be consistent with old behavior on Python 2
* Fix arvados-cwl-runner to coer...
Peter Amstutz
06:10 pm Bug #11681 (Resolved): [SDK] Cannot compare newstr() and str()
arvados-cwl-runner crashes on Python 2.7 since merging Python 3 compatability:... Peter Amstutz
08:35 pm Revision 93e4548d (arvados): Merge branch '11681-str-newstr'
refs #11681 Tom Clegg
08:33 pm Task #11663 (In Progress): Review
Radhika Chippada
07:37 pm Revision 9cd1bd27 (arvados): 11681: Return native str type from KeepLocator.__str__() and Collect...
Tom Clegg
07:01 pm Task #11633 (Resolved): Review 11626-crunch2-log-slurm-errors
Peter Amstutz
05:57 pm Bug #11669: [Crunch2] crunch-dispatch-slurm hits scancel too much
Sequence is
# crunch-run updates container state to Complete or Cancelled
# crunch-dispatch-slurm waits some amount...
Tom Clegg
02:56 pm Bug #11669: [Crunch2] crunch-dispatch-slurm hits scancel too much
Notes:
We've set up a race condition for ourselves. During normal operation, the last thing that @crunch-run@ doe...
Peter Amstutz
05:00 pm Story #10645 (Resolved): [Workbench] Better presentation of container request "mounts"
Applied in changeset arvados|commit:7f1e7f793bd1b5efa73df51b3070c5acb8fcdc82. Radhika Chippada
04:55 pm Story #10645: [Workbench] Better presentation of container request "mounts"
10645-cr-mounts-display LGTM @ commit:97d759d73e24017c9cab74325d866960389c11a5 Peter Amstutz
01:05 pm Story #10645: [Workbench] Better presentation of container request "mounts"
Branch 10645-cr-mounts-display @ commit:97d759d
> The container request panel seems to be missing a "Log" link.
...
Radhika Chippada
04:57 pm Task #11566 (Resolved): Review 10645-cr-mounts-display
Peter Amstutz
01:14 pm Task #11566 (In Progress): Review 10645-cr-mounts-display
Radhika Chippada
04:55 pm Revision 8cf32f39 (arvados): Merge branch 'master' into 10645-cr-display-perf
Radhika Chippada
04:54 pm Revision 7f1e7f79 (arvados): closes #10645
Merge branch '10645-cr-mounts-display' Radhika Chippada
04:54 pm Revision bc65e5b9 (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada
03:38 pm Revision fb5d2792 (arvados): 11590: Add "reuse jobs even if conflicting outputs exist" and "log r...
Tom Clegg
03:37 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Ok, for large workflows, it still takes forever to load, but it seems that "dot" is the bottleneck now. We need to r... Peter Amstutz
03:30 pm Bug #11680 (New): [Workbench] Provenance graph consolidates scatter steps
A workflow with a wide scatter produces a provenance graph that is basically unusable. The graph rendering needs a w... Peter Amstutz
03:13 pm Story #11580 (In Progress): [Workbench] Improve format of /container_requests index page
Radhika Chippada
03:00 pm Revision 7bb1d40e (arvados): Merge branch '11369-cwl-keepcache-ram' refs #11369
Peter Amstutz
02:34 pm Bug #11679: [Workbench] Logs containers with undefined exit code
arvados-ws sends properties.new_attributes.state but does not send properties.new_attributes.exit_code; see sendObjec... Tom Clegg
02:23 pm Bug #11679 (New): [Workbench] Logs containers with undefined exit code
Workbench logs container state change messages. When a container is complete, it looks at the exit code and reports ... Peter Amstutz
02:25 pm Bug #11626 (Resolved): [Crunch2] Still not propagating slurm errors to user
Applied in changeset arvados|commit:a309f8c5b4842075d6c83f99a8f2a1e1016976f5. Peter Amstutz
02:19 pm Revision a309f8c5 (arvados): Merge branch '11626-crunch2-log-slurm-errors' closes #11626
Peter Amstutz
02:19 pm Revision 3fa8a8a4 (arvados): 11626: Fix log.Printf -> log.Print
Peter Amstutz
12:54 pm Revision 97d759d7 (arvados): 10645: display collection link with disposition. preload mount and c...
Radhika Chippada
12:44 pm Revision 17432c8e (arvados): 10645: link_to locator with disposition
Radhika Chippada
01:10 am Revision a795e5e0 (arvados): Be smarter about how we handle packages built in previous runs.
This commit fixes a bug: packages that were built but failed testing
will no longer be uploaded later, when a subsequ...
Ward Vandewege

05/10/2017

11:53 pm Revision 94054d9f (arvados): Merge branch 'master' into 10645-cr-display-perf
Radhika Chippada
09:47 pm Bug #11669: [Crunch2] crunch-dispatch-slurm hits scancel too much
11669-tidy-scancel-race @ commit:f29a53dab584dcae3e4a38a7623efa6d3b30a663
Tom Clegg
09:44 pm Bug #11669 (In Progress): [Crunch2] crunch-dispatch-slurm hits scancel too much
Moved from #11626
Example:...
Tom Clegg
09:46 pm Revision f29a53da (arvados): 11669: Avoid 0.5 redundant scancel() when ctr disappears from squeue...
Tom Clegg
09:21 pm Bug #11626: [Crunch2] Still not propagating slurm errors to user
I missed that the example logs above are from a real site, not the test suite. So the mystery is why scancel wasn't e... Tom Clegg
08:04 pm Bug #11626: [Crunch2] Still not propagating slurm errors to user
oops, just noticed
log.Printf(text)
should be
log.Print(text)
Other than that, LGTM @ commit:f8675ad473b4...
Tom Clegg
07:56 pm Bug #11626: [Crunch2] Still not propagating slurm errors to user
Peter Amstutz wrote:
> Hhe problem I was trying to solve was that the logs are full of this:
>
> [...]
>
> Thi...
Tom Clegg
09:11 pm Feature #11668 (New): [SDKs] Exceptions raised by Python SDK should include the "error token" if ...
This will make debugging easier in cases where "check the API server logs" activity is involved.
Tom Clegg
09:09 pm Revision 02b75a37 (arvados): The Workbench package has a build-time dependency on tzdata for
Ubuntu1604, which is not included in the default upstream docker image.
Add it to our list of dependencies.
Also twe...
Ward Vandewege
08:21 pm Revision 411109e1 (arvados): Fix Workbench package dependency.
No issue # Ward Vandewege
07:47 pm Task #11667 (Resolved): Review 11580-container-requests
Radhika Chippada
07:45 pm Task #11664 (Resolved): Review
Tom Clegg
07:45 pm Task #11663 (Resolved): Review
Tom Clegg
07:45 pm Task #11661 (Resolved): Review 11644-mounts-api
Tom Clegg
07:31 pm Bug #11470: [API] container_requests#update alternately responds 422 or 404 for no apparent reason
update the clusters with Carlos next weeks and after all updates review the status with him ... there is a pipeline o... Nico César
07:24 pm Bug #11470: [API] container_requests#update alternately responds 422 or 404 for no apparent reason
To be reviewed after Roche upgrade Tom Morris
07:24 pm Task #11640 (Resolved): Review 11638-limit-conns
Radhika Chippada
07:22 pm Story #11308 (Resolved): Support Python3 for arvados-python-client & command line utilities
Packaging moved to #11657 Tom Clegg
07:21 pm Story #11657 (New): [SDKs] publish python3-arvados-python-client packages
Tom Clegg
07:19 pm Task #11379 (Resolved): Review 11308-python3
Tom Clegg
07:17 pm Task #11630 (Resolved): Review 11629-groups-contents-memory
Radhika Chippada
07:17 pm Task #11472 (Resolved): Review 10112-workflow-show
Radhika Chippada
07:15 pm Bug #11629 (Resolved): [API] NoMemoryError in GroupsController#contents
Tom Clegg
02:02 pm Bug #11629 (In Progress): [API] NoMemoryError in GroupsController#contents
https://ci.curoverse.com/job/run-tests-apps-workbench-units/888/console... Tom Clegg
07:11 pm Revision 8b6d154d (arvados): Our API server and Workbench packages depend on 2 gems that compile
native objects, which require g++ to build them: event-machine and
therubyracer. For our debian-based distros, that p...
Ward Vandewege
07:10 pm Bug #9438 (Resolved): keep-balance hung and needed ctrl-c
Lucas Di Pentima
06:58 pm Revision 303d374c (arvados): 11369: Add keep_cache to base RAM request.
Peter Amstutz
06:50 pm Revision 6ad31a32 (arvados): Merge branch '10827-checker-cwltool-update' refs #10827
Peter Amstutz
06:48 pm Revision 209c2259 (arvados): 10827: Bump cwltool & schema salad dependency.
Peter Amstutz
06:09 pm Story #10645: [Workbench] Better presentation of container request "mounts"
There is a bug in the link to the files inside collections: the link is url-encoded, which means the download doesn't... Ward Vandewege
06:08 pm Story #10645: [Workbench] Better presentation of container request "mounts"
The container request panel seems to be missing a "Log" link.
Peter Amstutz
05:00 pm Story #10112 (Resolved): [Crunch2][Workbench]workflow#show page (similar to pipeline_template#sho...
Applied in changeset arvados|commit:a5c53dceed8480e667249fc5fee3349bbaf20ec8. Radhika Chippada
03:46 pm Story #10112: [Crunch2][Workbench]workflow#show page (similar to pipeline_template#show page)
Preload collections @ commit:a7bd5fa Radhika Chippada
01:32 pm Story #10112: [Crunch2][Workbench]workflow#show page (similar to pipeline_template#show page)
Great, this LGTM. Lucas Di Pentima
12:36 am Story #10112: [Crunch2][Workbench]workflow#show page (similar to pipeline_template#show page)
> DockerImageId is not being rendered as a collection link as suggested on note-7, is this possible?
Peter said it...
Radhika Chippada
04:59 pm Revision c7a992a2 (arvados): 10827: Bump cwltool & schema salad dependency.
Peter Amstutz
04:56 pm Revision a5c53dce (arvados): closes #10112
Merge branch '10112-workflow-show' Radhika Chippada
04:55 pm Revision 62a0bbe9 (arvados): Merge branch 'master' into 10112-workflow-show
Radhika Chippada
04:47 pm Revision 776885d3 (arvados): 10645: preload child crs and they containers
Radhika Chippada
04:42 pm Task #11566 (Resolved): Review 10645-cr-mounts-display
Radhika Chippada
04:40 pm Task #11471 (Resolved): Review 11185-wb-disable-reuse
Radhika Chippada
03:45 pm Revision a7bd5fa6 (arvados): 10112: preload collections
Radhika Chippada
03:43 pm Revision ddbe4fd3 (arvados): 10827: Don't track file mutations.
Peter Amstutz
03:31 pm Revision 7162967c (arvados): 10827: Fix imports
Peter Amstutz
03:16 pm Revision 32d28ed5 (arvados): 10827: Bump cwltool & schema salad dependency.
Peter Amstutz
03:08 pm Revision 4644e14f (arvados): 10827: Add --validate, --print-dot and --enable-dev options (pass th...
existing cwltool functionality). Peter Amstutz
02:23 pm Revision c22bd855 (arvados): Merge branch '11629-groups-contents-memory'
refs #11629
refs #11458
Tom Clegg
02:22 pm Revision 96b33e5a (arvados): 11629: Fix ambiguous column reference.
Tom Clegg
06:19 am Bug #11652 (New): [Workbench] Search should search subprojects as well as selected project - to b...
Customer request:
Is it possible to change behaviour of the search in the following way: selection of project will...
Tom Morris
06:16 am Bug #4921: [Workbench] In search dialog, the preview panel on right should get cleared when the s...
Still a bug Tom Morris
06:10 am Bug #10706 (Duplicate): PostgreSQL error when searching
Tom Morris

05/09/2017

09:02 pm Story #10112: [Crunch2][Workbench]workflow#show page (similar to pipeline_template#show page)
A couple comments:
* @DockerImageId@ is not being rendered as a collection link as suggested on note-7, is this po...
Lucas Di Pentima
03:37 pm Story #10112: [Crunch2][Workbench]workflow#show page (similar to pipeline_template#show page)
Branch 10112-workflow-show @ commit:4c1d34e
Using the partial show_text_with_locators (from #10645), displays base...
Radhika Chippada
08:45 pm Bug #11634 (Resolved): [Tests] Rails install fails when bundle specifies a gem included in system...
Applied in changeset arvados|commit:9456885954b0bff02721bbb14da296be212efdc4. Tom Clegg
08:43 pm Bug #11626: [Crunch2] Still not propagating slurm errors to user
However I've rebased out the HasUUID changes so we can discuss it separately.
Now at @ commit:f8675ad473b45387b128...
Peter Amstutz
08:37 pm Revision 94568859 (arvados): Merge branch '11634-bundler-cache'
closes #11634 Tom Clegg
08:35 pm Bug #11629 (Resolved): [API] NoMemoryError in GroupsController#contents
Applied in changeset arvados|commit:ba917d72d48615cdd0c6da87d41b6bd0f9f26666. Tom Clegg
04:41 pm Bug #11629: [API] NoMemoryError in GroupsController#contents
> These updates also revealed that the changes had broken items_available in the groups#contents response by exiting ... Radhika Chippada
08:33 pm Revision ba917d72 (arvados): Merge branch '11629-groups-contents-memory'
closes #11629 Tom Clegg
08:33 pm Revision f8675ad4 (arvados): 11626: Log sbatch errors where the user can see them. Fix tests so ...
Peter Amstutz
08:15 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
11308-collection-keys LGTM, tested with arvados-cwl-runner Peter Amstutz
07:32 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
11308-collection-keys @ commit:47f29d75759db7e4d81203e110257f1a471f82e7
Tom Clegg
07:22 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
API incompatibility:
Collection.keys(), Collection.values() and Collection.items() return lists in Python 2.7. Ho...
Peter Amstutz
08:08 pm Revision 40ce7856 (arvados): Add libgnutls28-dev and python3-dev to install package list to satisfy
run-tests.sh dependencies. refs #11308 Peter Amstutz
07:49 pm Revision 4bc2f36b (arvados): Merge branch '11308-collection-keys'
refs #11308 Tom Clegg
07:31 pm Revision 47f29d75 (arvados): 11308: Fix modes not tested in test case.
Tom Clegg
07:30 pm Revision 8e0303a5 (arvados): 11308: collection.keys() supports indexing for python2 callers.
Tom Clegg
07:12 pm Story #7475: [Node manager] Better communication when job is unsatisfiable
For crunch2, when node manager is not in use, sbatch rejects unsatisfiable jobs and the user gets an error -- however... Tom Clegg
05:46 pm Task #11472 (In Progress): Review 10112-workflow-show
Lucas Di Pentima
05:43 pm Story #8304 (Resolved): [Crunch2] [Spike] SLURM plug-in to boot Arvados compute nodes in a cloud
Peter Amstutz
05:43 pm Story #8312 (Resolved): [Crunch2] [Spike] Prototype using SLURM's linear scheduler, node sharing
Peter Amstutz
05:22 pm Task #11478 (Resolved): Review 8019-crunchrun-log-throttle
Peter Amstutz
03:38 pm Feature #11646 (New): [keepstore] Support X-Keep-Desired-Tier header
Part of the [[Keep storage tiers]] story.
Accept X-Keep-Desired-Tier header in write requests.
* If X-Keep-Desire...
Tom Clegg
03:38 pm Feature #11645 (New): [keepstore] Add "tier" field to volume config
Part of the [[Keep storage tiers]] story.
Accept a "Tier" int for each configured volume.
* If config is zero, ne...
Tom Clegg
03:29 pm Revision 4c1d34ee (arvados): 10112: use show_text_with_locators partial to display the various at...
Radhika Chippada
03:19 pm Feature #11644 (Resolved): [keepstore] mount-oriented APIs
Keepstore's role in the management part of the [[Keep storage tiers]] story.
* A "mounts" request (@GET /mounts@) sh...
Tom Clegg
02:35 pm Revision 79aeb557 (arvados): Merge branch 'master' into 10112-workflow-show
Radhika Chippada
02:31 pm Revision 8ee2f839 (arvados): refs #10645
Merge branch '10645-cr-mounts-display' Radhika Chippada
02:28 pm Revision 537ff1df (arvados): 10645: added comment
Radhika Chippada
02:26 pm Story #10645: [Workbench] Better presentation of container request "mounts"
Looks great @ commit:2c60e15078452202ff58eac99d44920448fb7b9a Please merge. Peter Amstutz
11:35 am Revision 4e32a5d0 (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada
11:33 am Revision 2c60e150 (arvados): 10645: refactor the display of text with locators into a partial
Radhika Chippada
01:58 am Revision dd69c4ba (arvados): 10645: collection links in command
Radhika Chippada
01:34 am Bug #11519: arv-get should abort on ctrl/C
I haven't been able to reproduce the problem using a local virtualenv and PySDK at @master@.
Nevertheless, I've fo...
Lucas Di Pentima
01:21 am Revision 575b2f04 (arvados): 10645: regexp for keep_locator_in_json format
Radhika Chippada

05/08/2017

10:59 pm Story #10645: [Workbench] Better presentation of container request "mounts"
@ commit:2c60e15078452202ff58eac99d44920448fb7b9a
Addressed comments from note 21.
To test: container_requests/...
Radhika Chippada
07:41 pm Story #10645: [Workbench] Better presentation of container request "mounts"
Example HTML linking PDH and file separately:... Peter Amstutz
07:22 pm Story #10645: [Workbench] Better presentation of container request "mounts"
I pushed a couple of layout tweaks. Now @ commit:75a76c083c8f257c33ad31582aac7a86bc866bb5
Trying to get this fini...
Peter Amstutz
06:01 pm Story #10645: [Workbench] Better presentation of container request "mounts"
@ commit:aa6a63f
Addressed all the comments from the note 19.
Radhika Chippada
02:16 pm Story #10645: [Workbench] Better presentation of container request "mounts"
Using qr1hi-dz642-uypqs2exiy7so7l as my test case.
cwl.input.json display feels cramped. Suggest making it taller,...
Peter Amstutz
10:56 pm Revision 5a649613 (arvados): 10645: test update
Radhika Chippada
10:43 pm Revision e49882e5 (arvados): 10645: display PDHs in command as clickable collections.
Radhika Chippada
09:34 pm Revision 78408743 (arvados): 10645: display keep locator in mounts as pdh/file_name
Radhika Chippada
08:10 pm Bug #11638 (Resolved): [arvados-ws] Label and limit database connections
Applied in changeset arvados|commit:08e9c6aff46e182b9cd14fab9d42d21b2d7c6902. Tom Clegg
07:48 pm Bug #11638: [arvados-ws] Label and limit database connections
Just a small detail: Would it be useful if a warning message is logged when the PostgresPool is set to <= 0?
The res...
Lucas Di Pentima
07:04 pm Bug #11638: [arvados-ws] Label and limit database connections
11638-limit-conns @ commit:3652f3ed82e994c9eea74ba1afb4aa3c11ebd697
Tom Clegg
06:08 pm Bug #11638 (In Progress): [arvados-ws] Label and limit database connections
Tom Clegg
06:08 pm Bug #11638 (Resolved): [arvados-ws] Label and limit database connections
arvados-ws appears to open too many Postgres connections, eventually reaching the Postgres server limit and preventin... Tom Clegg
08:04 pm Revision 08e9c6af (arvados): Merge branch '11638-limit-conns'
closes #11638 Tom Clegg
08:04 pm Revision c4a0f7a7 (arvados): 11638: Log a warning if Postgres pool size is unlimited.
Tom Clegg
07:55 pm Bug #11626: [Crunch2] Still not propagating slurm errors to user
I'm not sure why the HasUUID() check is added in run() in crunch-dispatch-slurm.go. ctx.Done() fires as soon as HasUU... Tom Clegg
01:25 pm Bug #11626 (In Progress): [Crunch2] Still not propagating slurm errors to user
Peter Amstutz
07:14 pm Revision 75a76c08 (arvados): 10645: Tweak so that left column is col-md-6 (instead of col-md-5). ...
full width part to be col-md-12. Move "priority" back to right hand column. Peter Amstutz
07:12 pm Task #11633 (In Progress): Review 11626-crunch2-log-slurm-errors
Tom Clegg
01:26 pm Task #11633 (Resolved): Review 11626-crunch2-log-slurm-errors
Peter Amstutz
07:09 pm Task #11640 (In Progress): Review 11638-limit-conns
Lucas Di Pentima
07:08 pm Task #11640 (Resolved): Review 11638-limit-conns
Lucas Di Pentima
07:02 pm Revision 3652f3ed (arvados): 11638: Add PostgresPool config option, and default application_name
Tom Clegg
06:44 pm Bug #11519 (In Progress): arv-get should abort on ctrl/C
Lucas Di Pentima
06:27 pm Task #9562 (Resolved): Review 9438-http-default-timeout
Lucas Di Pentima
06:27 pm Bug #11629: [API] NoMemoryError in GroupsController#contents
Yes to both, thanks. These updates also revealed that the changes had broken items_available in the groups#contents r... Tom Clegg
04:28 pm Bug #11629: [API] NoMemoryError in GroupsController#contents
* Do we also want to add self.limit_index_columns_read on "components" column for jobs and pipeline_instances?
* I...
Radhika Chippada
06:23 pm Story #9587: [Workbench] Interface to list and untrash trashed collections
Branch @9587-wb-trash-tab@ pushed so Radhika can continue from there.
The parameter @include_trash@ is on the collec...
Lucas Di Pentima
06:21 pm Revision 4cacfe7a (arvados): 11629: Apply memory limits to job/pipeline "components" fields.
Tom Clegg
06:19 pm Revision 541d4760 (arvados): 9587: Trash tab added to projects controller.
Lucas Di Pentima
06:17 pm Revision 3f285d4c (arvados): 11629: Fix items_available in groups#contents response.
Tom Clegg
05:59 pm Revision aa6a63fb (arvados): 10645: L&F improvements
Radhika Chippada
04:42 pm Revision ee434dea (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada
04:27 pm Task #11630 (In Progress): Review 11629-groups-contents-memory
Radhika Chippada
04:06 pm Revision 9bf79049 (arvados): Merge branch '11308-python3'
refs #11308 Tom Clegg
04:05 pm Revision 8976780c (arvados): 11308: Merge branch 'master' into 11308-python3
Tom Clegg
03:27 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Updates at commit:b29ca38e4
Test run: https://ci.curoverse.com/job/developer-run-tests/278/
* Refactored the grap...
Lucas Di Pentima
02:52 pm Revision b29ca38e (arvados): 10111: When dealing with output collection nodes, render the specifi...
Lucas Di Pentima
02:31 pm Revision b20770d7 (arvados): 10111: Refactored the graph creation code to minimize the amount of ...
Lucas Di Pentima
02:16 pm Bug #11634 (In Progress): [Tests] Rails install fails when bundle specifies a gem included in sys...
Tom Clegg
02:16 pm Bug #11634: [Tests] Rails install fails when bundle specifies a gem included in system ruby
11634-bundler-cache @ commit:f21a495d7723e72ea61c62293d56aff1a65c130c
Tom Clegg
01:51 pm Bug #11634 (Resolved): [Tests] Rails install fails when bundle specifies a gem included in system...
If a system gem is used to satisfy a Gemfile dependency, Bundler tries to update the system gem cache:... Tom Clegg
02:07 pm Revision f21a495d (arvados): 11634: Prevent bundler from trying to use/update system gem cache.
Tom Clegg
01:59 pm Revision 3b11d2b5 (arvados): 11634: Fix whitespace.
Tom Clegg
01:52 pm Task #11566 (In Progress): Review 10645-cr-mounts-display
Peter Amstutz
01:18 pm Revision b58953e9 (arvados): 10645: improve location regex usage
Radhika Chippada

05/05/2017

09:11 pm Revision 26d7e180 (arvados): 11626: Don't run scancel if the job is already gone from the queue. ...
Peter Amstutz
07:35 pm Revision d4eead35 (arvados): 10645: table width 100%
Radhika Chippada
07:32 pm Bug #11629: [API] NoMemoryError in GroupsController#contents
11629-groups-contents-memory @ commit:cb109bfddd08bd8136b75e90b681e4af3d60ea30
Tom Clegg
06:13 pm Bug #11629 (In Progress): [API] NoMemoryError in GroupsController#contents
Tom Clegg
05:17 pm Bug #11629 (Resolved): [API] NoMemoryError in GroupsController#contents
Presumably, we are seeing this error because (unlike collections#index) the groups#contents controller does not respe... Tom Clegg
07:32 pm Task #11630 (Resolved): Review 11629-groups-contents-memory
Tom Clegg
07:30 pm Revision cb109bfd (arvados): 11629: Limit database reads for all list responses, not just index.
Tom Clegg
07:30 pm Revision 74d98b5f (arvados): 10112: add Definition tab to workflow page
Radhika Chippada
05:38 pm Revision 5cd02331 (arvados): Merge branch 'master' into 10112-workflow-show
Radhika Chippada
05:35 pm Story #10645: [Workbench] Better presentation of container request "mounts"
> Please remove the borders around the values of "runtime_constraints", "command" and "input_mounts". Instead, can we... Radhika Chippada
05:30 pm Revision 7ea93a3e (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada
05:29 pm Revision 50a1f829 (arvados): 10645: corrections to test fixture
Radhika Chippada
04:10 pm Feature #8019 (Resolved): [Crunch2] Throttle logs
Applied in changeset arvados|commit:3b4325c210516d1f61838fb26e06e0a11c31ce6d. Radhika Chippada
04:05 pm Revision 3b4325c2 (arvados): closes #8019
Merge branch '8019-crunchrun-log-throttle' Radhika Chippada
04:02 pm Revision 007d2919 (arvados): Merge branch 'master' into 8019-crunchrun-log-throttle
Radhika Chippada
03:54 pm Revision f389a552 (arvados): 8019: Make flush channel buffered again (because we want to be able ...
Fix "> crunchLogBytesPerEvent" to ">= crunchLogBytesPerEvent" for consistency. Peter Amstutz
03:32 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Another note. For labeling, if there are multiple collections but they have the same name, you don't need the "+N mo... Peter Amstutz
03:15 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
How hard would it be to fix the hyperlinks so that when you have a specific UUID associated with a collection, clicki... Peter Amstutz
02:47 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Updates at commit:4259263d2
Test run: https://ci.curoverse.com/job/developer-run-tests/276/
Addressed issue about...
Lucas Di Pentima
02:08 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Additional comment: where we have a container request with an explicit output_uuid, make sure to use the label corres... Peter Amstutz
01:51 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
An integration test was failing, updated fix at commit:e01823785
New test run: https://ci.curoverse.com/job/develope...
Lucas Di Pentima
03:12 pm Revision d23bc7c1 (arvados): 10645: small style updates
Radhika Chippada
03:06 pm Task #11628 (Resolved): Rwview 10111-collection-labels
Lucas Di Pentima
03:05 pm Bug #11627 (New): [node manager] not assigning weights
Node manager is supposed to assign a weight to compute nodes when they come up. The weight controls scheduling prefe... Peter Amstutz
03:05 pm Task #11619 (Resolved): Review 10111-cr-prov-regression-fixes
Lucas Di Pentima
02:46 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
11308-python3 @ commit:ffb304afad93f50fd6ee43ecda6584dd0ac000c4 https://ci.curoverse.com/job/developer-run-tests/272/
Tom Clegg
02:43 pm Revision 4259263d (arvados): 10111: On output collections, name the collection node after the out...
Lucas Di Pentima
01:49 pm Revision e0182378 (arvados): 10111: Fix integration test
Lucas Di Pentima
01:44 pm Revision 1bfda34f (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada
12:27 pm Bug #11626 (Resolved): [Crunch2] Still not propagating slurm errors to user
If a user submits a job and slurm rejects it, the slurm error should be propagated to the container log so the user c... Peter Amstutz
12:24 pm Story #8000 (Resolved): [Node Manager] Shut down nodes in SLURM 'down' state
This was fixed #8953 with the addition of an explicit state transition table. Peter Amstutz
12:02 pm Bug #11625 (New): [Workbench] Ensure that running processes are visible on the dashboard
The left column of the dashboard used to have two panels, one for "running" processes and one for "recently completed... Peter Amstutz
01:32 am Revision 4218f742 (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada
01:31 am Revision 2516e999 (arvados): 10645: mount inputs display
Radhika Chippada

05/04/2017

09:40 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Branch @10111-collection-labels@ - commit commit:39755f764
Test run: https://ci.curoverse.com/job/developer-run-test...
Lucas Di Pentima
05:37 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
Fixes at commit:a4a8d41f6 - branch @10111-cr-prov-regression-fixes@
Test run: https://ci.curoverse.com/job/developer...
Lucas Di Pentima
02:58 pm Story #10111: [Workbench][Crunch2] Provenance graph for Container Request
In the interests of time, let's limit it to these changes:
* Needs to use PDH so that inputs and output match up. ...
Peter Amstutz
02:28 pm Story #10111 (Feedback): [Workbench][Crunch2] Provenance graph for Container Request
Additional comments:
* Needs to use PDH so that inputs and output match up. For example, in this graph the output...
Peter Amstutz
09:37 pm Revision 39755f76 (arvados): 10111: Better labels on collection on the provenance graph.
Lucas Di Pentima
06:46 pm Bug #11621 (New): Example CWL code for cwl-runner doesn't work
when I run
arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml
I get the following:
017-05-04 18:38:57 cwltool INF...
Sarah Zaranek
06:30 pm Revision 6fd54bd2 (arvados): Merge branch '10111-cr-prov-regression-fixes'
Refs #10111 Lucas Di Pentima
06:26 pm Revision e42bf966 (arvados): 10111: Flipped arrow between the cr and its childs cr.
Lucas Di Pentima
06:05 pm Story #10645: [Workbench] Better presentation of container request "mounts"
While we're at it, can we look at "requesting_container_uuid" and provide a link to the container and/or a link to th... Peter Amstutz
02:44 pm Story #10645: [Workbench] Better presentation of container request "mounts"
* Please remove the borders around the values of "runtime_constraints", "command" and "input_mounts". Instead, can w... Peter Amstutz
06:02 pm Bug #11557: [CWL] arvados-cwl-runner with job reuse doesn't put the jobs/output collections in th...
Peter Amstutz wrote:
> if the job is not in the target project, arvados-cwl-runner creates a can_read link from the ...
Tom Clegg
05:37 pm Task #11619 (Resolved): Review 10111-cr-prov-regression-fixes
Lucas Di Pentima
05:32 pm Revision e4f03346 (arvados): 8019: uncomment the test statement
Radhika Chippada
05:17 pm Revision a4a8d41f (arvados): 10111: Only add cr input collections by PDH.
Lucas Di Pentima
05:02 pm Revision f10431fe (arvados): 8019: Use tl.stopping channel instead of tl.flush to signal Close. U...
Tom Clegg
03:42 pm Revision 11208178 (arvados): 10111: Skip log outputs from the provenance graph.
Container request's outputs are now added to the graph by their PDHs so that they connect with the CR childs inputs. Lucas Di Pentima
03:16 pm Revision ffb304af (arvados): 11308: Merge branch 'master' into 11308-python3
Tom Clegg
03:09 pm Revision 5ca261ee (arvados): 11308: Fix bytes vs. strings from recent merge.
Tom Clegg
01:01 am Revision dd0786de (arvados): We stopped building packages for Ubuntu 12.04, it is now officially ...
No issue # Ward Vandewege

05/03/2017

10:00 pm Bug #11522 (Resolved): Remove storage report page until it is fixed
Applied in changeset arvados|commit:8ed7b6dd5d4df93a3f37096afe6d6f81c2a7ef6e. Radhika Chippada
08:26 pm Bug #11522 (In Progress): Remove storage report page until it is fixed
Radhika Chippada
09:56 pm Revision 8ed7b6dd (arvados): closes #11522
Merge branch '11522-remove-user-storage-report-page' Radhika Chippada
09:08 pm Revision d66c9f93 (arvados): Merge branch 'master' into 10112-workflow-show
Radhika Chippada
09:00 pm Story #11250 (Resolved): [Workbench] Better user-facing terms for container states
Applied in changeset arvados|commit:f682de562d9c857616f81d48d5cd915748725197. Radhika Chippada
08:43 pm Story #11250: [Workbench] Better user-facing terms for container states
LGTM! Lucas Di Pentima
08:23 pm Story #11250: [Workbench] Better user-facing terms for container states
Branch 11250-container-state-display ready for review @ commit:8dd7e323798d36e3182c12868c6f1f3dd0eabf96
Tests @ ht...
Radhika Chippada
08:18 pm Story #11250 (In Progress): [Workbench] Better user-facing terms for container states
Radhika Chippada
08:56 pm Task #11568 (Resolved): Review
Radhika Chippada
08:55 pm Revision f682de56 (arvados): closes #11250
Merge branch '11250-container-state-display' Radhika Chippada
08:44 pm Task #11565 (Resolved): Review branch 11250-container-state-display
Lucas Di Pentima
08:30 pm Task #11565 (In Progress): Review branch 11250-container-state-display
Lucas Di Pentima
08:37 pm Revision 787e77b3 (arvados): 11522: remove link to users storage page in users activity page.
Radhika Chippada
08:20 pm Revision 8dd7e323 (arvados): 11250: when a container's priority is zero and in Queued state, show...
Radhika Chippada
06:37 pm Revision c3aa32a8 (arvados): 10645: rename file
Radhika Chippada
06:24 pm Revision 8de338d1 (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada
06:20 pm Revision 3c4b14cf (arvados): Merge branch '11369-log-sizes' refs #11369
Peter Amstutz
06:15 pm Revision 1586823b (arvados): Merge branch 'master' into 10112-workflow-show
Radhika Chippada
05:55 pm Feature #8019: [Crunch2] Throttle logs
Error log:... Radhika Chippada
04:59 pm Revision c64d7e1d (arvados): Merge branch '11469-volume-tmp' refs #11469
Peter Amstutz
04:48 pm Revision 1885991d (arvados): Merge branch '11369-dispatch-mem' refs #11369
Peter Amstutz
04:48 pm Revision 00175738 (arvados): Merge branch '11369-cwl-crunch2-capacity' refs #11369
Peter Amstutz
04:03 pm Story #10112: [Crunch2][Workbench]workflow#show page (similar to pipeline_template#show page)
Showing these would be helpful:
baseCommand: [bwa, mem]
arguments: - {prefix: -t, valueFrom: $(runtime.cores)} - ...
Bryan Cosca
02:49 pm Revision 65616466 (arvados): 11469: Add test for changed tmp mounting behavior.
Peter Amstutz
02:25 pm Bug #11606 (Resolved): [CWL] [API] empty array submitted as null
Applied in changeset arvados|commit:e24a892b920fd8cddf091a3c15acba1ab27f510d. Tom Clegg
02:20 pm Revision e24a892b (arvados): Merge branch '11606-rails4-no-munge'
closes #11606 Tom Clegg
02:20 pm Revision fa0ccc4c (arvados): 11308: Stop test suite if api etc. cannot be started.
Tom Clegg
01:52 pm Revision 35dde402 (arvados): 11469: All {kind: tmp} mounts set up under host tempdir, don't try t...
Docker volumes. Peter Amstutz
01:41 pm Revision 3f8f2a9b (arvados): 11369: Fix tests. Tweak test fixture to include a mounts entry with...
Peter Amstutz
01:35 pm Revision fe29b56a (arvados): 11369: Log node sizes at startup. Fix setting of scratch disk size.
Peter Amstutz
12:34 pm Story #10645: [Workbench] Better presentation of container request "mounts"
Branch 10645-cr-mounts-display @ commit:c3aa32a Radhika Chippada
04:24 am Revision 43a6204d (arvados): 10645: add test for container_request display with input mounts
Radhika Chippada
03:17 am Revision 7f9839be (arvados): 11369: Containers API: Add capacity to tmp and output mounts. Also ...
Peter Amstutz
03:13 am Revision 2bfa9d38 (arvados): 11369: Containers API: Add capacity to tmp and output mounts. Also ...
Peter Amstutz
02:53 am Revision 10441f4b (arvados): 11369: Fix tests. Tweak test fixture to include a mounts entry with...
Peter Amstutz
02:21 am Revision 9350a056 (arvados): 11369: RAM request includes keep cache RAM. Add request for temp disk.
Peter Amstutz
01:59 am Revision dd908b53 (arvados): 11369: Log node sizes at startup. Fix setting of scratch disk size.
Peter Amstutz
01:05 am Revision 23825c0b (arvados): Merge branch 'master' into 10645-cr-mounts-display
Radhika Chippada

05/02/2017

08:40 pm Bug #11609 (New): [Workbench] A cancelled container request page should show the "Re-run" button
Right now the Re-run button appears when a given container request's state is 'Final'.
Cancelled CRs (@state == 'Com...
Lucas Di Pentima
08:10 pm Feature #8019: [Crunch2] Throttle logs
At commit:e54bce82
Two tests "TestCrunchStat" and "TestNodeInfoLog" were failing. This was due to rate limiting al...
Radhika Chippada
08:03 pm Revision e54bce82 (arvados): 8019: need to check and write log if things are closing down
Radhika Chippada
06:57 pm Bug #11557: [CWL] arvados-cwl-runner with job reuse doesn't put the jobs/output collections in th...
# if the output and log are not in the target project, arvados-cwl-runner copies the collection into the target proje... Peter Amstutz
06:46 pm Bug #5076: [Workbench] Replace schizophrenic generic "groups" page with "roles" page that only cr...
Estimates:
* 0.5 points for a "roles" page that only lists groups with group_class="role", and has a "create new rol...
Tom Clegg
06:25 pm Bug #11590: job reuse / re-use logic does not log decisions
For crunch1 and crunch2: add a config flag "log_reuse_decisions". If true, in the "find reusable" code, build the Act... Tom Clegg
05:47 pm Bug #11606: [CWL] [API] empty array submitted as null
LGTM. Peter Amstutz
03:41 pm Bug #11606: [CWL] [API] empty array submitted as null
Our monkeypatch stopped working, and our test case wasn't running because it wasn't *_test.rb. Rails4 has a builtin f... Tom Clegg
03:09 pm Bug #11606: [CWL] [API] empty array submitted as null
Despite the test case added in #10935, commit:b07d8a573c74e7be56a2fd8c5064032c37d90c67, [] is getting munged to null ... Tom Clegg
02:39 pm Bug #11606 (Resolved): [CWL] [API] empty array submitted as null
https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-9xb0s71vn96wfjc#Log... Tom Clegg
05:42 pm Task #11607 (Resolved): Review 11606-rails4-no-munge
Peter Amstutz
05:34 pm Task #11607 (In Progress): Review 11606-rails4-no-munge
Peter Amstutz
03:48 pm Task #11607 (Resolved): Review 11606-rails4-no-munge
Tom Clegg
05:32 pm Task #11589 (Resolved): Review 11579-arvput-follow-symlinks
Peter Amstutz
05:02 pm Revision 607fe087 (arvados): 11308: Merge branch 'master' into 11308-python3
Conflicts:
sdk/python/arvados/arvfile.py
sdk/python/arvados/commands/put.py
sdk/python/tests/test_collections.py
Tom Clegg
04:50 pm Story #9587 (In Progress): [Workbench] Interface to list and untrash trashed collections
Lucas Di Pentima
03:50 pm Story #11185 (Resolved): [Workbench] Add ability to disable job reuse when running a workflow
Applied in changeset arvados|commit:fca805a18c671ccbb03cef640c15172d1f02ffe3. Lucas Di Pentima
03:44 pm Revision fca805a1 (arvados): Merge branch '11185-wb-disable-reuse'
Closes #11185 Lucas Di Pentima
03:39 pm Revision 974799e0 (arvados): 11606: Disable deep_munge behavior. Rename test so it runs in test s...
Tom Clegg
03:30 pm Bug #11579 (Resolved): arv-put doesn't traverse symlinked directories
Applied in changeset arvados|commit:613155d587da60dbe04c7635649b1f3694938adc. Lucas Di Pentima
02:56 pm Bug #11579: arv-put doesn't traverse symlinked directories
Updated code & merged master at commit:fe45b1b66c730f2546d78a7899375707c0816518
Running tests before merging to mast...
Lucas Di Pentima
01:47 pm Bug #11579: arv-put doesn't traverse symlinked directories
11579-arvput-follow-symlinks @ 4d012b23a4ac88f433986054fc0085ee6714b5b3
The "file not found" message is still prin...
Peter Amstutz
03:24 pm Revision 613155d5 (arvados): Merge branch '11579-arvput-follow-symlinks'
Closes #11579 Lucas Di Pentima
02:56 pm Revision be4852ec (arvados): 11185: Merge branch 'master' into 11185-wb-disable-reuse
Lucas Di Pentima
02:55 pm Revision fe45b1b6 (arvados): 11579: Merge branch 'master' into 11579-arvput-follow-symlinks
Lucas Di Pentima
02:52 pm Revision 18f3b51a (arvados): 11579: Added assertion on test to check for file symlinks to be copi...
Don't treat a PathDoesNotExist exception as an abnormal termination, just log the proper error message, and quit. Lucas Di Pentima
02:32 pm Story #11580: [Workbench] Improve format of /container_requests index page
* Display the columns show icon, name, description, status, created_at (sort by this column).
* Optionally, includ...
Radhika Chippada
01:55 pm Feature #11605 (New): arv-put detects symlinks to common files and directories to avoid duplicate...
When uploading a set of files, if some subdirectories or files are referenced more than once by using symbolic links,... Lucas Di Pentima
01:43 am Revision 6f62739b (arvados): 10645: container request mounts display
Radhika Chippada

05/01/2017

09:12 pm Bug #11579: arv-put doesn't traverse symlinked directories
Updates at commit:4d012b23a
Test run: https://ci.curoverse.com/job/developer-run-tests/266/
As discussed on the c...
Lucas Di Pentima
09:05 pm Revision 4d012b23 (arvados): 11579: Removed feature about not uploading a symlinked dir twice.
When using --no-follow-links parameter, symlinked files will be also ignored.
Updated tests.
Lucas Di Pentima
08:31 pm Story #10645: [Workbench] Better presentation of container request "mounts"
Radhika Chippada wrote:
> Added a mockup of the container request page with the additional items. Please see cr-moun...
Peter Amstutz
06:26 pm Story #10645: [Workbench] Better presentation of container request "mounts"
Added a mockup of the container request page with the additional items. Please see cr-mounts-display.png Radhika Chippada
08:03 pm Story #11185: [Workbench] Add ability to disable job reuse when running a workflow
Updates at commit:c6452ff3c
Test run: https://ci.curoverse.com/job/developer-run-tests/265/
* Kept JS code inside...
Lucas Di Pentima
04:28 pm Story #11185: [Workbench] Add ability to disable job reuse when running a workflow
* It appears that you might be able to change the cmd for the completed-old CR fixture rather than adding another "co... Radhika Chippada
07:58 pm Revision c6452ff3 (arvados): 11185: Removed newly added completed-acr fixture and modified the al...
to be able to use it on the tests.
Change the reset_form() JS function to something more specific to this usage, to a...
Lucas Di Pentima
07:52 pm Task #11482 (Resolved): Review
Tom Clegg
06:10 pm Feature #10988 (Resolved): [DOC] update documentation to list ubuntu1604
Applied in changeset arvados|commit:ed375727b95e6f1cd56c599a177bbd7979e1cf78. Tom Clegg
06:03 pm Revision ed375727 (arvados): Merge branch '10988-document-ubuntu1604-support'
closes #10988 Tom Clegg
06:02 pm Revision 70166077 (arvados): 10988: Remove apt.arvados.org repo priority: we no longer publish/re...
Tom Clegg
06:02 pm Revision fb72bf49 (arvados): 10988: Document our Ubuntu 16.04 support.
Ward Vandewege
05:29 pm Revision 5d00ecb0 (arvados): Merge branch 'master' into origin-8019-crunchrun-log-throttle
Peter Amstutz
04:44 pm Revision d2ebb8fe (arvados): Merge branch '11469-volume-tmp' refs #11469
Peter Amstutz
04:43 pm Revision fea486f9 (arvados): Crunch script doesn't fail in file/directory literals that don't hav...
Peter Amstutz
02:36 pm Revision 8266a4cc (arvados): 11549: Crunch script doesn't fail in file/directory literals that do...
Peter Amstutz
02:02 pm Revision 43d1a5d0 (arvados): 11469: Docker-managed volumes go in "Volumes" not "Binds".
Peter Amstutz
01:32 pm Bug #11594 (New): [API] [Wrokbench] set a default config.log_level
log_level isn't set by default, triggering the following deprecation warning: ... Nico César
01:30 pm Bug #11593 (New): [API] [Workbench] config option serve_static_assets to serve_static_files
After #7709 we start having a deprecation warning:... Nico César
 

Also available in: Atom