Activity

From 10/26/2017 to 11/24/2017

Today

06:10 am Revision b3d29244 (arvados): 12627: Any token can permit scope, if acting as the right user.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

11/23/2017

09:22 pm Task #12583 (In Progress): Review 12475-asyncbuf
Lucas Di Pentima
09:03 pm Task #12656 (In Progress): Review 12018-tool-docs
Lucas Di Pentima
09:02 pm Feature #12018: Synchronize group membership with external data source
Added documentation at commit:4b16a16d0 - branch @12018-tool-docs@
Lucas Di Pentima
08:53 pm Revision 4b16a16d (arvados): 12018: Added documentation for arv-sync-groups
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
07:25 pm Story #12519 (Resolved): Multi-cluster project search
Applied in changeset arvados|commit:3737e05681b6cfb22ea0af0da08598e458da16f0. Anonymous
07:19 pm Story #12519: Multi-cluster project search
Updated as suggested at commit:90a8c9cd8 Lucas Di Pentima
05:53 pm Story #12519: Multi-cluster project search
LGTM
one more optimization/simplification though:...
Tom Clegg
07:22 pm Revision 3737e056 (arvados): Merge branch '12519-multisite-project-search'
Closes #12519
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
07:20 pm Revision 742cbf87 (arvados): 12627: Use first reader_token as primary token, if no primary given.
Workbench relies on this for anonymous access.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
07:13 pm Revision 90a8c9cd (arvados): 12519: Code simplification
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
07:10 pm Revision 1d074dfe (arvados): 12627: Use first reader_token as primary token, if no primary given.
Workbench relies on this for anonymous access.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
06:55 pm Revision be02efee (arvados): 12627: Use first reader_token as primary token, if no primary given.
Workbench relies on this for anonymous access.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
06:35 pm Revision b81e8380 (arvados): Merge branch '12627-token-scope'
fixes #12627
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
03:13 pm Revision 08fe6b07 (arvados): 12475: Faster asyncbuf.
Unlock earlier so multiple readers can copy bytes concurrently.
100 readers: improves from 4.5 GB/s to 17 GB/s
1 rea...
Tom Clegg
02:52 pm Revision 1cb2f6a1 (arvados): 12389: new build for Roche including changes from master up to 197d9...
Arvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat@veritasgenetics.com> Fernando Monserrat
02:37 pm Revision ad7649c3 (arvados): refs 12389: Merge branch '12513-make-dashboard-fast-again' into 1238...
Arvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat@veritasgenetics.com> Fernando Monserrat
02:27 pm Revision 03e384ee (arvados): 12389: new build for Roche including changes from master up to 197d9...
Fernando Monserrat
02:15 pm Revision 056b6611 (arvados): 12389: Merge branch '12389-git-from-tag' into 12389-roche-packages-111
Arvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat@veritasgenetics.com> Fernando Monserrat
06:37 am Bug #12475: keepproxy panic: send on closed channel
12475-asyncbuf @ commit:4f1a135e93df78bb833dff32562efe713c6f690e
* complete rewrite of sdk/go/streamer as sdk/go/asy...
Tom Clegg
06:27 am Revision 4f1a135e (arvados): 12475: Add TestManyFailedPuts with a short timeout.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:27 am Revision abc241fb (arvados): 12475: Rewrite streamer -> asyncbuf.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

11/22/2017

10:49 pm Revision b3054892 (arvados): 12627: Ensure current_user relates to the token that permits scope.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
09:39 pm Story #12519: Multi-cluster project search
Updates at commit:2f83263d2
Test run: https://ci.curoverse.com/job/developer-run-tests/510/
On sprint review demo...
Lucas Di Pentima
09:34 pm Revision 2f83263d (arvados): 12519: Remove trailing slash when guessing the workbench url.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
09:32 pm Story #11453: Federated user identity which works across a network of Arvados clusters
Related to #12627 it might be a good idea to move more of the token handling into the middleware instead of the appli... Peter Amstutz
08:47 pm Feature #12018: Synchronize group membership with external data source
Where is this tool packaged? Documented?
As an aside, it's too late now, but when did we switch from using Python ...
Tom Morris
08:38 pm Revision 36e2a9db (arvados): 12616: Fix validate
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:30 pm Revision bef091f6 (arvados): Merge branch '12377-arvbox-composer' refs #12377
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:25 pm Task #12647 (New): Review
Peter Amstutz
08:21 pm Task #12646 (New): Review
Lucas Di Pentima
08:20 pm Task #12645 (New): Review
Lucas Di Pentima
08:19 pm Task #12643 (New): Review
Tom Morris
08:05 pm Feature #12640 (New): Workbench links to composer
* Workbench configuration with URL of Composer app.
* Workbench title bar includes link to Composer app
Peter Amstutz
08:01 pm Feature #12639 (New): Composer theme to match Workbench color scheme
Change "Rabix Composer" to "Arvados Composer" and adjust color scheme to match Workbench. Peter Amstutz
07:57 pm Task #12637 (New): Review
Peter Amstutz
07:57 pm Task #12636 (New): Fix
Peter Amstutz
07:56 pm Task #12634 (New): Review
Peter Amstutz
07:54 pm Bug #12573 (In Progress): Priority is ignored in Crunch2
Peter Amstutz
07:51 pm Bug #12630 (New): [Crunch2] GPU support
In CWL:... Peter Amstutz
07:49 pm Bug #12475: keepproxy panic: send on closed channel
The behavior of a second call to Close() is "undefined", but that doesn't mean it's allowed to crash. There are code ... Tom Clegg
07:33 pm Feature #12260 (Resolved): Healthcheck endpoint aggregator
Tom Clegg
07:31 pm Revision eb84fae9 (arvados): Set composer config. Bump SSO script for updated passenger.
Peter Amstutz
07:25 pm Feature #12522 (In Progress): [Composer] Browse Arvados git repos
Peter Amstutz
07:12 pm Story #12527 (In Progress): [Composer] Make pull request to upstream with run-as-webapp configura...
Jasmin Besic
07:06 pm Feature #12521 (Resolved): [Composer] API server configuration option
Jasmin Besic
06:58 pm Story #8233 (Resolved): [Arvbox] Use arvbox on jenkins
Used for cwl conformance tests Peter Amstutz
06:57 pm Bug #8223 (Resolved): [Workbench] view log via keep-web test fails in arvbox
Fixed by upgrading phantomJS Peter Amstutz
05:45 pm Revision 2fcbfc10 (arvados): 12483: Fix cadaver tests.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:45 pm Revision 907fdae0 (arvados): 12483: Fix multiple WriteHeader(). Skip updates on readonly methods.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:30 pm Bug #12559 (Resolved): [SSO] Update gem dependencies, especially Rails
Peter Amstutz
04:21 pm Feature #12626 (New): Merge user accounts
New API end point to migrate user accounts, accepts tokens of old user and new user.
Workbench will provide a "mer...
Peter Amstutz
03:35 pm Feature #12483: [keep-web] writable webdav
12483-writable-fs @ commit:2fcbfc106c807aa17d2f73ce40fe2a64ed4c5b13
* rename extent&rarr;segment, file&rarr;filehand...
Tom Clegg
03:25 pm Feature #12483: [keep-web] writable webdav
On the topic of WebDAV, I noticed this request against the current keep-web master is very slow to respond:... Peter Amstutz
03:17 pm Revision f089d5d3 (arvados): 12483: Update comments.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
03:17 pm Story #11876: [R SDK] Create a Bioconductor/R SDK
Here's an example curl line to get collection contents via WebDAV:... Peter Amstutz
02:57 pm Revision 49fa2bdd (arvados): 12483: Drop writableSegment interface.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:56 pm Revision 0a5e96b6 (arvados): 12483: Rename extent->segment, file->filehandle.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:11 pm Revision ba584a7c (sso-provider): Merge branch '12559-update-gems' refs #12559
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz

11/21/2017

11:10 pm Bug #12318 (Resolved): Honor Retry-After headers on libcloud exceptions
Applied in changeset arvados|commit:3156c76f53b4142beb912cb57fd66e343c9e09d4. Anonymous
09:23 pm Bug #12318: Honor Retry-After headers on libcloud exceptions
This LGTM, thanks. Peter Amstutz
11:07 pm Revision 3156c76f (arvados): Merge branch '12318-update-libcloud-pin'
Closes #12318
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
09:55 pm Story #12519: Multi-cluster project search
Updates at commit:cf6ebed6c
Test run: https://ci.curoverse.com/job/developer-run-tests/509/
Removed superfluous c...
Lucas Di Pentima
09:16 pm Story #12519: Multi-cluster project search
I don't think onupdate is the right way to hook up bootstrap-jquery stuff: it does N^2 work on dom updates. We should... Tom Clegg
06:45 pm Story #12519: Multi-cluster project search
Updates at commit:fda471556
Test run: https://ci.curoverse.com/job/developer-run-tests/508/
Fixed workbench integ...
Lucas Di Pentima
03:45 pm Story #12519: Multi-cluster project search
Some tests are failing because of Javascript errors. While tracking them down, here's a new screenshot with object ty... Lucas Di Pentima
02:43 pm Story #12519: Multi-cluster project search
Running tests at: https://ci.curoverse.com/job/developer-run-tests/507/ Lucas Di Pentima
02:41 pm Story #12519: Multi-cluster project search
Thanks! Adding a screenshot. Lucas Di Pentima
02:17 pm Story #12519: Multi-cluster project search
found a way to move to /search: 12519-multisite-project-search @ commit:9ea1f795a9c4050d8f01cd2f130a3c6c3ea1fd69 Tom Clegg
09:52 pm Revision cf6ebed6 (arvados): 12519: Activate every search results' tooltip on creation.
Removed unnecessary code.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
09:43 pm Task #12567 (Resolved): Review 12559-update-gems on SSO
Ward Vandewege
09:43 pm Bug #12559: [SSO] Update gem dependencies, especially Rails
This LGTM, I have tested this branch and it works. Ward Vandewege
09:28 pm Feature #4637: [SSO] Use "authentications" table and support account linking
This seems to be mostly about catering to the sso-provider design. IMO our use of sso-provider is an awful hack that ... Tom Clegg
07:12 pm Revision 197d9db5 (arvados): Merge branch '12513-make-dashboard-fast-again'
refs #12513
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege
07:11 pm Revision 58be2f1c (arvados): Merge branch 'master' into 12513-make-dashboard-fast-again
refs #12513
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege
07:08 pm Revision 25ce0f65 (arvados): Match the actual name of the CR in the
ApplicationLayoutTest#test_visit_dashboard_as_* tests, rather than the
state of the CR. It's confusing that the name ...
Ward Vandewege
07:03 pm Revision a14d8268 (arvados): Make sure to apply the text-overflow-ellipsis class to the name of
running container requests, too.
refs #12513
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenet...
Ward Vandewege
06:11 pm Revision fda47155 (arvados): 12519: Make Capybara happy by using a different kind of for loop.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
06:07 pm Feature #12483: [keep-web] writable webdav
12483-writable-fs @ commit:dc56b929215f826fb057ee5b9b7dfa58ff5ab3ed
* connects webdav move + delete functions
* cha...
Tom Clegg
06:02 pm Story #11876: [R SDK] Create a Bioconductor/R SDK
https://github.com/fmuhic/ArvadosSDK Peter Amstutz
04:18 pm Revision b7cd87b8 (arvados): 12519: Use icons on 'show' buttons instead of 'C'/'P' labels.
Added bootstrap-style tooltips.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
02:28 pm Revision c8a75b03 (arvados): Add another missing dependency to the sanity check in run-tests.sh.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege
02:15 pm Revision 9ea1f795 (arvados): 12519: Move multisite search page to /search
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
08:03 am Revision dc56b929 (arvados): 12483: Fix receiver type.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:16 am Revision 70111d82 (arvados): 12483: Delay flushing collection until successful http response.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:59 am Revision c70be5b3 (arvados): 12483: Delay flushing collection until successful http response.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

11/20/2017

11:48 pm Revision 11127c5b (arvados): 12483: Make webdav rename/remove work. Tidy up code.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:04 pm Feature #12483: [keep-web] writable webdav
12483-writable-fs @ commit:127e916894bcd16f6978aa6488c81e79a9ca2812
* Fix seek+write-beyond-EOF semantics
* Support...
Tom Clegg
05:56 pm Feature #10666 (In Progress): All Arvados components should report their version
Lucas Di Pentima
05:11 pm Revision 127e9168 (arvados): 12483: More loading speed.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:11 pm Revision 415e6410 (arvados): 12483: Speed up manifest loading.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
03:37 pm Task #12534 (Resolved): Review 12521-web-app-config
Peter Amstutz
03:17 pm Feature #12521: [Composer] API server configuration option
git checkout master
git merge --no-ff 12521-web-app-config
git push github:master (curoverse/composer)
Peter Amstutz
02:34 pm Task #12609 (New): Diagnose
Peter Amstutz
12:16 am Revision 359c2058 (arvados): Add missing dependencies to the sanity check in run-tests.sh.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege

11/19/2017

08:13 pm Revision 56776607 (arvados): 12513: Fix 2 more tests.
refs #12513
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege

11/18/2017

07:28 am Revision 33d4ec96 (arvados): 12483: Allow seek and write beyond EOF.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision 0373c781 (arvados): 12483: Support O_APPEND. Check for invalid/unsupported file modes.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision 5759c046 (arvados): 12483: Compress adjacent segments when writing filetokens.
"0:100:foo 100:100:foo 200:100:foo" --> "0:300:foo"
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics...
Tom Clegg
07:28 am Revision 228699ab (arvados): 12483: Avoid empty segments.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision fdc73931 (arvados): 12483: Support O_TRUNC flag.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision d09c6f1c (arvados): 12483: Add RemoveAll() to CollectionFileSystem.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision 14f46669 (arvados): 12483: Add comments.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision e789dc0f (arvados): 12483: Enable rename via webdav.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision 00a4d2f8 (arvados): 12483: Add Rename(old,new) to CollectionFileSystem.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:28 am Revision 307188e3 (arvados): 12483: Support file create/write via webdav.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:53 am Revision 705c9cc8 (arvados): 12483: Allow seek and write beyond EOF.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision 09061421 (arvados): 12483: Add RemoveAll() to CollectionFileSystem.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision 7f1d66f3 (arvados): 12483: Avoid empty segments.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision 5d8b5dd2 (arvados): 12483: Enable rename via webdav.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision 9d2c8277 (arvados): 12483: Add Rename(old,new) to CollectionFileSystem.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision 17f75ea8 (arvados): 12483: Support file create/write via webdav.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision 9a35977b (arvados): 12483: Add comments.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision 11d7f4d0 (arvados): 12483: Compress adjacent segments when writing filetokens.
"0:100:foo 100:100:foo 200:100:foo" --> "0:300:foo"
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics...
Tom Clegg
05:23 am Revision 57d36542 (arvados): 12483: Support O_TRUNC flag.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:23 am Revision e2432dff (arvados): 12483: Support O_APPEND. Check for invalid/unsupported file modes.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
12:28 am Story #12519: Multi-cluster project search
Updates at commit:d4de94839
Couldn't make @/search@ to work as it's already being used by the normal search featur...
Lucas Di Pentima
12:13 am Revision d4de9483 (arvados): 12519: Moved multisite search to its own URL
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima

11/17/2017

11:43 pm Feature #12483: [keep-web] writable webdav
re: Note-12/13
If manifest "loading" is code for parsing entire text-based manifests which are 10s or 100s of MB e...
Tom Morris
09:48 pm Feature #12483: [keep-web] writable webdav
Peter Amstutz wrote:
> Does os.OpenFile with the Dir mode bit set actually create a directory? (If not, then the be...
Tom Clegg
09:39 pm Feature #12483: [keep-web] writable webdav
Failing test:... Peter Amstutz
09:15 pm Feature #12483: [keep-web] writable webdav
Does os.OpenFile with the Dir mode bit set actually create a directory? (If not, then the behavior is different from... Peter Amstutz
08:03 am Feature #12483: [keep-web] writable webdav
OpenFile() is analogous to "os.OpenFile()":https://golang.org/pkg/os/#OpenFile, which is essentially "open()":https:/... Tom Clegg
03:45 am Feature #12483: [keep-web] writable webdav
This method needs documentation:... Peter Amstutz
09:52 pm Revision 856b9bb8 (arvados): Merge branch '12530-faster-scopes-filter'
closes #12530
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
09:09 pm Story #12519: Multi-cluster project search
Seems like this page is turning into "/search", and /collections/multisite should redirect there.
Maybe Collection...
Tom Clegg
08:59 pm Story #12519: Multi-cluster project search
You were right, it allowed to simplify the code: commit:357cbabe1
Any ideas about component organization and search ...
Lucas Di Pentima
05:57 pm Story #12519: Multi-cluster project search
I wonder if it would be easier to do a deeper tree: a MergingLoader with one child per session, where each child has ... Tom Clegg
08:58 pm Revision 357cbabe (arvados): 12519: Updated search box placeholder message
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
08:53 pm Revision 6247858b (arvados): 12519: Simplified code by nesting an additional MergingLoader
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
07:09 pm Bug #12606: Symlink in output points to invalid location -- no such file or directory
Thanks Peter. I'll work around this by avoiding the symlinks and copying the files out of the strelka2 specific direc... Brad Chapman
04:01 pm Bug #12606: Symlink in output points to invalid location -- no such file or directory
I think what it is trying to tell you that the output file strelka2/chr19/NA24385-chr19_0_16089283-block.vcf.gz is a ... Peter Amstutz
02:48 pm Bug #12606: Symlink in output points to invalid location -- no such file or directory
Sorry the error message got formatted all on one line for some reason. Here it is in a more readable format:... Brad Chapman
02:46 pm Bug #12606 (New): Symlink in output points to invalid location -- no such file or directory
Hi all;
I'm running into an issue on runs with symlinked outputs, specifically for strelka2 variant calling. The run...
Brad Chapman
05:54 pm Revision 1b26b24b (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
05:52 pm Revision ede2bd8d (arvados-dev): refs #1572: commit not committed changes on ci server bringing u...
update to avoid overwritte uncommited changes on ci
Arvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserra...
Fernando Monserrat
05:25 pm Story #11908: Migrate Collections.properties to JSONB
As a reminder, in July #11807 did (what we expect to be) the slowest part of the yaml-to-json migration, i.e., the jo... Tom Clegg
05:20 pm Bug #12318: Honor Retry-After headers on libcloud exceptions
Updates at commit:2e26522a30b1bb866be81b1013e3329b4d21ed6e
Test run: https://ci.curoverse.com/job/developer-run-test...
Lucas Di Pentima
05:19 pm Revision 2e26522a (arvados): 12318: Merge branch 'master' into 12318-update-libcloud-pin
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
05:16 pm Revision 2cd5de4a (arvados): 12318: Updated integration test to correctly raise a RateLimitReache...
exception, matching libcloud updates.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
04:06 pm Revision c7e1c587 (arvados): 12483: Add Rename(old,new) to CollectionFileSystem.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
03:30 pm Revision c8f55a96 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
03:28 pm Revision 1a5ba147 (arvados-dev): Merge branch '11572-separate-repositories'
refs #11572
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@veritasgenetics.com>
Javier Bértoli
08:04 am Revision c03306b7 (arvados): 12483: Add comments.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

11/16/2017

09:58 pm Revision a9755547 (arvados): 12483: Support O_TRUNC flag.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
09:20 pm Story #11163: [GA4GH] Participate in workflow interoperability activities
todos:
* Fix $import munging (already does the right thing for regular cwltool --pack but not for a-c-r, figure ou...
Peter Amstutz
09:11 pm Story #11163: [GA4GH] Participate in workflow interoperability activities
https://cloud.curoverse.com/container_requests/qr1hi-xvhdp-ip4bwv2mt09qecq#Status Peter Amstutz
09:10 pm Story #11163 (In Progress): [GA4GH] Participate in workflow interoperability activities
Peter Amstutz
09:19 pm Revision 7399705e (arvados): 12483: Compress adjacent segments when writing filetokens.
"0:100:foo 100:100:foo 200:100:foo" --> "0:300:foo"
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics...
Tom Clegg
09:06 pm Revision 0d29dc46 (arvados): 12483: Support O_APPEND. Check for invalid/unsupported file modes.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
09:04 pm Revision dc78526b (arvados): Bump schema-salad for extension schema loading bugfix. refs #11163
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:26 pm Task #12581 (In Progress): Review 12519-multisite-project-search
Lucas Di Pentima
08:26 pm Story #12519: Multi-cluster project search
Updates at commit:336973bf2 - branch @12519-multisite-project-search@
Took advantage of MergingLoader and added a ...
Lucas Di Pentima
08:15 pm Revision 336973bf (arvados): 12519: Added project search to multisite search
First draft, for approach validaton purposes.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgen...
Lucas Di Pentima
07:04 pm Revision e48ebb43 (arvados): 12530: Speed up query by skipping unneeded ActiveRecord loading.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:59 pm Revision 93bc61c0 (arvados): 12530: Speed up query by skipping unneeded ActiveRecord loading.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
04:32 pm Feature #12483: [keep-web] writable webdav
Tom Clegg wrote:
> So far I've been using "segments" to mean portions of a stream (a "manifest v1" concept). Those a...
Peter Amstutz
04:12 am Feature #12483: [keep-web] writable webdav
So far I've been using "segments" to mean portions of a stream (a "manifest v1" concept). Those are related to so-cal... Tom Clegg
03:00 pm Bug #12601 (Resolved): [CWL] RunInSingleContainer cwl.input.json needs to be consistent with path...
Applied in changeset arvados|commit:6be9a2c8066a633327f797e537c228fb9b9d91c5. Anonymous
12:32 pm Bug #12601: [CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper mani...
I'm having lots of issues trying to run the tests successfully on my arvbox instance (required to install a newer doc... Lucas Di Pentima
02:56 pm Task #12602 (Resolved): Review 12601-pathmap-subwf
Peter Amstutz
02:54 pm Revision 6be9a2c8 (arvados): Merge branch '12601-pathmap-subwf' closes #12601
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
02:09 pm Bug #12593 (Closed): Unable to resolve localhost in GATK4 log4j setup
Nico and Tom -- thanks again for the help. The `arv:APIRequirement` trick seems to have done it and I can now progres... Brad Chapman

11/15/2017

09:45 pm Revision c23c3a36 (arvados): Merge branch '10767-remove-puma-gem'
refs #10767
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege
09:36 pm Feature #12483: [keep-web] writable webdav
Is there any chance you'd consider renaming "extent" to "segment" in your code for consistency with the Python SDK? Peter Amstutz
06:44 pm Feature #12483: [keep-web] writable webdav
12483-writable-fs @ commit:9cee2680c3b1f74d2a43d9a0e2b572168e3b488d
working:
* writable filesystem
* webdav clie...
Tom Clegg
08:37 pm Bug #12601: [CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper mani...
Lucas Di Pentima wrote:
> Running @sdk/cwl@ tests with master's version of @sdk/cwl/arvados_cwl/arvworkflow.py@ pass...
Peter Amstutz
08:35 pm Bug #12601: [CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper mani...
Running @sdk/cwl@ tests with master's version of @sdk/cwl/arvados_cwl/arvworkflow.py@ pass ok, is that a false positi... Lucas Di Pentima
05:55 pm Bug #12601: [CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper mani...
12601-pathmap-subwf @ commit:2b713e548510a99803c41ceb13d47fa2f38b51bf
I'm working on a test case.
Peter Amstutz
04:11 pm Bug #12601 (In Progress): [CWL] RunInSingleContainer cwl.input.json needs to be consistent with p...
Peter Amstutz
03:49 pm Bug #12601 (Resolved): [CWL] RunInSingleContainer cwl.input.json needs to be consistent with path...
Container request mounts:... Peter Amstutz
08:03 pm Revision e20fbcb9 (arvados): Remove puma from the API server Gemfile.
refs #10767
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege
07:42 pm Bug #12573: Priority is ignored in Crunch2
For the crunch-dispatch-slurm case: Suggest submitting each job with nice=10000, monitoring the actual priority assig... Tom Clegg
07:17 pm Feature #12521: [Composer] API server configuration option
Could you rename @src/arvados-configuration.yml@ to @src/arvados-configuration.yml.example@ ? The developer should b... Peter Amstutz
07:06 pm Task #12534 (In Progress): Review 12521-web-app-config
Peter Amstutz
06:44 pm Task #12578 (In Progress): Review 12483-writable-fs
Tom Clegg
06:34 pm Revision 423364e3 (arvados): 12601: Add test
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
06:27 pm Task #12602 (In Progress): Review 12601-pathmap-subwf
Lucas Di Pentima
06:20 pm Task #12602 (Resolved): Review 12601-pathmap-subwf
Lucas Di Pentima
05:59 pm Task #12602 (Resolved): Review 12601-pathmap-subwf
Lucas Di Pentima
06:24 pm Revision 9cee2680 (arvados): 12483: Support file create/write via webdav.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:14 pm Revision b33889df (arvados): 12483: Fix makeParentDirs bug when higher levels already exist.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:01 pm Revision 794d493c (arvados): 12483: Persist empty files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
04:10 pm Revision 2b713e54 (arvados): 12601: Apply pathmap to job order for RunInSingleContainer subwf.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz

11/14/2017

08:35 pm Revision 056a3fa0 (arvados): 12483: Track memory use. Flush filled blocks while writing.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:51 pm Bug #12593: Unable to resolve localhost in GATK4 log4j setup
Sorry I shouldn't read this in a meeting: `arv:APIRequirement` to the `requirements`. Got it. I'll give that a try an... Brad Chapman
06:48 pm Bug #12593: Unable to resolve localhost in GATK4 log4j setup
Nico and Tom -- thanks so much for digging into this. This is really helpful. log4j bundled with the GATK jar so I do... Brad Chapman
05:51 pm Bug #12593: Unable to resolve localhost in GATK4 log4j setup
I suspect this is a side effect of disabling networking in the container environment, which Arvados does by default.
...
Tom Clegg
05:45 pm Bug #12593: Unable to resolve localhost in GATK4 log4j setup
I see that the current version of log4j does a try/catch on that particular line (look like android devices fail too)... Nico César
03:34 pm Story #12519 (In Progress): Multi-cluster project search
Lucas Di Pentima
02:47 pm Bug #12318: Honor Retry-After headers on libcloud exceptions
Re running tests https://ci.curoverse.com/job/developer-run-tests/498/ Lucas Di Pentima
02:40 pm Bug #12475: keepproxy panic: send on closed channel
After adding lots of logging and running a test case with @keepclient@'s @HTTPClient@ Timeout set to 1 microsecond to... Lucas Di Pentima
02:34 pm Revision 375602be (arvados): 12483: Fix missing Rlock() on inode accesses.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:23 pm Revision 1ee57409 (arvados): 12483: Return error on impossible manifest.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:23 pm Revision 90fca0f2 (arvados): 12453: Persist written data to keep.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:23 pm Revision ec69b9a0 (arvados): 12483: Avoid write/marshal race. Remove dead code.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

11/13/2017

09:50 pm Revision 0e95e763 (arvados): 12453: Persist written data to keep.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
09:50 pm Revision cd45aed0 (arvados): 12483: Remove unnecessary OpenFile() from inode interface.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
09:00 pm Revision c5d42c41 (arvados): 12513: Update tests, remove more unused code.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
08:04 pm Revision 4f21d0e5 (arvados): 12483: Add Mkdir(), Remove().
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:21 pm Task #12577 (Resolved): Review 12515-configured-workbench-address & 12515-use-configured-wb
Tom Clegg
03:45 pm Bug #12515 (Resolved): Workbench base URL for multi-site search results should be configurable
Applied in changeset arvados|commit:2f66ce454917c8d9d016a9438e529a0a20317028. Anonymous
03:38 pm Revision 2f66ce45 (arvados): Merge branch '12515-use-configured-wb'
closes #12515
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
03:38 pm Revision 9e6a74ae (arvados): Merge branch '12515-configured-workbench-address'
refs #12515
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
03:05 pm Bug #12594 (New): [crunch1] crunchrunner should report "signal: SIGKILL" instead of "exit code -1...
Currently, if a crunchrunner-based crunch1 job exits via signal, it doesn't log the signal. We just get this:
<pre...
Tom Clegg

11/12/2017

08:42 am Revision 236ede14 (arvados): 12483: Add MarshalManifest().
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
08:30 am Revision 8bd41679 (arvados): 12483: Remove unnecessary repack.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

11/11/2017

11:42 pm Revision 015e3798 (arvados): 12483: Add tests for concurrent writers and random write sequences.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
10:26 pm Revision da2f8a10 (arvados): 12483: Add tests for concurrent writers and random write sequences.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
08:48 pm Revision 9009baaf (arvados): 12483: Add (File)Truncate().
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
08:32 pm Revision 6ce00fb7 (arvados): 12483: Simplify extent packing, reduce type casting.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
11:08 am Bug #12593 (Closed): Unable to resolve localhost in GATK4 log4j setup
I'm running into an issue running bcbio workflows with GATK4, where they fail in a localhost lookup when setting up l... Brad Chapman
08:30 am Revision cad7d333 (arvados): 12483: Implement some writable cases.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

11/10/2017

09:34 pm Revision da3f2283 (arvados): 12483: Add basic file writing tests.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
09:33 pm Revision 3e3abb01 (arvados): 12483: Rewrite collection filesystem.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:51 pm Task #12591 (In Progress): Review 12318-update-libcloud-pin
Lucas Di Pentima
05:50 pm Bug #12318: Honor Retry-After headers on libcloud exceptions
Finally PR 1139 was merged to libcloud trunk.
Update commit:dc003f39b (branch @12318-update-libcloud-pin@) has a n...
Lucas Di Pentima
05:42 pm Revision dc003f39 (arvados): 12318: Use libcloud fork 2.2.2.dev3
Includes a fix for delete_old_vhd, limit number of retries on
destroy_node & a fix on Retry-After header handling on
...
Lucas Di Pentima

11/09/2017

08:40 pm Revision 3ff7fc1d (arvados): Add missing module name.
refs #11220
fixes #12590
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
07:57 pm Bug #12515: Workbench base URL for multi-site search results should be configurable
Sorry, for the delay.
This LGTM, thanks for the explanation, will have to read a lot more about Mithril.
Lucas Di Pentima
06:27 pm Bug #12515: Workbench base URL for multi-site search results should be configurable
Lucas Di Pentima wrote:
> * Can we have a workbench integration test to check that the correct URL is being selected...
Tom Clegg
05:50 pm Bug #12515: Workbench base URL for multi-site search results should be configurable
Some questions:
* Can we have a workbench integration test to check that the correct URL is being selected? How ab...
Lucas Di Pentima
06:26 pm Revision e1da2448 (arvados): 12515: Add discovery doc test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:21 pm Feature #12483 (In Progress): [keep-web] writable webdav
Tom Clegg
02:14 pm Task #12577 (In Progress): Review 12515-configured-workbench-address & 12515-use-configured-wb
Lucas Di Pentima

11/08/2017

09:56 pm Bug #12515: Workbench base URL for multi-site search results should be configurable
part 2 (use advertised wb site when possible): 12515-use-configured-wb @ commit:63f1542e94f3f1e66cbf0d88f557105ff5d10... Tom Clegg
05:59 pm Bug #12515: Workbench base URL for multi-site search results should be configurable
part 1 (add to discovery doc): 12515-configured-workbench-address @ commit:a5d1a2aa40ad211b4faf870c603ae7ad5263b6bc
Tom Clegg
09:53 pm Revision 63f1542e (arvados): 12515: Use workbenchUrl advertised by API server discovery doc.
Fall back to guessing "workbench."+apiHost (as before) if discovery
doc does not offer a workbenchUrl (e.g., API serv...
Tom Clegg
08:46 pm Task #12588 (New): Review
Peter Amstutz
08:45 pm Task #11933 (Resolved): Review 11908-properties-column-json
Tom Clegg
08:40 pm Task #12583 (In Progress): Review 12475-asyncbuf
Lucas Di Pentima
08:40 pm Task #12581 (In Progress): Review 12519-multisite-project-search
Lucas Di Pentima
08:27 pm Task #12579 (New): Review
Tom Clegg
08:25 pm Task #12578 (In Progress): Review 12483-writable-fs
Tom Clegg
08:25 pm Task #12577 (Resolved): Review 12515-configured-workbench-address & 12515-use-configured-wb
Tom Clegg
06:16 pm Story #12484 (Resolved): Prototype using js-git to access Arvados repo
Peter Amstutz
06:05 pm Task #12554 (Resolved): Review 12538-crunchrun-kill-arvmount
Peter Amstutz
06:05 pm Task #12569 (Resolved): Review 12568-cwl-collection-cache
Peter Amstutz
05:59 pm Revision a5d1a2aa (arvados): 12515: Advertise Workbench address in discovery doc.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:15 pm Bug #12568 (Resolved): [CWL] Cap size of collection cache
Applied in changeset arvados|commit:fc128bc6497aa266f925e2aa4821bde6fce9aade. Anonymous
05:11 pm Bug #12568: [CWL] Cap size of collection cache
Lucas Di Pentima wrote:
> Ok, this fix seems to be enough, I was just thinking that maybe we can give the user more ...
Peter Amstutz
05:07 pm Bug #12568: [CWL] Cap size of collection cache
Ok, this fix seems to be enough, I was just thinking that maybe we can give the user more knobs to try to solve a pot... Lucas Di Pentima
04:54 pm Bug #12568: [CWL] Cap size of collection cache
Lucas Di Pentima wrote:
> The cache size management looks good to me.
>
> If some enormous collections get into t...
Peter Amstutz
05:09 pm Revision fc128bc6 (arvados): Merge branch '12568-cwl-collection-cache' closes #12568
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
05:07 pm Bug #12538: crunch-run failing to terminate after complete
Tom Clegg wrote:
> It looks like you went to some trouble to sigkill arv-mount only if it hasn't exited 9 seconds af...
Peter Amstutz
05:05 pm Bug #12538 (Resolved): crunch-run failing to terminate after complete
Applied in changeset arvados|commit:4b23309ecedb0010d907a6930844c6928b3755b6. Anonymous
04:59 pm Bug #12538: crunch-run failing to terminate after complete
(still curious about the waiting thing but) LGTM, thanks Tom Clegg
04:56 pm Bug #12538: crunch-run failing to terminate after complete
It looks like you went to some trouble to sigkill arv-mount only if it hasn't exited 9 seconds after we call @arv-mou... Tom Clegg
04:12 pm Bug #12538: crunch-run failing to terminate after complete
Tom Clegg wrote:
> Hm. I'm not sure refactoring the shutdown code was a good idea. I'm not sure I'm seeing all of ...
Peter Amstutz
12:02 am Bug #12538: crunch-run failing to terminate after complete
Peter Amstutz wrote:
> Tom Clegg wrote:
> > The deferred "crunch-run finished" message looks like it's too late: by...
Tom Clegg
05:02 pm Revision 4b23309e (arvados): Merge branch '12538-crunchrun-kill-arvmount' closes #12538
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
04:55 pm Task #12564 (Resolved): Review 12306-dont-stat-mounts
Peter Amstutz
04:15 pm Bug #12306 (Resolved): [arv-mount] --unmount should work on an unresponsive mount
Applied in changeset arvados|commit:0af053088c83d1107866cb06fd6c5736d9065eee. Anonymous
04:00 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
Otherwise, the main goal of this bugfix (don't get stuck on realpath()) seems to be accomplished, so declare victory ... Peter Amstutz
03:10 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
My preferred method to bring the hammer down:
# abort if available
# sigkill
# fusermount -u -z
Peter Amstutz
03:06 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
Tom Clegg wrote:
> Indeed, the previous version would have ended up calling realpath() on @~/keep@ on a system where...
Peter Amstutz
04:07 pm Revision 0af05308 (arvados): Merge branch '12306-dont-stat-mounts'
fixes #12306
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
03:58 pm Revision 47508624 (arvados): 12538: Put cleanup back into separate defer
* Always report unmount command
* Send kill signal to arv-mount on timeout (but don't wait for it)
Arvados-DCO-1.1-...
Peter Amstutz
03:58 pm Revision 899d369b (arvados): Force kill arv-mount wip
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
03:58 pm Revision e2bf56f0 (arvados): 12538: Refactor crunch-run shutdown
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
03:58 pm Revision fff3b19b (arvados): 12538: Fix tests.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
03:58 pm Revision ec11b576 (arvados): 12538: Refactor force-unmount behavior.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
03:58 pm Revision 3b278b89 (arvados): 12538: Refactor and assume arv-mount --unmount does all the hard work.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
03:58 pm Revision 3c5de241 (arvados): 12538: Make handling of arv-mount process more robust
* If initial "fusermount -u -z" of arv-mount dir times out, use "arv-mount --unmount"
* If that doesn't work, send s...
Peter Amstutz

11/07/2017

11:28 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
Indeed, the previous version would have ended up calling realpath() on @~/keep@ on a system where @$HOME@ contains sy... Tom Clegg
02:51 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
Tom Clegg wrote:
> Peter Amstutz wrote:
> > So following symlinks to mounts seems weird and not something you would...
Peter Amstutz
11:22 pm Revision 08a4ebba (arvados): 12306: Resolve any readable symlinks on unmount path.
This avoids getting stuck in realpath/lstat in certain cases: e.g.,
the mount point is $HOME/keep, $HOME is "/home/fo...
Tom Clegg
09:31 pm Bug #12568: [CWL] Cap size of collection cache
The cache size management looks good to me.
If some enormous collections get into the cache and make a-c-r to be O...
Lucas Di Pentima
06:58 pm Bug #12568: [CWL] Cap size of collection cache
One more test update for completeness! commit:a18005f8b35a68b4fcd9ccdf76832b28e564289c Peter Amstutz
06:53 pm Bug #12568: [CWL] Cap size of collection cache
Added a couple unit tests. Now @ commit:a70727762dafee667b022331307f6c0f949fd7e7
Peter Amstutz
06:35 pm Bug #12568 (In Progress): [CWL] Cap size of collection cache
12568-cwl-collection-cache @ commit:d59645f3e566e691ba757f74bba503c13773dbe8 Peter Amstutz
04:09 pm Bug #12568 (Resolved): [CWL] Cap size of collection cache
The collection cache in arvados-cwl-runner is unbounded. When running workflows with many very large collections, th... Peter Amstutz
08:52 pm Revision 8de691c2 (arvados): 12568: Reduce min_entries
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:26 pm Task #12569 (In Progress): Review 12568-cwl-collection-cache
Lucas Di Pentima
06:41 pm Task #12569 (New): Review 12568-cwl-collection-cache
Peter Amstutz
06:35 pm Task #12569 (In Progress): Review 12568-cwl-collection-cache
Peter Amstutz
06:35 pm Task #12569 (Resolved): Review 12568-cwl-collection-cache
Peter Amstutz
08:08 pm Story #12574 (New): [API] Propagate priority of parent container to children
Propagate from the requesting container to children.
A committed container request's priority should be no less th...
Tom Morris
08:07 pm Bug #12573 (In Progress): Priority is ignored in Crunch2
As a workflow submitter, I would like to be able to set the priority on a workflow and have it influence the scheduli... Tom Morris
07:37 pm Story #12571 (New): arv-mount by_tag directory support for properties
Placholder
With the move from the links to properties for tags we need to figure out what needs to be done with th...
Tom Morris
06:58 pm Revision a18005f8 (arvados): 12568: Update test to check which collection gets evicted
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
06:52 pm Revision a7072776 (arvados): 12568: Add test for collection cache management behavior
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
06:32 pm Task #12442 (Resolved): Review retry-after-headers (libcloud fork)
Peter Amstutz
06:27 pm Feature #12430: Crunch2 limit output collection to glob patterns
Tom Clegg wrote:
> I'm not keen on this feature. It seems to creep in an awkward direction:
> # output everything i...
Peter Amstutz
05:34 pm Feature #12430: Crunch2 limit output collection to glob patterns
I'm not keen on this feature. It seems to creep in an awkward direction:
# output everything in this dir
# output e...
Tom Clegg
05:38 pm Bug #12475 (In Progress): keepproxy panic: send on closed channel
Lucas Di Pentima
04:44 pm Revision d59645f3 (arvados): 12568: Mange collection cache to avoid excessive memory usage.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
04:04 pm Bug #12538: crunch-run failing to terminate after complete
Tom Clegg wrote:
> Looking at 12538-crunchrun-kill-arvmount @ commit:e16989ed7897833f14766341838abd4e382a32a1
>
> ...
Peter Amstutz
04:00 pm Revision f830a945 (arvados): 12538: Refactor crunch-run shutdown
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz

11/06/2017

10:10 pm Bug #11994 (Resolved): [arv-mount] Do not crash if /sys/fs/fuse/connections is empty
Applied in changeset arvados|commit:d493d06b4b93b87c10c59954e47a18d85db13150. Anonymous
09:08 pm Bug #11994: [arv-mount] Do not crash if /sys/fs/fuse/connections is empty
Tom Clegg wrote:
> > I'm not sure what the next best thing is, probably @kill -9 arv-mount@ and then try @fusermount...
Peter Amstutz
06:22 pm Bug #11994 (In Progress): [arv-mount] Do not crash if /sys/fs/fuse/connections is empty
> I'm not sure what the next best thing is, probably @kill -9 arv-mount@ and then try @fusermount -u -z@ again?
Wi...
Tom Clegg
09:56 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
Peter Amstutz wrote:
> So following symlinks to mounts seems weird and not something you would normally do
On our...
Tom Clegg
09:29 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
So following symlinks to mounts seems weird and not something you would normally do, however, the other thing that re... Peter Amstutz
06:25 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
12306-dont-stat-mounts @ commit:7bc55d65082b3a39639508fcaebd1185b7e04089
Tom Clegg
06:13 pm Bug #12306 (In Progress): [arv-mount] --unmount should work on an unresponsive mount
Tom Morris
09:44 pm Revision aabf1ca0 (arvados): 12306: Resolve relative paths early, to avoid realpath() later.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
09:41 pm Revision 8d60bb5b (arvados): 12538: Refactor and assume arv-mount --unmount does all the hard work.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
09:30 pm Task #12567 (Resolved): Review 12559-update-gems on SSO
Peter Amstutz
09:30 pm Task #12563 (Resolved): Review 11994-no-sysfs-fuse
Peter Amstutz
07:31 pm Task #12563 (In Progress): Review 11994-no-sysfs-fuse
Tom Clegg
07:30 pm Task #12563 (Resolved): Review 11994-no-sysfs-fuse
Tom Clegg
09:11 pm Revision 4526f362 (sso-provider): 12559: Update to Rails 3.2.22.5 for security fixes.
"bundle update" updated many other gems as well. Peter Amstutz
09:11 pm Revision d493d06b (arvados): Merge branch '11994-no-sysfs-fuse'
fixes #11994
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
08:50 pm Bug #12538: crunch-run failing to terminate after complete
Offering two variants of unmount (nice/mean, soft/hard, normal/force?) sounds like a good feature. Added #12566. Curr... Tom Clegg
08:08 pm Bug #12538: crunch-run failing to terminate after complete
Tom Clegg wrote:
> Looking at 12538-crunchrun-kill-arvmount @ commit:e16989ed7897833f14766341838abd4e382a32a1
>
>...
Peter Amstutz
07:28 pm Bug #12538: crunch-run failing to terminate after complete
Looking at 12538-crunchrun-kill-arvmount @ commit:e16989ed7897833f14766341838abd4e382a32a1
Why call "fusermount -u...
Tom Clegg
08:49 pm Feature #12566 (New): [arv-mount] --unmount should sync/wait and unmount gracefully unless otherw...
Currently, the "arv-mount --unmount" strategy is to kill the arv-mount process and disconnect the mount point as quic... Tom Clegg
08:16 pm Revision a524fabd (arvados): Merge branch '12533-token-n-ip-logging'
Closes #12533
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
08:13 pm Revision 105c22f4 (arvados): 12533: Use @remote_ip to take into account the reverse proxy
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
07:31 pm Task #12564 (In Progress): Review 12306-dont-stat-mounts
Tom Clegg
07:31 pm Task #12564 (Resolved): Review 12306-dont-stat-mounts
Tom Clegg
06:26 pm Bug #12559 (Resolved): [SSO] Update gem dependencies, especially Rails
Peter Amstutz
06:21 pm Bug #12558 (New): [Workbench] Bad dashboard time formatting - 60m == 1h
I saw this on the workbench dashboard:... Tom Morris
06:16 pm Revision 4a8561a3 (arvados): 11994: Fix wrong class in "except ENOENT" handler.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:14 pm Task #12489 (In Progress): Groom & implementation plan
Peter Amstutz
06:02 pm Revision 7bc55d65 (arvados): 12306: Avoid calling realpath/lstat unnecessarily during --unmount.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:29 pm Task #12554 (In Progress): Review 12538-crunchrun-kill-arvmount
Peter Amstutz
02:24 pm Bug #11519 (In Progress): arv-get should abort on ctrl/C
Lucas Di Pentima

11/05/2017

04:20 pm Bug #11625: [Workbench] Ensure that running processes are visible on the dashboard
The original change described in this ticket sounds like a regression. What story was that associated with? Tom Morris

11/03/2017

08:41 pm Bug #12538: crunch-run failing to terminate after complete
This is somewhat dependent on #11994 and #12306 because it now assumes that @arv-mount --unmount@ will clean up whate... Peter Amstutz
08:37 pm Bug #11994: [arv-mount] Do not crash if /sys/fs/fuse/connections is empty
> Instead of crashing, arv-mount --unmount and friends should print a warning and try the next best thing.
I'm not...
Peter Amstutz
08:30 pm Task #12491 (Resolved): Review 12478-duration-fmt
Tom Clegg
08:30 pm Task #12438 (Resolved): Review 11220-manifest-fetch-error
Tom Clegg
08:30 pm Task #12554 (Resolved): Review 12538-crunchrun-kill-arvmount
Peter Amstutz
08:14 pm Revision e16989ed (arvados): 12538: Refactor force-unmount behavior.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:05 pm Revision d6c06f75 (arvados): 12533: Add client IP address & authorization token UUID to the logs.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
07:04 pm Revision ce9c7737 (arvados): 12538: Fix tests.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
06:46 pm Revision 7c02cd4b (arvados): 11220: Fix test setup.
refs #11220
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
05:25 pm Bug #12551 (Resolved): crunch-job should check errors from open() calls
Applied in changeset arvados|commit:98e9073e7ca36edbe8dd0569d67405e2e030f8db. Anonymous
05:11 pm Bug #12551: crunch-job should check errors from open() calls
Ok, LGTM, if any of these things are failing silently then at least they will fail loudly.
Peter Amstutz
05:03 pm Bug #12551: crunch-job should check errors from open() calls
Peter Amstutz wrote:
> Why not @exit_retry_unlocked();@ ?
Less likely to introduce more complications. I stuck wi...
Tom Clegg
05:00 pm Bug #12551: crunch-job should check errors from open() calls
Why not @exit_retry_unlocked();@ ? Peter Amstutz
03:00 pm Bug #12551: crunch-job should check errors from open() calls
If this makes any difference at all, it'll make the affected jobs fail (not as good as tempfail, but better than caus... Tom Clegg
02:47 pm Bug #12551 (In Progress): crunch-job should check errors from open() calls
Tom Clegg
02:47 pm Bug #12551: crunch-job should check errors from open() calls
12551-check-open-errors @ commit:ab82f5b47625e76b47893c992b31e9b2d2208d3f
Tom Clegg
02:17 pm Bug #12551 (Resolved): crunch-job should check errors from open() calls
Undetected open() errors could be causing the "missing stderr" bug mentioned in #12550.
Specifically, each task ch...
Tom Clegg
05:18 pm Revision 98e9073e (arvados): Merge branch '12551-check-open-errors'
fixes #12551
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
04:00 pm Bug #12552 (New): Job scheduler should prioritize based on time workflow is submitted
Currently I believe that job prioritization is done based on priority followed by job creation time.
Prioritizatio...
Tom Morris
03:34 pm Revision e9d5acfe (arvados): 12538: Make handling of arv-mount process more robust
* If initial "fusermount -u -z" of arv-mount dir times out, use "arv-mount --unmount"
* If that doesn't work, send s...
Peter Amstutz
03:30 pm Bug #11220 (Resolved): [SDKs] Fix misleading arv-mount/pysdk error messages by removing obsolete ...
Applied in changeset arvados|commit:e67d0f5d43c56f78694ea4a5f93acec5c93cd0fb. Anonymous
03:27 pm Revision e67d0f5d (arvados): Merge branch '11220-manifest-fetch-error'
closes #11220
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
03:02 pm Bug #12416 (Resolved): keepstore does not work with s3_volume on Ceph RadosGW
Tom Clegg
02:45 pm Revision ab82f5b4 (arvados): 12551: Check for errors after calling open().
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
02:04 pm Bug #12550 (New): crunch-job should tempfail jobs with empty stderr
Sometimes under high load, for a reason we don't know yet, @crunch-job@ does not receive any stderr from its slurm jo... Peter Amstutz

11/02/2017

10:21 pm Task #12442 (In Progress): Review retry-after-headers (libcloud fork)
Lucas Di Pentima
10:21 pm Bug #12318: Honor Retry-After headers on libcloud exceptions
PR Submitted: https://github.com/apache/libcloud/pull/1139 - branch @retry-after-headers@ on our fork. Lucas Di Pentima
09:52 pm Revision a7d2dbf5 (arvados): Force kill arv-mount wip
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:00 pm Bug #12546 (Resolved): [crunchstat-summary] should obey --skip-child-jobs when working on a conta...
Applied in changeset arvados|commit:4bc810b40c34c43efc6be4b1696aa14cfe865b79. Anonymous
07:48 pm Bug #12546: [crunchstat-summary] should obey --skip-child-jobs when working on a container
LGTM
I'm not thrilled that the usage message if you give it a bad switch only includes --job and not --container-r...
Tom Morris
07:06 pm Bug #12546 (In Progress): [crunchstat-summary] should obey --skip-child-jobs when working on a co...
12546-skip-container-children @ commit:805dbba0797fc40d0df9f664902f34be369f7525
Tom Clegg
07:04 pm Bug #12546 (Resolved): [crunchstat-summary] should obey --skip-child-jobs when working on a conta...
Currently there is no way to get a summary for a single container without following its child containers.
I want t...
Tom Clegg
07:57 pm Bug #12538 (In Progress): crunch-run failing to terminate after complete
Peter Amstutz
07:55 pm Revision 4bc810b4 (arvados): Merge branch '12546-skip-container-children'
fixes #12546
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
07:50 pm Bug #12478 (Resolved): [crunchstat-summary] bad time formatting for jobs > 24 hours
Applied in changeset arvados|commit:c4ac98db757e738431dbcde54ac738fb554e16e8. Anonymous
07:44 pm Bug #12478: [crunchstat-summary] bad time formatting for jobs > 24 hours
Tom Morris wrote:
> Looks great. I don't suppose you could be talked into creating a PR for the upstream too?
It ...
Tom Clegg
07:27 pm Bug #12478: [crunchstat-summary] bad time formatting for jobs > 24 hours
Looks great. I don't suppose you could be talked into creating a PR for the upstream too? Tom Morris
06:02 pm Bug #12478 (In Progress): [crunchstat-summary] bad time formatting for jobs > 24 hours
Tom Clegg
06:02 pm Bug #12478: [crunchstat-summary] bad time formatting for jobs > 24 hours
!example-10m.png!
12478-duration-fmt @ commit:805b4e7f801853e69514b11942bc2e5f79bc9dbd
Tom Clegg
07:48 pm Revision 603ab5b4 (arvados): 12543: crunch-dispatch.rb salloc gets TOP priority
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
07:45 pm Revision c4ac98db (arvados): Merge branch '12478-duration-fmt'
fixes #12478
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
07:09 pm Revision 805dbba0 (arvados): 12546: Update help message.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:05 pm Revision ab911da8 (arvados): 12546: Obey --skip-child-jobs flag for containers, too.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
06:55 pm Bug #12199: Don't schedule jobs on nodes which are bigger than requested
This approach isn't going to be as simple as originally thought. The problem is that jobs can have other constraints ... Ward Vandewege
06:13 pm Task #12491 (In Progress): Review 12478-duration-fmt
Tom Clegg
05:59 pm Revision 805b4e7f (arvados): 12478: Fix time-offset formatting + use more natural tick intervals.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
05:26 pm Bug #11220: [SDKs] Fix misleading arv-mount/pysdk error messages by removing obsolete "fetch mani...
Tom Clegg wrote:
> We could provide a more helpful error message/hint ("looks like you're trying to do ..., which ...
Lucas Di Pentima
04:52 pm Task #12544 (New): Tracing for keepstore, keepproxy, Go & Python SDKs
Tom Clegg
04:52 pm Task #12493 (Resolved): Review 12167-workbench-request-id
Tom Clegg
04:35 pm Feature #12521: [Composer] API server configuration option
Please note our convention for configuration file path and language:
/etc/arvados/component-name/component-name.yml
Ward Vandewege
04:30 pm Feature #12521 (In Progress): [Composer] API server configuration option
Jasmin Besic
04:04 pm Revision 289d2cf5 (arvados): Merge branch '12462-search-hyphen'
fixes #12462
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
03:56 pm Revision 0767da07 (arvados): Merge branch '12533-row-lock-contention'
Refs #12533
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
03:54 pm Revision e663d52f (arvados): Merge branch 'master' into 12533-row-lock-contention
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
03:06 pm Revision bb1d0c74 (arvados): Merge branch '12167-workbench-request-id'
refs #12167
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg
03:05 pm Feature #12018 (Resolved): Synchronize group membership with external data source
Applied in changeset arvados|commit:cc6f86f15e0d187cc1c84b874be3d2da7b20d19f. Anonymous
02:42 pm Feature #12018: Synchronize group membership with external data source
This LGTM @ commit:8fc7e0dd5d214e3881b8a56669f82d76aa70bfdb Peter Amstutz
03:00 pm Task #12205 (Resolved): Review 12183-crunch-run-symlinks
Peter Amstutz
02:58 pm Revision cc6f86f1 (arvados): Merge branch '12018-sync-groups-tool'
Closes #12018
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
02:56 pm Revision eae1286b (arvados): Merge branch 'master' into 12018-sync-groups-tool
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
02:55 pm Bug #12183 (Resolved): [crunch-run] Handle symlinks with absolute paths into output directory
Applied in changeset arvados|commit:295156a99251bf26d705272c329a64d58089bb9e. Anonymous
02:38 pm Bug #12183: [crunch-run] Handle symlinks with absolute paths into output directory
Just one nit in the documentation part, and only because someone who bothers to read this kind of documentation deser... Tom Clegg
01:34 pm Bug #12183: [crunch-run] Handle symlinks with absolute paths into output directory
Tom Clegg wrote:
> NotInOutputDirError should be ErrNotInOutputDir (convention is FooError is a type, ErrFoo is an o...
Peter Amstutz
02:53 pm Revision 295156a9 (arvados): Merge branch '12183-crunch-run-symlinks' closes #12183
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
02:52 pm Revision 5913a5c0 (arvados): 12183: Tweak documenation for clarity.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
02:48 pm Revision 797d5212 (arvados): 12533: Do not update the API Client Authorization record to avoid
row contention when lots of jobs are queued with the same token.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ld...
Lucas Di Pentima
02:41 pm Task #12264 (Resolved): Review 12018-sync-groups-tool
Peter Amstutz
02:02 pm Revision ca92724a (arvados): 12183: Fix section headers.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
01:35 pm Task #12499 (Resolved): Review 12494-arv-git-cors
Peter Amstutz
01:32 pm Revision cd0e7f06 (arvados): 12183: Fix renamed symbol
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz

11/01/2017

09:34 pm Bug #12538: crunch-run failing to terminate after complete
tom:
> seems like crunch-run needs to be more aggressive about killing arv-mount (perhaps abandoning it after some...
Peter Amstutz
06:39 pm Bug #12538: crunch-run failing to terminate after complete
So crunch-dispatch-slurm knows to scancel containers which are "Complete" but still in squeue, which has the effect o... Peter Amstutz
06:36 pm Bug #12538: crunch-run failing to terminate after complete
Tentatively it looks looks like we are expecting arv-mount to terminate, but it never does:... Peter Amstutz
05:21 pm Bug #12538 (Resolved): crunch-run failing to terminate after complete

/tmp/slurm-43045.out...
Peter Amstutz
09:10 pm Revision 0eb79244 (arvados): 12183: Update documentation.
Improve error message about follow symlink chains that are too long.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <p...
Peter Amstutz
08:50 pm Revision 511ff9e8 (arvados): 12183: Simplify walking symlinks in upload.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:50 pm Revision aa9fbcd3 (arvados): 12183: Add comment about purpose of "relocated"
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:50 pm Revision 68c4f5c4 (arvados): 12183: Refactor file upload so there is one walk to handle both syml...
regular files.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>
Peter Amstutz
08:50 pm Revision 3371bf4c (arvados): 12183: Handle circular directory symlinks
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:50 pm Revision f93e925c (arvados): 12183: Improve error handling if unable to delete symlinks marked fo...
Add unit test for EvalSymlinks.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>
Peter Amstutz
08:50 pm Revision 174d343c (arvados): 12183: Support uploading symlinks to directories.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:50 pm Revision 780e9fb8 (arvados): 12183: Refactor derefOutputSymlink into its own method.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:50 pm Revision cf3f89d7 (arvados): 12183: Fix tests.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
08:50 pm Revision e9eb0975 (arvados): 12183: Evaluate symlinks to handle chained links and absolute paths.
Ensure that all symlinks either resolve within the output directory or to a
keep mount.
Arvados-DCO-1.1-Signed-off-b...
Peter Amstutz
08:43 pm Feature #3316: Keep logs block writes to metadata server
Yes, this stuff (minus secret token) is included in the request log on stderr, which seems like a better place. Tom Clegg
08:00 pm Feature #3316 (Rejected): Keep logs block writes to metadata server
This sounds like a bad idea to me. Tom Morris
08:02 pm Task #3242 (Closed): add websockets server
The parent task is closed, so I'm assuming that this is either closed or doesn't need to be done. Tom Morris
06:34 pm Task #12493 (In Progress): Review 12167-workbench-request-id
Lucas Di Pentima
06:23 pm Feature #12167: [SDKs] [API] [keepstore] [keepproxy] Facilitate request tracing for arv-put
The updates at branch @12167-workbench-request-id@ LGTM.
If the req-id is created from the API when not provided, w...
Lucas Di Pentima
05:06 pm Task #12537 (New): Review
Peter Amstutz
05:06 pm Task #12536 (New): Review
Peter Amstutz
05:06 pm Task #12535 (New): Review
Peter Amstutz
05:06 pm Task #12534 (Resolved): Review 12521-web-app-config
Peter Amstutz
05:00 pm Feature #12494 (Resolved): arv-git-httpd supports CORS headers
Applied in changeset arvados|commit:c2a4869875daca18e19954c1beb4fab1ac20d821. Anonymous
08:02 am Feature #12494: arv-git-httpd supports CORS headers
"Access-Control-Expose-Headers: ETag" might be useful as well, but git protocol says servers _may_ return ETag, so pr... Tom Clegg
04:54 pm Revision c2a48698 (arvados): Merge branch '12494-arv-git-cors' closes #12494
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
03:44 pm Bug #11220: [SDKs] Fix misleading arv-mount/pysdk error messages by removing obsolete "fetch mani...
Lucas Di Pentima wrote:
> * Ran tests both on Jenkins and locally, I got some @services/fuse@ failures: https://ci.c...
Tom Clegg
03:36 pm Revision 9dbeb1f3 (arvados): 11220: Avoid implying that manifests can be loaded from Keep.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
03:31 pm Revision 60887cc6 (arvados): 11220: Fix test case for new API call sequence.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
03:31 pm Revision d9666e97 (arvados): 11220: Remove test for unsupported case.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
01:23 pm Bug #12515 (In Progress): Workbench base URL for multi-site search results should be configurable
API server config already has this config. Need to add it to discovery doc, and have Workbench fetch discovery doc to... Tom Clegg
07:58 am Revision fb7a5ed1 (arvados): 12494: Add CORS test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg

10/31/2017

09:32 pm Feature #12167: [SDKs] [API] [keepstore] [keepproxy] Facilitate request tracing for arv-put
Workbench and API server portions:
12167-workbench-request-id @ commit:5f46fe83d8a15048c4a5b90553c379803e99b135
Tom Clegg
09:31 pm Revision 5f46fe83 (arvados): 12167: Include a request ID in each API response and log entry.
Use the value of the X-Request-Id request header if given, otherwise
generate one.
Arvados-DCO-1.1-Signed-off-by: To...
Tom Clegg
09:27 pm Bug #12318 (In Progress): Honor Retry-After headers on libcloud exceptions
Lucas Di Pentima
09:11 pm Feature #12018: Synchronize group membership with external data source
Updates at:8fc7e0dd5
* Added API server start/stop calls to test suite.
* Removed manual database cleanup in favo...
Lucas Di Pentima
08:00 pm Feature #12018: Synchronize group membership with external data source
For the tests, you should call arvadostest.StartAPI() instead of assuming the environment.
Instead of adding a bun...
Peter Amstutz
02:19 am Feature #12018: Synchronize group membership with external data source
Updates at commit:9b83c3ee1
* Enhanced readability of @ProcessFile()@ function.
* Added test using usernames inst...
Lucas Di Pentima
09:07 pm Revision 8fc7e0dd (arvados): 12018: Added API server start/stop calls to test suite.
Removed manual database cleanup in favour of using the database
reset endpoint after every test run.
Moved tool confi...
Lucas Di Pentima
08:37 pm Revision 0e12d049 (arvados): 12167: Improve request-id tracking in Workbench.
Include current request ID in API requests via X-Request-Id header
instead of a form parameter. (Also move Arvados A...
Tom Clegg
07:31 pm Revision fc63eab4 (arvados): 12462: Treat "/" and "-" as word chars, too.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
07:12 pm Story #12527 (In Progress): [Composer] Make pull request to upstream with run-as-webapp configura...
Make pull request to upstream Rabix project with changes needed to support run-as-webapp configuration.
Peter Amstutz
07:01 pm Feature #12526 (New): Setting priority field through arvados-cwl-runner
I would like a --priority flag that will pass a priority number to the rest of the pipeline, for new jobs that spawn. Bryan Cosca
06:48 pm Story #4019: [API] Support query of "properties" field on objects
Test for exact key-value:
["properties.mytag1", "=", "myvalue1"]
Test for existence of 'mytag1', any value, w...
Peter Amstutz
05:24 pm Feature #12524 (New): [Composer] js-git porcelain API
Provide a "porcelain" (highlevel) API for js-git. Basic operations: clone, add, commit, push.
Also investigate ...
Peter Amstutz
05:19 pm Feature #12523 (New): [Composer] New workflow / new command line tool buttons can be used
The "New workflow" and "New command line tool" buttons require selecting a "Destination path". The user must be able... Peter Amstutz
05:10 pm Feature #12522 (In Progress): [Composer] Browse Arvados git repos
The "My Projects" side bar should be populated with a list of the Arvados git repositories visible to the user.
Th...
Peter Amstutz
05:08 pm Feature #12521 (Resolved): [Composer] API server configuration option
Composer should have a configuration file with the address of the cluster's API server. Instead of prompting the use... Peter Amstutz
04:40 pm Revision 2778384c (arvados): 12377: Arvbox support for Composer
Peter Amstutz
04:20 pm Bug #9843 (Resolved): [Testing] FUSE tests sometimes hang
Tom Clegg
03:32 pm Story #12519 (Resolved): Multi-cluster project search
In the same way that we can search collections across cluster we would also like to be able to search for projects as... Tom Morris
02:27 pm Feature #12517: [Workbench] page with a view of all the top level container requests
My current workaround is to name pipelines after some heuristic like issue tracker number so I can filter on that. Ma... Bryan Cosca
02:24 pm Feature #12517 (New): [Workbench] page with a view of all the top level container requests
Similar to workbench.xxxxx.arvadosapi.com/pipeline instances, I would like a page that has top level container reques... Bryan Cosca
02:00 pm Revision 326c5107 (arvados): 12018: Set up fields per record, so it won't try to guess it from
the first one being read.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
10:53 am Bug #12475: keepproxy panic: send on closed channel
We have increased the number of keepproxy and keepstore services in our cluster which works around this issue, but it... Joshua Randall
02:17 am Revision 9b83c3ee (arvados): 12018: Enhanced readability of ProcessFile() function.
Added test using usernames instead of emails.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgen...
Lucas Di Pentima
01:26 am Revision ea2c3097 (arvados): While we are at it, also throw out the broken progress bar and replace
it with an indication that the workflow is running.
refs #12513
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvan...
Ward Vandewege
01:15 am Bug #12515 (Resolved): Workbench base URL for multi-site search results should be configurable
Currently the workbench URL is hardcoded to be https://workbench.foo.arvadosapi.com, which doesn't work if, for examp... Tom Morris
01:11 am Revision cd45d6a3 (arvados): Rip out a bunch of very very slow workbench code that doesn't seem to
serve much purpose.
refs #12513
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege

10/30/2017

10:50 pm Feature #12018: Synchronize group membership with external data source
Updates at commit:eb772d62e
* Link & LinkList types moved to Go SDK
* Enhanced error message when having a parsin...
Lucas Di Pentima
07:00 pm Feature #12018: Synchronize group membership with external data source
* The "Link" struct should be added to the "arvados" package.
The signature to ProcessFile() is 247 characters lon...
Peter Amstutz
02:40 pm Feature #12018: Synchronize group membership with external data source
Updates at commit:7327a28eb
* Membership removal fix
* Changed @-path <input-file>@ to be a positional argument
...
Lucas Di Pentima
10:42 pm Revision eb772d62 (arvados): 12018: Enhanced error message when having a parse error.
Added test cleanup.
Added test to confirm records with empty fields are ignored.
Arvados-DCO-1.1-Signed-off-by: Luca...
Lucas Di Pentima
09:25 pm Story #12512 (New): [Workbench] Implement X-Request-ID to enable request tracing
This is a rename of header which was implemented in #10029.
What was client_session_id/current_request_id should i...
Tom Morris
09:17 pm Revision 43d9a42f (arvados): 12018: Moved Link & LinkList types to GoSDK arvados package
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
09:11 pm Bug #12510: [API] Don't create an index if it exists already. Will break install if we manually a...
I'm not sure that having an index with the same name guarantees that it'll match the characteristics of the index in ... Tom Morris
09:01 pm Bug #12510 (New): [API] Don't create an index if it exists already. Will break install if we manu...
if the index is already there, we should have some logic to avoid the upgrade of the package to fail. also in postgre... Nico César
08:59 pm Story #11453: Federated user identity which works across a network of Arvados clusters
Peter Amstutz wrote:
> * When validating a remote user, @is_admin@ is forced to be false, but is_active is accepted ...
Tom Clegg
01:35 pm Story #11453: Federated user identity which works across a network of Arvados clusters
All new tokens given to the client should be v2/ format tokens so the client can tell if it is talking to the home cl... Peter Amstutz
08:58 pm Feature #12167 (In Progress): [SDKs] [API] [keepstore] [keepproxy] Facilitate request tracing for...
Tom Clegg
08:56 pm Revision 530b25ab (arvados): 11453: Do not generate local tokens for remote users.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com> Tom Clegg
08:39 pm Bug #11220: [SDKs] Fix misleading arv-mount/pysdk error messages by removing obsolete "fetch mani...
* Ran tests both on Jenkins and locally, I got some @services/fuse@ failures: https://ci.curoverse.com/job/developer-... Lucas Di Pentima
08:07 pm Bug #12509 (Closed): project-uuid doesn't propagate correctly on crunch v1 and crunch v2?
nvm the project-uuid is right... Bryan Cosca
08:06 pm Bug #12509 (Closed): project-uuid doesn't propagate correctly on crunch v1 and crunch v2?
https://workbench.e51c5.arvadosapi.com/container_requests/e51c5-xvhdp-5m88z6b3zz5fdgi#
https://workbench.e51c5.arv...
Bryan Cosca
05:22 pm Bug #12306: [arv-mount] --unmount should work on an unresponsive mount
These stuck mounts come up occasionally on Jenkins. When they do, all builds get stuck ("UnmountTest" -- presumably b... Tom Clegg
02:35 pm Revision 7327a28e (arvados): 12018: Fixed positional argument handling.
Fixed & added tests.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
01:19 pm Revision f8067ef4 (arvados): Merge branch '12462-search-periods'
refs #12462
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>
Tom Clegg

10/27/2017

09:52 pm Story #11453: Federated user identity which works across a network of Arvados clusters
@test 'remote api server is not an api server'@ should return status 200.
should bail out if 'uuid' or 'secret' is...
Peter Amstutz
06:48 pm Story #11453: Federated user identity which works across a network of Arvados clusters
* When validating a remote user, @is_admin@ is forced to be false, but is_active is accepted from the remote site. I... Peter Amstutz
06:46 pm Revision 36d2b2d6 (arvados): Migration for indexes on containers and container_requests.
refs #12500
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>
Peter Amstutz
01:30 pm Task #12487 (Resolved): Create minimal prototype to clone & push
Peter Amstutz
01:30 pm Task #12487 (In Progress): Create minimal prototype to clone & push
Peter Amstutz
01:30 pm Task #12499 (In Progress): Review 12494-arv-git-cors
Peter Amstutz
01:30 pm Task #12499 (Resolved): Review 12494-arv-git-cors
Peter Amstutz
01:30 pm Feature #12494 (In Progress): arv-git-httpd supports CORS headers
Peter Amstutz
01:38 am Story #12484: Prototype using js-git to access Arvados repo
js-git fork. Fixes a few things and adds git-send-pack:
https://github.com/curoverse/js-git/tree/composer
test...
Peter Amstutz

10/26/2017

10:00 pm Bug #12465: [crunchv2] Improve crunch-run environment reporting
While we're at it, we should fix the formatting of the existing logging. For example, the missing spaces after the "n... Tom Morris
09:18 pm Bug #12465: [crunchv2] Improve crunch-run environment reporting
Crunch v1 logs the following information:... Tom Morris
08:31 pm Revision 2da11ace (arvados): 12018: Changed the input file parameter to be positional.
Enhanced usage message.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
Lucas Di Pentima
08:12 pm Bug #12199: Don't schedule jobs on nodes which are bigger than requested
I think we need a configuration that matches arvados-node-manager available sizes. this should be a centralized confi... Nico César
07:49 pm Bug #12199: Don't schedule jobs on nodes which are bigger than requested
Peter Amstutz wrote:
> The core counts could be an array [1, 2, 4, 8, 16, 20] for rounding the requested core count ...
Ward Vandewege
07:16 pm Bug #12199: Don't schedule jobs on nodes which are bigger than requested
The core counts could be an array [1, 2, 4, 8, 16, 20] for rounding the requested core count up to the nearest size w... Peter Amstutz
07:04 pm Bug #12199: Don't schedule jobs on nodes which are bigger than requested
After brainstorming with Peter, I made a very crude, simple version of this in branch 12199-cost-control at commit:8d... Ward Vandewege
07:48 pm Revision 924c37b6 (arvados): Take 2
refs #12199
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>
Ward Vandewege
07:02 pm Revision 8dc4be85 (arvados): This is a crude first cut of a feature to effectively turn
min_cores_per_node into exact_cores_per_node. Azure specific.
Definitely not merge ready.
refs #12199
Arvados-DCO-...
Ward Vandewege
06:21 pm Revision a442ad28 (arvados): 12018: Create synchronized remote groups as "role" groups.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
04:41 pm Revision 0788ccee (arvados): 12389: ubuntu build fix for iteration
Arvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat@veritasgenetics.com> Fernando Monserrat
04:25 pm Revision 03e0ebb3 (arvados): 12018: Fixed membership removal call. Added input validation.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com> Lucas Di Pentima
02:11 pm Feature #12494: arv-git-httpd supports CORS headers
12494-arv-git-cors @ commit:1ff58b4325d1d0dcfc091742c64f1f87118b55ce Peter Amstutz
02:08 pm Feature #12494 (Resolved): arv-git-httpd supports CORS headers
Peter Amstutz
02:08 pm Revision 1ff58b43 (arvados): 12494: Add CORS header support to arv-git-httpd
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com> Peter Amstutz
01:26 pm Revision 12c1e654 (arvados): 12389: verbose output for ubuntu build
Arvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat@veritasgenetics.com> Fernando Monserrat
 

Also available in: Atom