Activity

From 03/25/2017 to 04/23/2017

04/21/2017

09:25 pm Revision f1fb3915 (arvados): 8019: rateLimit crunch-run logging using API configuration parameters
Radhika Chippada
06:48 pm Revision f4661a02 (arvados): 11507: Put self._pending_write_size update back under the lock.
Peter Amstutz
06:31 pm Revision 52c6f13d (arvados): 11507: More small block packing fixes:
* Repack writes before small block merging.
* Update segments of file with new locator/segment offset instead of repl...
Peter Amstutz
01:55 pm Revision 0ecea550 (arvados): 11507: Cleanup
Peter Amstutz
02:40 am Task #11506 (Resolved): Review 11502-unstripped-manifest-fix
Lucas Di Pentima

04/20/2017

09:45 pm Revision 7116da15 (arvados): 11507: Put bufferblocks into DELETED state when deleted so they don'...
reused accidentally if there are dangling references. When packing small
blocks, update segments with the block hash...
Peter Amstutz
09:10 pm Revision bef5d901 (arvados): 11507: Test cases that reproduce bug
Peter Amstutz
07:36 pm Bug #11531 (New): [API] clean up stale/conflicting dns data from deleted node records
Problem scenario:
* node compute100 comes up with ip address = 10.2.3.4
* node record compute100 is deleted
* node...
Tom Clegg
07:36 pm Task #11484 (In Progress): Pass libcloud test suite
Peter Amstutz
07:36 pm Story #11350 (In Progress): [Nodemanager] Merge cloud_environment support to upstream libcloud
Peter Amstutz
06:28 pm Task #11528 (Resolved): Review 11510-sdk-extend-files
Peter Amstutz
03:50 pm Task #11528 (In Progress): Review 11510-sdk-extend-files
Tom Clegg
03:50 pm Task #11528 (Resolved): Review 11510-sdk-extend-files
Tom Clegg
06:18 pm Revision b0463827 (arvados): Merge branch '11510-sdk-extend-files' refs #11510
Peter Amstutz
06:17 pm Bug #11470: [API] container_requests#update alternately responds 422 or 404 for no apparent reason
11470-update-task-fields LGTM
Peter Amstutz
02:17 am Bug #11470: [API] container_requests#update alternately responds 422 or 404 for no apparent reason
Bugfix encountered by accident while investigating:
11470-update-task-fields @ commit:9aa83ad7b4de05dd2818885ed341...
Tom Clegg
06:15 pm Revision c39ba519 (arvados): 11510: Tweak interface to better match Pythons "io.IOBase" class.
Peter Amstutz
06:10 pm Revision 2333472a (arvados): Merge branch '11470-update-task-fields'
refs #11470 Tom Clegg
05:53 pm Bug #11510: [SDK] Support writes to offsets beyond end of file
Peter Amstutz wrote:
> > * test return value of seek() is new file position
>
> Also tested in test_truncate3().
...
Tom Clegg
05:28 pm Bug #11510: [SDK] Support writes to offsets beyond end of file
Tom Clegg wrote:
> 11510-sdk-extend-files @ commit:f5638851ddb5a859a54a16fe963ee98591af17a9
> * remove debugging pr...
Peter Amstutz
04:44 pm Bug #11510: [SDK] Support writes to offsets beyond end of file
11510-sdk-extend-files @ commit:f5638851ddb5a859a54a16fe963ee98591af17a9
* remove debugging print statements in arvf...
Tom Clegg
05:47 pm Task #11526 (In Progress): Review 11470-update-task-fields
Peter Amstutz
02:17 am Task #11526 (In Progress): Review 11470-update-task-fields
Tom Clegg
05:47 pm Task #11479 (New): Review
Peter Amstutz
05:47 pm Task #11479 (In Progress): Review
Peter Amstutz
05:25 pm Revision 5180238a (arvados): 11510: Fix truncate() to not change file pointer. Additional tests.
Peter Amstutz
04:24 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
There do generally seem to be race conditions with updating /etc/mtab by FUSE filesystems:
https://bugzilla.redhat...
Joshua Randall
04:07 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
Another set of machines were wedged today. Did some more testing on the call with Tom:
First, on humgen-02-02 we e...
Joshua Randall
03:19 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
kernel and system versions:... Joshua Randall
03:16 pm Revision bfc9660a (arvados): 11510: Repack writes any time there's more than one segment referenc...
Peter Amstutz
02:41 pm Revision 1d61934f (arvados): 11510: Repack writes any time there's more than one segment referenc...
Peter Amstutz
02:27 pm Revision aed7702a (arvados): 11510: Fix longstanding bug in replace_range() when appending data f...
Peter Amstutz
01:18 pm Task #11477 (Resolved): Review 11450-workflows-page
Radhika Chippada
02:16 am Revision 9aa83ad7 (arvados): 11470: Fix identifiers to conventional Go.
Tom Clegg
02:09 am Revision 3c34e713 (arvados): 11470: Fix accidentally cleared fields during job_task update.
Updating with a Task struct had the effect of clearing job_uuid,
created_by_job_task_uuid, and parameters fields, and...
Tom Clegg

04/19/2017

08:46 pm Story #11524 (New): Implement disk usage reporting for the administrator
The current report is no longer updated by the backend job that used to do it, so we need something to replace this. ... Tom Morris
08:45 pm Bug #11502 (Resolved): Flaky test on sdk/python/tests/test_arv_get.py
Applied in changeset arvados|commit:65e339856daf4b5c3a4a810cd3a5f1a8e386dc8c. Lucas Di Pentima
07:38 pm Bug #11502: Flaky test on sdk/python/tests/test_arv_get.py
Updates at commit:e7284afa8
Waiting for tests on jenkins to merge to master.
Lucas Di Pentima
06:35 pm Bug #11502: Flaky test on sdk/python/tests/test_arv_get.py
In @test_get_collection_unstripped_manifest@, how about replacing r"\+A[0-9a-f@]+" with "+Axxxxx" in both expected an... Tom Clegg
08:42 pm Revision 65e33985 (arvados): Merge branch '11502-unstripped-manifest-fix'
Closes #11502, #11511 Lucas Di Pentima
08:34 pm Bug #11523 (New): User activity page takes too long to load
This page takes 25 seconds to load, even on a relatively new cluster with few users. On cloud.curoverse.com it takes ... Tom Morris
08:32 pm Bug #11522 (New): Remove storage report page until it is fixed
There is a link on the admin page https://workbench.qr1hi.arvadosapi.com/users#Activity to https://workbench.e51c5.ar... Tom Morris
07:38 pm Revision 151df8c3 (arvados): 11502: Merge branch 'master' into 11502-unstripped-manifest-fix
Lucas Di Pentima
07:28 pm Revision e7284afa (arvados): 11502: Instead of asking for a stripped manifest, replace the tokens...
Lucas Di Pentima
07:05 pm Task #11506 (In Progress): Review 11502-unstripped-manifest-fix
Tom Clegg
07:04 pm Revision 82c40aa7 (arvados): Merge branch '11517-indexes'
refs #11517 Tom Clegg
06:00 pm Revision 38a4b3c4 (arvados): 11517: Add indexes to help slow queries.
Tom Clegg
05:44 pm Bug #11520 (New): "arv collection list" command parsing allows invalid lines
This command:... Tom Morris
05:38 pm Bug #11519 (New): arv-get should abort on ctrl/C
1423 MiB / 43967 MiB 3.2%^CTraceback (most recent call last):
File "/home/tfmorris/venv/local/lib/python2.7/site-p...
Tom Morris
03:30 pm Story #11450 (Resolved): [Crunch2][Workbench] Improve formatting of /workflow page
Applied in changeset arvados|commit:f79536fda9dc40f480383caa69a35663702b2ba4. Radhika Chippada
02:02 pm Story #11450: [Crunch2][Workbench] Improve formatting of /workflow page
This LGTM, thanks! Lucas Di Pentima
03:27 pm Revision f79536fd (arvados): closes #11450
Merge branch '11450-workflows-page' Radhika Chippada
03:27 pm Revision c14ae0ed (arvados): 11450: workflows index page
Radhika Chippada
02:56 pm Bug #11283 (In Progress): duplicate key value violates unique constraint "index_nodes_on_slot_nu...
11283-available-slot-query @ commit:3bb2699f6d418b508dd8bf35cec06f9f54ac6bba
Tom Clegg
02:44 pm Revision 3bb2699f (arvados): 11283: Fix "available slot number" query.
Fixes repetitive queries and excessive Postgres and Rails log
messages:
2017-04-10 16:39:09 UTC [4734-1] arvados@arv...
Tom Clegg
02:31 pm Feature #11515 (New): [CWL] [Workbench] Support for copying/moving pipelines
A common practice for writing a pipeline is to first test it in a private project, e.g. Home, because we don't want t... Jiayong Li
04:33 am Bug #11459 (Resolved): SSO packages for Centos7 don't build
Tom Clegg

04/18/2017

11:50 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
Found one last node that is wedged, and managed to do some more diagnosing. It looks like when it is wedged on our sy... Joshua Randall
11:31 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
Another wedged node. On this one, neither --unmount-all nor --unmount worked until after I attempted to mount at the ... Joshua Randall
11:25 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
Another wedged node, this time arv-mount --unmount did work:... Joshua Randall
11:20 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
I've just been looking through the code to try to figure this out and it looks like the issue is that the wedged moun... Joshua Randall
10:53 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
I now have a wedged arv-mount on one of my compute nodes on which I have the new arv-mount.
Unfortunately, the ne...
Joshua Randall
09:54 pm Revision f5638851 (arvados): 11510: Fix longstanding bug in replace_range() when appending data f...
buffer that was written out of order. Peter Amstutz
08:40 pm Revision 04951581 (arvados): 11510: Add support for sparse writes to ArvadosFile.
Allow seeking past end of file. Allow using truncate() to extend file size. Add zero padding to fill in gaps. Peter Amstutz
08:35 pm Revision 5e4bf683 (arvados): 8019: expose crunch log throttling config parameters via discovery d...
Radhika Chippada
07:18 pm Task #11504 (Resolved): review 11209-crunch-unmount-all
Tom Clegg
07:15 pm Revision de99c0b2 (arvados): 11459: Move RAILS_PACKAGE_ITERATION to run-library.
Otherwise, the SSO build script cannot see it.
refs #11459
Tom Clegg
06:48 pm Story #9582 (Resolved): [Workbench] Don't display trashed collections in regular collection listings
The API server automatically filters out trashed collections, so this is effectively done. Tom Morris
05:15 pm Bug #11509 (Resolved): [Keep-web] Support CORS requests with Range headers
Applied in changeset arvados|commit:e3ac17f8a8aa439e21a8bf56a571f91a671313f7. Tom Clegg
04:40 pm Bug #11509: [Keep-web] Support CORS requests with Range headers
Just a nit about the TestCORSPreflight. It would be nice to add a comment for each block doing "h.ServeHTTP(resp, req... Radhika Chippada
05:12 pm Task #11512 (Resolved): Review 11509-keep-web-cors-range
Tom Clegg
04:40 pm Task #11512 (In Progress): Review 11509-keep-web-cors-range
Radhika Chippada
05:02 pm Bug #11502: Flaky test on sdk/python/tests/test_arv_get.py
New updated at commit:67ba19113
Test run: https://ci.curoverse.com/job/developer-run-tests/240/
As I've unwilling...
Lucas Di Pentima
03:37 pm Bug #11502: Flaky test on sdk/python/tests/test_arv_get.py
Updates at branch @11502-unstripped-manifest-fix@ - commit:03188ad6e
Test run: https://ci.curoverse.com/job/develope...
Lucas Di Pentima
02:18 pm Bug #11502: Flaky test on sdk/python/tests/test_arv_get.py
Reviewing the old code for arv-get, the behavior was to call CollectionReader.manifest_text() which is unnormalized/u... Peter Amstutz
02:02 pm Bug #11502 (Feedback): Flaky test on sdk/python/tests/test_arv_get.py
Peter Amstutz
05:01 pm Revision e3ac17f8 (arvados): Merge branch '11509-keep-web-cors-range'
closes #11509 Tom Clegg
05:00 pm Revision 244c4743 (arvados): 11509: Add comments.
Tom Clegg
04:45 pm Revision 67ba1911 (arvados): 11502: Use os.path.join instead of formatting path-like strings on t...
Lucas Di Pentima
04:36 pm Revision 9609f9a5 (arvados): 11502: Avoid races on test by getting the unstripped manifest versio...
Lucas Di Pentima
03:31 pm Revision 03188ad6 (arvados): 11502: Fix regression by writing a collection manifest with its acce...
Added --strip-manifest argument to allow writing manifests without tokens.
Updated tests accordingly.
Lucas Di Pentima
03:25 pm Revision 692a66af (arvados): Fix ignored symlink failure.
No issue # Tom Clegg
01:57 pm Revision 243701b6 (arvados): refs #11465
Merge branch '11465-disabled-collection-file-tooltips' Radhika Chippada
01:56 pm Revision ea40bcae (arvados): 11465: tooltip on disabled pencil icon
Radhika Chippada
01:37 pm Bug #11510 (In Progress): [SDK] Support writes to offsets beyond end of file
Peter Amstutz

04/17/2017

11:56 pm Task #11512 (Resolved): Review 11509-keep-web-cors-range
Tom Clegg
11:55 pm Bug #11509: [Keep-web] Support CORS requests with Range headers
11509-keep-web-cors-range @ commit:cf311e8e16ba74467c77b5353afedc29b40a6a41 Tom Clegg
07:23 pm Bug #11509 (In Progress): [Keep-web] Support CORS requests with Range headers
Tom Clegg
07:15 pm Bug #11509 (Resolved): [Keep-web] Support CORS requests with Range headers
h3. Background
The Workbench log viewer uses an ajax request to retrieve log data. It uses the POST method so it c...
Tom Clegg
11:50 pm Revision cf311e8e (arvados): 11509: Allow cross-origin requests with Range headers.
Tom Clegg
09:09 pm Bug #11510: [SDK] Support writes to offsets beyond end of file
So apparently this is POSIX behavior for creating sparse files, and the Arvados Python SDK (and hence FUSE) currently... Peter Amstutz
08:55 pm Bug #11510: [SDK] Support writes to offsets beyond end of file
... Peter Amstutz
08:31 pm Bug #11510 (In Progress): [SDK] Support writes to offsets beyond end of file
Two distinct but closely related issues encountered by a customer:
# Using s3.download_fileobj() with an ArvadosF...
Peter Amstutz
08:17 pm Task #11476 (Resolved): Review 11465-disabled-collection-file-tooltips
Radhika Chippada
07:20 pm Story #11465 (Resolved): [Workbench] In collection#show page, display a tooltip on disabled trash...
Applied in changeset arvados|commit:20364bc1aed2dcad71b006f4314ba0720d20ae05. Radhika Chippada
07:14 pm Story #11465: [Workbench] In collection#show page, display a tooltip on disabled trash icon, disa...
LGTM, thanks! Lucas Di Pentima
05:42 pm Story #11465: [Workbench] In collection#show page, display a tooltip on disabled trash icon, disa...
> When locked, the tooltip works fine on Chrome, but not (at least for me) on Safari ...
I can't test in Safari in...
Radhika Chippada
02:09 pm Story #11465: [Workbench] In collection#show page, display a tooltip on disabled trash icon, disa...
* When locked, the tooltip works fine on Chrome, but not (at least for me) on Safari. Tried changing the @<span>@ to ... Lucas Di Pentima
01:14 pm Story #11465: [Workbench] In collection#show page, display a tooltip on disabled trash icon, disa...
> Although when hovering the buttons with the mouse it shows the tooltip and the prohibited sign, it’s accepting my c... Radhika Chippada
07:17 pm Revision 20364bc1 (arvados): closes #11465
Merge branch '11465-disabled-collection-file-tooltips' Radhika Chippada
05:34 pm Revision 72a377c5 (arvados): 11465: correct disabled tooltip behavior on image types
Radhika Chippada
03:44 pm Feature #11508 (New): HTTP 503 status for planned downtime
In order to facilitate upgrade without downtime, designate 503 (Service Unavailable) to indicates "planned downtime".... Peter Amstutz
02:12 pm Task #11477 (In Progress): Review 11450-workflows-page
Lucas Di Pentima
02:01 pm Revision fc2eaa20 (arvados): Fix crunch-run tests.
refs #11209 Tom Clegg
01:20 am Revision bf55ee80 (arvados): 11465: add tooltips to disabled collection file controls
Radhika Chippada
12:12 am Revision 45847ed8 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
12:11 am Revision 60d9f791 (arvados-dev): Merge branch '00000-a-v-s-cleanups'
No issue # Ward Vandewege
12:02 am Revision f75eaa03 (arvados-dev): Merge branch 'master' into 00000-a-v-s-cleanups
Ward Vandewege

04/14/2017

08:43 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
commit:d64ed33e94700f8204ec8089c7b235cff918f9f7 LGTM. Lucas Di Pentima
05:25 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
11209-crunch-unmount-all @ commit:d64ed33e94700f8204ec8089c7b235cff918f9f7... Tom Clegg
08:42 pm Revision 9cab6a09 (arvados): Merge branch '11209-crunch-unmount-all'
refs #11209 Tom Clegg
08:30 pm Bug #11502 (Resolved): Flaky test on sdk/python/tests/test_arv_get.py
Applied in changeset arvados|commit:cbcb0fc8c3ebc85bf81ba9d50795d62db75efa6c. Lucas Di Pentima
08:09 pm Bug #11502: Flaky test on sdk/python/tests/test_arv_get.py
Discovered a regression on the new arv-get, when asked to retrieve a manifest using a PDH, its output was a non-strip... Lucas Di Pentima
03:54 pm Bug #11502: Flaky test on sdk/python/tests/test_arv_get.py
As Peter noted, the comparison on the test should be done with stripped manifests. Lucas Di Pentima
03:54 pm Bug #11502 (In Progress): Flaky test on sdk/python/tests/test_arv_get.py
Lucas Di Pentima
03:53 pm Bug #11502 (Resolved): Flaky test on sdk/python/tests/test_arv_get.py
... Lucas Di Pentima
08:26 pm Revision cbcb0fc8 (arvados): Merge branch '11502-arvget-flaky-test'
Closes #11502 Lucas Di Pentima
08:16 pm Revision a501e1d8 (arvados): 11308: Stop if fuse.h or gnutls.h sanity check fails.
Tom Clegg
08:12 pm Revision b9d004b5 (arvados): 11308: Add sanity check for python3-dev.
Tom Clegg
08:09 pm Task #11506 (Resolved): Review 11502-unstripped-manifest-fix
Lucas Di Pentima
08:04 pm Revision e507e67e (arvados): 11502: When getting a manifest using a PDH, the manifest is stripped.
Corrected test so that when comparing manifests, it does so by comparing stripped ones so that it doesn't intermitten... Lucas Di Pentima
07:34 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
11308-python3 @ commit:a501e1d874f01acc318e7963837fbeb58a29f744 https://ci.curoverse.com/job/developer-run-tests/236/ Tom Clegg
07:16 pm Task #11504 (Resolved): review 11209-crunch-unmount-all
Tom Clegg
07:16 pm Task #11353 (Resolved): use arv-mount --unmount-all in crunch-job
Tom Clegg
07:12 pm Revision f2c1db34 (arvados): 11308: Fix bytes vs. strings in arv-get command and tests.
Tom Clegg
07:12 pm Revision 84737fbd (arvados): 11308: Remove unused import.
Tom Clegg
07:12 pm Revision ebc9ff19 (arvados): 11308: Futurize test_arv_get.py and commands/get.py
Tom Clegg
07:11 pm Revision db7e17f8 (arvados): 11308: Fix whitespace
Tom Clegg
07:11 pm Revision 0c5cb575 (arvados): 11308: Restore dependency lost in merge.
Tom Clegg
05:34 pm Story #11450 (In Progress): [Crunch2][Workbench] Improve formatting of /workflow page
Radhika Chippada
05:32 pm Story #11450: [Crunch2][Workbench] Improve formatting of /workflow page
Branch 11450-workflows-page @ commit:2113a518
* Added /workflows page with Run and Show options
* Manually veri...
Radhika Chippada
05:29 pm Revision 2113a518 (arvados): 11450: workflows index page
Radhika Chippada
05:19 pm Revision d64ed33e (arvados): 11209: Use arv-mount --unmount-all instead of mount|awk|xargs script...
Tom Clegg
03:17 pm Revision 88c26c6b (arvados): Merge branch '11462-cwl-cache-collectionreader' refs #11462
Peter Amstutz
03:08 pm Revision 91f1e700 (arvados): 11462: Log ArgumentError in make_output_collection. Add test for Co...
Peter Amstutz
02:35 pm Revision 2001423a (arvados): Fix regression, restore a line that shouldn't have been removed. ref...
Peter Amstutz
01:59 pm Story #11465: [Workbench] In collection#show page, display a tooltip on disabled trash icon, disa...
Although when hovering the buttons with the mouse it shows the tooltip and the prohibited sign, it’s accepting my cli... Lucas Di Pentima
01:20 pm Task #11476 (In Progress): Review 11465-disabled-collection-file-tooltips
Lucas Di Pentima
12:27 am Bug #11495: bcbio NA12878 validation runs: job re-use failure with non-existant collection
Thanks Bryan for the digging on this. I didn't intentionally delete any log files but am not sure if the failed retri... Brad Chapman

04/13/2017

09:40 pm Revision 3a88a5d8 (arvados): 11462: Fix crunch script for fetcher_constructor constructor change.
Peter Amstutz
09:02 pm Revision defb299d (arvados): Add missing test file. refs #10401
Peter Amstutz
07:54 pm Revision 5152e0bf (arvados): 11462: Runner asks for 3000 MB of RAM by default instead of 1024.
Peter Amstutz
07:48 pm Revision a3488ff3 (arvados): 11462: Add missing test file.
Peter Amstutz
07:46 pm Revision 574dc168 (arvados): 11462: Bump cwltool version for mergedirs() fix
Peter Amstutz
07:41 pm Story #11450: [Crunch2][Workbench] Improve formatting of /workflow page
Using the same icons as the crunch1 equivalents sounds like the right call to me. Tom Morris
07:26 pm Story #11450: [Crunch2][Workbench] Improve formatting of /workflow page
Just noticed that we haven't yet configured a font icon for workflows and container_requests and using the default fa... Radhika Chippada
07:27 pm Story #11465: [Workbench] In collection#show page, display a tooltip on disabled trash icon, disa...
Branch 11465-disabled-collection-file-tooltips
Tests passed @ https://ci.curoverse.com/job/developer-run-tests/233/
Radhika Chippada
03:28 pm Story #11465 (In Progress): [Workbench] In collection#show page, display a tooltip on disabled tr...
Radhika Chippada
07:13 pm Revision da32ae2a (arvados): 11462: Store CollectionReader objects in a central cache to avoid re...
fetches & storing multiple copies. On pipeline with large inputs, reduced
memory usage by up to 70%.
Peter Amstutz
06:20 pm Revision 2f06f0e4 (arvados): 11465: add tooltips to disabled collection file controls
Radhika Chippada
05:07 pm Bug #11495: bcbio NA12878 validation runs: job re-use failure with non-existant collection
It's a missing log file.
2017-04-13 09:17:03 arvados.cwl-runner INFO: [job alignment_to_rec] reused job qr1hi-8i9s...
Bryan Cosca
01:46 pm Bug #11495 (New): bcbio NA12878 validation runs: job re-use failure with non-existant collection
Due to failures reported in #11494 I tried to re-run the bcbio CWL validation pipeline with re-use enabled and I get ... Brad Chapman
03:40 pm Revision 29d77300 (arvados): Merge branch '11493-cwl-six' refs #11493
Peter Amstutz
03:13 pm Revision af78d389 (arvados): 11493: Bump cwltool version for relaxed dependency on 'six'
Peter Amstutz
01:43 pm Bug #11494 (New): bcbio NA12878 validation runs: failed steps due to being unable to setup container
I'm running bcbio CWL validation workflows on NA12878 chr20 in preparation for the GA4GH workflow challenge:
https...
Brad Chapman

04/12/2017

08:19 pm Task #11485 (Closed): Review
Radhika Chippada
08:13 pm Task #11485 (Closed): Review
Tom Morris
08:10 pm Revision 0035033f (arvados): Merge branch '8567-moar-docker' refs #8567
Peter Amstutz
08:05 pm Bug #4124: [Crunch] Socket timed out on send/recv operation causes pipeline failure
The instances found in the production logs are a year old, so marking this as obsolete/closed. Tom Morris
08:04 pm Bug #4124 (Resolved): [Crunch] Socket timed out on send/recv operation causes pipeline failure
Tom Morris
08:05 pm Task #11484 (In Progress): Pass libcloud test suite
Peter Amstutz
08:04 pm Task #11483 (New): Review
Tom Clegg
08:03 pm Task #11482 (New): Review
Tom Clegg
08:03 pm Task #11481 (New): Review
Tom Clegg
08:02 pm Task #11479 (New): Review
Tom Clegg
08:02 pm Task #11478 (New): Review
Radhika Chippada
08:01 pm Task #11477 (Resolved): Review 11450-workflows-page
Radhika Chippada
08:01 pm Task #11476 (Resolved): Review 11465-disabled-collection-file-tooltips
Radhika Chippada
07:50 pm Task #11472 (New): Review
Lucas Di Pentima
07:50 pm Task #11471 (New): Review
Lucas Di Pentima
07:42 pm Bug #11470 (New): [API] container_requests#update alternately responds 422 or 404 for no apparent...
Seems like a race condition -- perhaps related to permission lookups or other container request updates that are happ... Tom Clegg
07:30 pm Story #11468 (New): [VG] Automated download from Azure / upload to Keep
Tom Morris
07:21 pm Story #11465 (Resolved): [Workbench] In collection#show page, display a tooltip on disabled trash...
Radhika Chippada
07:14 pm Task #6246 (Closed): Add test case for dns configs
Lucas Di Pentima
07:14 pm Bug #11436 (Resolved): Jenkins failing to install pycurl 7.43.0
run-tests.sh now checks for gnutls.h Peter Amstutz
07:13 pm Task #6245 (Closed): Improve race and exception handling in dns updates
Lucas Di Pentima
07:11 pm Task #11446 (Resolved): Update arvados-cwl-runner
Radhika Chippada
07:10 pm Task #11384 (Resolved): Review branch 11365-collection-lock-button
Radhika Chippada
07:10 pm Task #11447 (Resolved): update wiki
Tom Clegg
07:10 pm Task #11430 (Resolved): Review 8104-pycurl-721
Peter Amstutz
07:10 pm Task #11194 (Resolved): Review 10401-limit-dir-expansion
Tom Clegg
07:01 pm Revision 0b8673b6 (arvados): 11308: Merge branch 'master' into 11308-python3
Conflicts:
sdk/python/arvados/__init__.py
sdk/python/arvados/api.py
sdk/python/arvados/commands/ls.py
sdk/python/...
Tom Clegg
06:50 pm Story #10401 (Resolved): [CWL] Limit expansion of Directory inputs
Applied in changeset arvados|commit:49a841a5179307f0c8d84f647a71f44fb2b4b26d. Peter Amstutz
06:36 pm Story #10401: [CWL] Limit expansion of Directory inputs
LGTM Tom Clegg
04:26 pm Story #10401: [CWL] Limit expansion of Directory inputs
Tom Clegg wrote:
> Code LGTM.
>
> (from discussion offline)
> * should add a test.
> * cwl spec PR should still...
Peter Amstutz
03:29 pm Story #10401: [CWL] Limit expansion of Directory inputs
Code LGTM.
(from discussion offline)
* should add a test.
* cwl spec PR should still happen (but not necessarily...
Tom Clegg
02:09 pm Story #10401: [CWL] Limit expansion of Directory inputs
Merged master / tests fixed, now @ commit:76e3d80bcc1548daba7bca97711684e5d68c1624 Peter Amstutz
06:45 pm Revision 49a841a5 (arvados): Merge branch '10401-limit-dir-expansion' closes #10401
Peter Amstutz
06:38 pm Revision bafff694 (arvados): 8567: Migrate unnamed images
Peter Amstutz
06:37 pm Revision cb47df6c (arvados): Merge branch '11341-arvput-resume-error'
Closes #11341 Lucas Di Pentima
06:30 pm Feature #8465 (Resolved): [Crunch2] Support stdin/stderr redirection
Applied in changeset arvados|commit:a734789218122d8ab0d8f766bac4d69c04db91bf. Peter Amstutz
06:27 pm Feature #8465: [Crunch2] Support stdin/stderr redirection
Branch 8465-cwl-containers-stdin-stderr LGTM Radhika Chippada
06:27 pm Revision a7347892 (arvados): Merge branch '8465-cwl-containers-stdin-stderr' closes #8465
Peter Amstutz
06:26 pm Task #11448 (Resolved): Review 8465-cwl-containers-stdin-stderr
Radhika Chippada
05:35 pm Story #11349 (Resolved): [Node Manager] Add status URL for node manager
Tom Clegg
04:50 pm Story #11365 (Resolved): [Workbench] Add lock button for collection display to enable rename & de...
Applied in changeset arvados|commit:00573cf7d28472bf926e8f610a256cd991879c8b. Radhika Chippada
04:14 pm Story #11365: [Workbench] Add lock button for collection display to enable rename & delete
Updates @ commit:022d82146b1f004cec158d9801fecd8a17637e47 LGTM, thanks! Lucas Di Pentima
04:42 pm Revision 00573cf7 (arvados): closes #11365
Merge branch '11365-collection-lock-button' Radhika Chippada
04:34 pm Revision 4431b9c7 (arvados): 11341: Fixed log message
Lucas Di Pentima
04:32 pm Revision f61c7d94 (arvados): 11341: Added additional information log for cases when the user ask ...
Lucas Di Pentima
04:24 pm Revision 9ea73e2a (arvados): 10401: Document LoadListingRequirement in Arvados extensions.
Peter Amstutz
04:22 pm Revision 22409849 (arvados): 11365: "lock" collection to prevent user from deleting / renaming fi...
Radhika Chippada
04:08 pm Revision d3a82af3 (arvados): 10401: Add integration tests for directory listing behavior.
Peter Amstutz
04:07 pm Revision f70c457c (arvados): 11341: Use NamedTemporaryFile instead of mkstemp for cache saving.
Added logging messages to inform the user what's happening with the cache file. Lucas Di Pentima
04:02 pm Revision 022d8214 (arvados): 11365: lock button appearance improvements
Radhika Chippada
04:02 pm Revision 06e1c231 (arvados): 11308: Remove unused import.
Tom Clegg
03:39 pm Revision 1b290e51 (arvados): 11349: Fix section name in example configs.
refs #11349 Tom Clegg
02:38 pm Bug #11460 (New): [SDK] avoid interfering with socket open/close - use pycurl sockoptfunction ins...
Tom Clegg
02:33 pm Revision cb4f8b33 (arvados): 8465: Add unit test for generating container request with std* redir...
Peter Amstutz
02:19 pm Bug #11237 (Resolved): python-arvados-python-client depends on old versions of packages
Peter Amstutz
02:12 pm Revision d72f1d79 (arvados): Merge branch 'master' into 10401-limit-dir-expansion
Peter Amstutz
01:38 pm Task #11382 (Resolved): Review 6304-dns-update-fix
Lucas Di Pentima
01:34 pm Bug #11457 (Resolved): [Arvbox] ci.commonwl.org failing due to SSO server not starting
Green!
http://ci.commonwl.org/job/arvados-conformance/596/
Peter Amstutz
01:33 pm Bug #11459 (Resolved): SSO packages for Centos7 don't build
https://ci.curoverse.com/job/build-packages-sso/38/console... Peter Amstutz
04:20 am Revision 51655962 (sso-provider): Update Gemfile.lock. refs #11457
Tom Clegg

04/11/2017

11:03 pm Revision c03283d4 (arvados): cwl/test_with_arvbox.sh pulls arvados/jobs:latest refs #11457
Peter Amstutz
10:34 pm Revision e1bc4307 (arvados): Build SSO packages for Ubuntu 1604. refs #11457
Peter Amstutz
09:46 pm Revision e4196ce8 (sso-provider): Add test-unit gem to workaround broken Rails dependency. refs ...
Peter Amstutz
09:44 pm Bug #11457: [Arvbox] ci.commonwl.org failing due to SSO server not starting
Seems to be https://github.com/rspec/rspec-rails/issues/1273 Peter Amstutz
09:35 pm Bug #11457 (Resolved): [Arvbox] ci.commonwl.org failing due to SSO server not starting
arvbox localdemo
SSO server fails to initialize/start with this error:...
Peter Amstutz
08:30 pm Story #11365: [Workbench] Add lock button for collection display to enable rename & delete
Some cosmetic questions:
* Seems to me that the lock/unlock icon is too small in comparison to the buttons on its ...
Lucas Di Pentima
07:19 pm Story #11365: [Workbench] Add lock button for collection display to enable rename & delete
Branch 11365-collection-lock-button implements:
* Add a "lock" button for collection#show page with confirmation d...
Radhika Chippada
07:56 pm Revision 76e3d80b (arvados): 10401: Get structure.sql from master.
Peter Amstutz
07:55 pm Revision 777e7167 (arvados): Merge branch 'master' into 10401-limit-dir-expansion
Peter Amstutz
07:35 pm Story #11454 (New): Support federated search across a set of Arvados clusters
System administrator can configure a set of API servers which are part of a federation and which should be contacted ... Tom Morris
07:24 pm Story #11453 (New): Federated user identity which works across a network of Arvados clusters
Basic elements:
- a single login server which provides authentication for all clusters in the network
- a single us...
Tom Morris
07:21 pm Task #11384 (In Progress): Review branch 11365-collection-lock-button
Lucas Di Pentima
07:08 pm Story #11452 (New): Support rendering of CWL description fields with embedded markup
There is currently no standardized markup language for CWL descriptions, but it seems unlikely that if one is defined... Tom Morris
07:02 pm Task #1671 (Closed): Convert markdown to textile
Tom Morris
06:57 pm Story #11450 (Resolved): [Crunch2][Workbench] Improve formatting of /workflow page
Remove non-functional checkbox
Add Run button
Remove definition
Make sure that paging or infinite scrolling work (...
Tom Morris
06:52 pm Revision a6be53f6 (arvados): Build packages for python "future" module.
refs #11349
refs #11308
Tom Clegg
06:51 pm Revision 69a23af5 (arvados): 11365: tweak js for lock button
Radhika Chippada
06:34 pm Revision 95ef2a6f (arvados): refs #8465
Merge branch '8465-stderr-redirection' Radhika Chippada
06:12 pm Feature #10988 (New): [DOC] update documentation to list ubuntu1604
Tom Morris
06:12 pm Feature #10988 (Resolved): [DOC] update documentation to list ubuntu1604
Tom Morris
06:08 pm Revision 8be16cfc (arvados): Merge branch 'master' into 8465-stderr-redirection
Radhika Chippada
05:40 pm Task #11448 (Resolved): Review 8465-cwl-containers-stdin-stderr
Peter Amstutz
05:40 pm Task #11446 (In Progress): Update arvados-cwl-runner
Peter Amstutz
01:51 pm Task #11446 (Resolved): Update arvados-cwl-runner
Peter Amstutz
05:40 pm Revision 828161a7 (arvados): 8465: arvados-cwl-runner support for stdin and stderr redirection wi...
Peter Amstutz
05:12 pm Task #11264 (Resolved): Review 7709-api-rails4
Lucas Di Pentima
05:12 pm Feature #7709: [API] Upgrade API server to Rails 4.2
Sorry for the delay, lgtm! Lucas Di Pentima
02:47 pm Feature #7709: [API] Upgrade API server to Rails 4.2
Yes, that fixed it. We were doing that in the User unit tests but never did it in the controller tests.
Replaced t...
Tom Clegg
04:55 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
* @sdk/python/tests/run_test_server.py:3@ - @viewkeys@ is not used, is the import needed?
* Why the use of @viewkeys...
Lucas Di Pentima
02:04 am Story #11308: Support Python3 for arvados-python-client & command line utilities
Changed most of the list(...) stuff to viewkeys(), listitems(), etc.
> * @sdk/python/tests/run_test_server.py:740@...
Tom Clegg
04:34 pm Revision eee5ba33 (arvados): 11365: accept alert in tests with some wait time
Radhika Chippada
03:43 pm Task #11433 (Resolved): Review branch 8465-stderr-redirection
Peter Amstutz
03:43 pm Task #11385 (Resolved): Review branch 8465-stdin-redirection
Peter Amstutz
03:42 pm Feature #8465: [Crunch2] Support stdin/stderr redirection
8465-stderr-redirection @ commit:c1ea9a2fdccc5e24554f8cf28296cbc0764c8bbc LGTM, please merge Peter Amstutz
03:42 pm Task #11447 (Resolved): update wiki
Tom Clegg
03:34 pm Task #11353 (In Progress): use arv-mount --unmount-all in crunch-job
Tom Clegg
03:33 pm Task #11375 (Resolved): Review 11349-nodemanager-status-api
Tom Clegg
03:33 pm Revision 2c094e28 (arvados): Merge branch '11349-nodemanager-status-api'
refs #11349 Tom Clegg
03:32 pm Story #11349: [Node Manager] Add status URL for node manager
LGTM, thanks! Lucas Di Pentima
03:18 pm Story #11349: [Node Manager] Add status URL for node manager
Lucas Di Pentima wrote:
> Local test run was successful
> * @services/nodemanager/arvnodeman/status.py@ - Do you th...
Tom Clegg
02:32 pm Story #11349: [Node Manager] Add status URL for node manager
Local test run was successful
Questions:
* @services/nodemanager/arvnodeman/status.py@ - Do you think it would be...
Lucas Di Pentima
03:17 pm Revision a7793826 (arvados): 11349: Move assertion outside loop for clarity.
Tom Clegg
03:16 pm Revision 26eb8f4c (arvados): 11349: Warn if management/status server is not enabled.
Tom Clegg
02:44 pm Revision de083a9f (arvados): 7709: Clear deliveries before each test. Fixes flaky test.
Tom Clegg
02:44 pm Revision f5d09a49 (arvados): 7709: Remove Ruby websocket server.
Tom Clegg
02:39 pm Revision cb8407c0 (arvados): 11365: collection_upload_test updated to click on "unlock" button be...
Radhika Chippada
02:38 pm Story #10111 (In Progress): [Workbench][Crunch2] Provenance graph for Container Request
Lucas Di Pentima
01:35 pm Bug #6304 (Resolved): [API] Fix config file race condition and improve test cases for DNS update ...
Applied in changeset arvados|commit:be2a34c57eab7cc184c08ec1059554cb97911276. Lucas Di Pentima
01:33 pm Revision be2a34c5 (arvados): Merge branch '6304-dns-update-fix'
Closes #6304 Lucas Di Pentima
01:32 pm Revision 533bf8f1 (arvados): 6304: Change assertion type for better failure message.
Lucas Di Pentima
02:04 am Revision f6b16b5a (arvados): 11308: Fix bytes vs. strings in test case.
Tom Clegg
01:53 am Revision a603d0f7 (arvados): 11308: Remove superfluous parens.
Tom Clegg
01:53 am Revision 83f3b043 (arvados): 11308: Fix futurize misinterpretation.
Tom Clegg
01:53 am Revision 9b976cfb (arvados): 11308: Avoid Python2-inefficient list() operations.
Tom Clegg
01:53 am Revision 62be7d11 (arvados): 11308: Put back exception handling
Tom Clegg

04/10/2017

09:26 pm Revision 96facc8c (arvados): 11365: add integration test to click on lock / unlock and verify fil...
Radhika Chippada
09:02 pm Bug #6304: [API] Fix config file race condition and improve test cases for DNS update hooks
nit: prefer "assert_empty foo" to "assert foo.empty?" so the failure message says what unexpected stuff was found ins... Tom Clegg
06:02 pm Bug #6304: [API] Fix config file race condition and improve test cases for DNS update hooks
Updates at commit:f08a738f1565b2dabde28dc9e8d1bad9622d5529
Test run: https://ci.curoverse.com/job/developer-run-test...
Lucas Di Pentima
04:53 pm Bug #6304: [API] Fix config file race condition and improve test cases for DNS update hooks
Updates at commit:2f46fb576
Ensure dangling temp files are cleaned up.
Pending: test case for this is not working...
Lucas Di Pentima
08:56 pm Revision 1fad1bfb (arvados): 7709: Clear deliveries before each test. Fixes flaky test.
Tom Clegg
08:54 pm Feature #7709: [API] Upgrade API server to Rails 4.2
Maybe the @ActionMailer::Base.deliveries@ list is not cleared between tests, and if the tests are run in random order... Lucas Di Pentima
07:55 pm Feature #7709: [API] Upgrade API server to Rails 4.2
This happens sporadically... Tom Clegg
08:22 pm Revision 86f3f4fb (arvados-dev): Add --disable-reuse to run-cwl-test.sh/arvados-cwl-runner-with-c...
Peter Amstutz
08:04 pm Task #11375 (In Progress): Review 11349-nodemanager-status-api
Lucas Di Pentima
08:02 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
After lots of reading, this is my compiled list of things that may need a second look:
* @sdk/python/arvados/cache...
Lucas Di Pentima
07:45 pm Revision 96b6e267 (arvados): Secure link to google search.
No issue # Tom Clegg
06:57 pm Revision e9bf021d (arvados): 7709: Remove Ruby websocket server.
Tom Clegg
05:58 pm Revision f08a738f (arvados): 6304: Merge branch 'master' into 6304-dns-update-fix
Lucas Di Pentima
05:56 pm Revision 466e0e26 (arvados): 6304: Corrected test setup to make it pass.
Added a separator to the temp file so that its name is more readable. Lucas Di Pentima
05:33 pm Revision c1ea9a2f (arvados): Merge branch '8465-stdin-redirection' into 8465-stderr-redirection
Radhika Chippada
05:32 pm Revision f9a14680 (arvados): 8465: test cleanup
Radhika Chippada
05:07 pm Revision 0d5be4fa (arvados): 7709: Fix ActiveRecord::ImmutableRelation.
Tom Clegg
04:51 pm Revision 2f46fb57 (arvados): 6304: Clean up temp files that could exist in case of writing errors.
Also added a test for this case, not working properly atm. Lucas Di Pentima
04:37 pm Revision 77d9c05d (arvados): 7709: Merge branch 'master' into 7709-api-rails4
Conflicts:
services/api/app/controllers/arvados/v1/nodes_controller.rb
services/api/app/models/container_request.rb...
Tom Clegg
04:00 pm Revision 78ae065d (arvados): 11365: "remove selected files" dropdown selection + collection lock ...
Radhika Chippada
03:34 pm Feature #11442 (New): [SDK] Resolve symbolic names to uuid/pdh (cwl + other tools)
SDK method to resolve paths in the form of @groupname/projectname/collectionname/filename@ to UUID and PDH.
(Consi...
Peter Amstutz
02:59 pm Revision 909313b2 (arvados): Merge branch '11332-cwl-url-quoting' refs #11332
Peter Amstutz
02:58 pm Revision 4f68551a (arvados): 11332: Add @ to quote safelist.
Peter Amstutz
02:24 pm Story #11349: [Node Manager] Add status URL for node manager
11349-nodemanager-status-api @ commit:e7876a3ac520b128be7836e30172079ab2af5e45
Tom Clegg
02:22 pm Revision e7876a3a (arvados): 11349: Report wishlist size in status["nodes_wish"].
Tom Clegg
02:14 pm Story #11365 (In Progress): [Workbench] Add lock button for collection display to enable rename &...
Radhika Chippada
02:07 pm Revision f49bc279 (arvados): 11332: Use urllib quote/unquote to handle special chars (e.g. '#')
Peter Amstutz

04/07/2017

09:36 pm Story #11349: [Node Manager] Add status URL for node manager
I see 2 clusters of questions I get often, one group is about "orchestration-related" or "pipeline-wide" and the ot... Nico César
09:29 pm Story #11349: [Node Manager] Add status URL for node manager
11349-nodemanager-status-api @ commit:ab9a73d2c0b567d3c05d1d4d8463633a69eafda2 Tom Clegg
07:31 pm Story #11349: [Node Manager] Add status URL for node manager
details (proposed):
New config section "[Manage]" with "port" (127.0.0.1) and "address" (default -1, which disable...
Tom Clegg
05:21 pm Story #11349 (In Progress): [Node Manager] Add status URL for node manager
Tom Clegg
09:28 pm Revision ab9a73d2 (arvados): 11349: Add management server with /status.json
Tom Clegg
09:08 pm Revision 63c37e7e (arvados): 11365: "lock" collection to prevent user from deleting / renaming fi...
Radhika Chippada
09:06 pm Revision 3f5c2b95 (arvados): 8465: Set proper container flags to provide data on stdin.
Peter Amstutz
07:26 pm Revision 86a6e4fe (arvados): Merge branch '11237-google-api-python-client' refs #11237
Peter Amstutz
06:43 pm Bug #11237: python-arvados-python-client depends on old versions of packages
(cf. irc) tests also pass with oauth2client 1.5.2 ... so commit:4405570 LGTM Tom Clegg
06:34 pm Bug #11237 (In Progress): python-arvados-python-client depends on old versions of packages
Peter Amstutz
06:34 pm Bug #11237: python-arvados-python-client depends on old versions of packages
Update to latest google-api-python-client, and relaxes version pin on oauth2client:
11237-google-api-python-client...
Peter Amstutz
06:31 pm Revision 15784761 (arvados): 11237: Tweak version dependencies.
Peter Amstutz
06:21 pm Revision 1b226fee (arvados): 11237: Don't build oauth2client on ubuntu1604
Peter Amstutz
06:18 pm Revision 44055703 (arvados): 11237: Bump google-api-python-client to 1.6.2 for better compatabili...
oauth2client and Python versions. Peter Amstutz
05:24 pm Revision 2f52a6e7 (arvados): Merge branch '11209-unmount-subtype'
refs #11209 Tom Clegg
03:43 pm Revision 0654fa16 (arvados): Increase ping delay in WatchdogActorTest to try and reduce spurious ...
Peter Amstutz
03:36 pm Revision 330f544d (arvados): Add check that gnutls.h is available so pycurl can compile. no issue #
Peter Amstutz
03:21 pm Bug #11436: Jenkins failing to install pycurl 7.43.0
From https://github.com/pycurl/pycurl/blob/master/INSTALL.rst
We can set an environment variable to influence whic...
Peter Amstutz
03:09 pm Bug #11436 (Resolved): Jenkins failing to install pycurl 7.43.0
Starting with https://ci.curoverse.com/job/run-tests/793/... Peter Amstutz
03:11 pm Feature #8465 (In Progress): [Crunch2] Support stdin/stderr redirection
Radhika Chippada
01:22 pm Feature #8465: [Crunch2] Support stdin/stderr redirection
New branch 8465-stderr-redirection implements stderr redirection. This branch is off of 8465-stdin-redirection. Thanks. Radhika Chippada
02:10 pm Task #11379 (In Progress): Review 11308-python3
Lucas Di Pentima
02:10 pm Task #11376 (Resolved): Review 11209-unmount-subtype
Lucas Di Pentima
02:05 pm Task #11376 (In Progress): Review 11209-unmount-subtype
Lucas Di Pentima
02:09 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
LGTM. Lucas Di Pentima
07:43 am Bug #11209: stuck keep fuse mounts not cleared by crunch-job
11209-unmount-subtype @ commit:75184884ed798b474e8b9e254045dc0f4354379e Tom Clegg
01:58 pm Task #11272 (Resolved): Review 7824-arvls-arvput-collection-api-usage
Peter Amstutz
01:52 pm Revision f61bd0a7 (arvados-dev): 11332: Check exit code after running CWL conformance tests
Peter Amstutz
01:42 pm Revision 40d2a07f (arvados-dev): 11332: Add /arvados/v1/repositories to token scope documentation of
run-cwl-test.sh. Refactor into --scopes option. Peter Amstutz
01:21 pm Task #11385 (In Progress): Review branch 8465-stdin-redirection
Radhika Chippada
01:21 pm Task #11433 (Resolved): Review branch 8465-stderr-redirection
Radhika Chippada
11:29 am Revision 351f06f6 (arvados): Ensure chown does not fail on disappearing files and dirs
refs #11363 Javier Bértoli
07:42 am Task #11378 (Resolved): Warn that most users don't want --unmount-all
Tom Clegg
07:42 am Task #11377 (Resolved): Honor subtype arg
Tom Clegg
07:41 am Revision 75184884 (arvados): 11209: Restrict --unmount* operations to given --subtype.
Add warnings about affecting other fuse mounts with --unmount flags. Tom Clegg
12:12 am Revision 2ebd4496 (arvados): 8465: added stderr redirection and tests
Radhika Chippada

04/06/2017

09:24 pm Revision f2f47b27 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
09:23 pm Revision e19e2673 (arvados-dev): Merge branch '11332-arv-cwl-integration-testing' refs #11332
Peter Amstutz
09:23 pm Revision a869fa40 (arvados): Add missing test file refs #11423
Peter Amstutz
09:21 pm Revision 2b9ef5ac (arvados-dev): 11332: clone repos with --depth 1 to reduce bandwidth/disk usage.
Peter Amstutz
08:50 pm Bug #8104 (Resolved): [SDK] PyCurl 7.21.5 breaks keepclient
Applied in changeset arvados|commit:9dae3e6dde2993bc44e90539141b442c899cf114. Tom Clegg
08:30 pm Bug #8104: [SDK] PyCurl 7.21.5 breaks keepclient
(On further discussion, disregard previous note)
What's the reasoning here? Explicitly closing the socket after e...
Peter Amstutz
07:57 pm Bug #8104: [SDK] PyCurl 7.21.5 breaks keepclient
... Peter Amstutz
07:43 pm Bug #8104: [SDK] PyCurl 7.21.5 breaks keepclient
Amended: stop building pycurl deb/rpm packages, except for ubuntu1204 and centos7.
8104-pycurl-721 @ commit:625745...
Tom Clegg
07:05 pm Bug #8104: [SDK] PyCurl 7.21.5 breaks keepclient
8104-pycurl-721 @ commit:605405d932ef0ba9168c4f1cfe6edd7738e7b813 Tom Clegg
03:32 pm Bug #8104 (In Progress): [SDK] PyCurl 7.21.5 breaks keepclient
Tom Clegg
08:47 pm Revision 9dae3e6d (arvados): Merge branch '8104-pycurl-721'
closes #8104 Tom Clegg
08:43 pm Revision 1c1c11c2 (arvados): 8104: Do not close sockets after giving them to pycurl. Reverts 7b31...
Tom Clegg
08:20 pm Revision 1d0d28c7 (arvados): 8104: Drop unnecessary lambda.
Tom Clegg
08:17 pm Feature #8465: [Crunch2] Support stdin/stderr redirection
> Can write this as: stdinRdr, err = runner.Kc.ManifestFileReader(manifest.Manifest{Text: stdinColl.ManifestText}, st... Radhika Chippada
07:20 pm Feature #8465 (New): [Crunch2] Support stdin/stderr redirection
crunchrun.go L690:... Peter Amstutz
12:01 am Feature #8465: [Crunch2] Support stdin/stderr redirection
Branch 8465-stdin-redirection @ commit:c3ae1e2
* Added stdin redirection for collection and json mount points
*...
Radhika Chippada
08:13 pm Revision 3ceb42c8 (arvados): 8465: do not wait in the go routine while doing io.Copy of stdin and...
Radhika Chippada
08:12 pm Revision e3c48eaa (arvados): 8104: Stop building pycurl package for ubuntu1604.
Tom Clegg
07:59 pm Revision eb6aa92e (arvados): 8104: Relax dependency to pycurl >= 7.19.3.
Tom Clegg
07:49 pm Task #11377 (In Progress): Honor subtype arg
Tom Clegg
07:46 pm Revision ed46350a (arvados): Merge branch '11423-cwl-collection-dir-ref' closes #11423
Peter Amstutz
07:43 pm Revision 921d4321 (arvados): 11423: Restore check to use pdh_path because it is only supposed to ...
references. Peter Amstutz
07:42 pm Revision 625745e0 (arvados): 8104: Stop building pycurl packages, except for ubuntu1204 and cento...
Tom Clegg
07:20 pm Story #7824 (Resolved): [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
Applied in changeset arvados|commit:1affdcd3cd34424a817ae350fd9ca236927fd538. Lucas Di Pentima
06:51 pm Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
LGTM @ commit:dfd3260 Peter Amstutz
07:17 pm Revision 1affdcd3 (arvados): Merge branch '7824-arvls-arvput-collection-api-usage'
Closes #7824 Lucas Di Pentima
07:15 pm Revision 38fcd08d (arvados): 7824: Merge branch 'master' into 7824-arvls-arvput-collection-api-usage
Lucas Di Pentima
07:12 pm Task #11430 (Resolved): Review 8104-pycurl-721
Tom Clegg
07:05 pm Revision 605405d9 (arvados): 8104: Build pycurl 7.43.0 package.
Tom Clegg
06:49 pm Revision 33ec0939 (arvados): 8104: Remove pycurl<7.21.5 dependency.
Tom Clegg
06:38 pm Revision b5946780 (arvados): 8104: OPENSOCKETFUNCTION accepts calls from pycurl 7.21.
Tom Clegg
06:29 pm Revision 7b31dd40 (arvados): 11308: Close socket after pycurl.
Tom Clegg
06:28 pm Revision 8fd18fc4 (arvados): Merge branch 'master' into 8465-stdin-redirection
Radhika Chippada
06:28 pm Revision ae69864e (arvados): 8465: test clean up
Radhika Chippada
04:52 pm Revision 46df14bb (arvados-dev): 11332: Run arvados-specific CWL integration tests defined in
arvados/sdk/cwl/tests/arvados-tests.yml Peter Amstutz
04:31 pm Revision 7ce46899 (arvados): 11423: Fix bug checking existence of Directories which refer to root...
Peter Amstutz
03:58 pm Revision ef49a174 (arvados): 11423: Fix bug checking existence of Directories which refer to root...
collections. Peter Amstutz
02:42 pm Bug #6304: [API] Fix config file race condition and improve test cases for DNS update hooks
An ensure block should delete the stray temp file if we got as far as creating one (e.g., if f.puts fails). Something... Tom Clegg
02:08 pm Task #11382 (In Progress): Review 6304-dns-update-fix
Tom Clegg
01:08 pm Task #11388 (Resolved): Review 11100-cr-output-ttl
Peter Amstutz
01:05 pm Revision b50a3bcb (arvados): Merge branch '11413-nodemanager-gce-fixes' refs #11413
Peter Amstutz
05:34 am Revision 9e3fd2fa (arvados): 11308: Make imports epydoc-parseable.
Tom Clegg

04/05/2017

11:53 pm Revision c3ae1e2f (arvados): 8465: added stdin redirection for json mount.
Radhika Chippada
07:51 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
11308-python3 @ commit:4a8883858028d4068b52710acd9c349108b6fc7c
* raises an exception if files are opened in text mo...
Tom Clegg
07:48 pm Revision 4a888385 (arvados): 11308: Remove superfluous hashbangs
Tom Clegg
07:48 pm Revision 56b51a9e (arvados): 11308: Move stub server setup to keepstub.StubKeepServers.
Tom Clegg
07:48 pm Revision 4ef81741 (arvados): 11308: Raise exception on invalid/unsupported open() mode.
Tom Clegg
07:48 pm Revision e5d94838 (arvados): 11308: Fix exception handling in test case.
Tom Clegg
07:48 pm Revision f5409660 (arvados): 11308: Fix useless assertion.
Tom Clegg
07:48 pm Revision 249b3aba (arvados): 11308: Fix misleading argument name.
Tom Clegg
05:30 pm Bug #6304: [API] Fix config file race condition and improve test cases for DNS update hooks
Updates at branch @6304-dns-update-fix@ - commit:6da9f3666
Test run: https://ci.curoverse.com/job/developer-run-test...
Lucas Di Pentima
05:23 pm Revision 6da9f366 (arvados): 6304: Avoid possible dns update race condition by writing the temp f...
Catch IO specific exceptions when trying to write the config files.
Added test to assert that no config file is writt...
Lucas Di Pentima

04/04/2017

09:10 pm Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
Added test cases at commit:dfd3260e8
Test run: https://ci.curoverse.com/job/developer-run-tests/219/
Lucas Di Pentima
09:08 pm Revision dfd3260e (arvados): 7824: Added test cases.
Lucas Di Pentima
08:48 pm Revision 42e6998c (arvados): 8465: added stdin redirection for collection
Radhika Chippada
08:36 pm Bug #8104: [SDK] PyCurl 7.21.5 breaks keepclient
(I think this is the minimum version that has the #6073 fix: https://github.com/pycurl/pycurl/blob/master/ChangeLog#L... Peter Amstutz
08:35 pm Bug #8104: [SDK] PyCurl 7.21.5 breaks keepclient
If this happens before #11308 merges, please consider cherry-picking commit:adac4b967382c280022d202d56d7fee41764ef35 ... Tom Clegg
08:28 pm Bug #8104: [SDK] PyCurl 7.21.5 breaks keepclient
Suggest the following version dependency change:... Peter Amstutz
07:23 pm Feature #11146: [Crunch2] [Workbench] Show slurm queue position of containers submitted to slurm ...
from squeue(1): "The default value of sort for jobs is "P,t,-p" (increasing partition name then within a given partit... Tom Clegg
07:15 pm Bug #8371 (Closed): [Deployment] Upgrading Rails packages on CentOS uninstalls them
Tried on arvagrant, with CentOS 7, deploying first version
20161229114527-997479d1408139e96ecdb42a60b4f727f814f6c9...
Javier Bértoli
06:35 pm Story #11421 (New): [Workbench] UUIDs & PDHs should be hyperlinked
Use regex to recognize UUIDs anywhere on a workbench page and hyperlink them. Tom Morris
06:11 pm Task #11409 (Resolved): Review 11407-go-sdk-debug
Peter Amstutz
06:03 pm Revision 4bb024ae (arvados): 11413: Use getattr() in exception handler.
Peter Amstutz
05:12 pm Revision 0487aeb5 (arvados): 11308: Create missing temp dir. Unset proxy env during tests.
Tom Clegg
05:11 pm Revision 46f6db13 (arvados): 11308: Remove obsolete test setup steps.
Tom Clegg
03:41 pm Task #11418 (Resolved): pass tests with python3
Tom Clegg
03:41 pm Task #11418 (Resolved): pass tests with python3
Tom Clegg
03:41 pm Task #11419 (New): support text-mode open() in Python 3
Tom Clegg
03:40 pm Revision a03ce405 (arvados): 11413: Wrap destroy_node with similar logic to create_node: on excep...
the node list to determine if the node was actually destroyed successfully. Peter Amstutz
03:39 pm Revision 77f5a84c (arvados): Merge branch '11100-cr-output-ttl'
refs #11100 Tom Clegg
03:21 pm Revision 4a35e06b (arvados): 11413: Fix issues with node manager on GCE:
* Always override Node.size with CloudSizeWrapper
* Get updated node record before setting metadata to minimize 'Sup...
Peter Amstutz
03:18 pm Story #11308: Support Python3 for arvados-python-client & command line utilities
... Tom Clegg
02:35 pm Task #11388 (In Progress): Review 11100-cr-output-ttl
Tom Clegg
06:44 am Revision c633f38b (arvados): 11308: Add stderr assertion for easier debugging.
Tom Clegg
06:44 am Revision e8074a3c (arvados): 11308: Fix key order sensitivity.
Tom Clegg
06:44 am Revision 86409934 (arvados): 11308: Update assertion to accommodate different exception in py3.
Tom Clegg
06:44 am Revision d761fb1e (arvados): 11308: Fix keepstub bugs and Python 3 compatibility.
Tom Clegg
06:44 am Revision 6d9cf49e (arvados): 11308: pep8
Tom Clegg
06:44 am Revision adac4b96 (arvados): 11308: Close socket after pycurl.
Tom Clegg
06:44 am Revision cb371701 (arvados): 11308: Expect full class name in traceback.
Tom Clegg
06:44 am Revision 991d037e (arvados): 11308: Fix string/bytes confusion for Python 3
Tom Clegg
06:44 am Revision 04f38e70 (arvados): 11308: Fix traceback.format_exc() usage.
Tom Clegg
06:44 am Revision cd708c67 (arvados): 11308: assertRegexpMatches -> assertRegex
Tom Clegg
04:06 am Bug #11136: Nginx config should speak JSON when returning its own response for unproxyable API re...
When apiserver itself responds &ge;400, it returns a body like this... Tom Clegg
01:52 am Revision 3fcd28c6 (arvados): 11308: Fix keepstub bugs and Python 3 compatibility.
Tom Clegg
01:52 am Revision 5f238a36 (arvados): 11308: Add stderr assertion for easier debugging.
Tom Clegg
01:52 am Revision 86ab369f (arvados): 11308: Update assertion to accommodate different exception in py3.
Tom Clegg
01:52 am Revision 5f17b001 (arvados): 11308: Fix string/bytes confusion for Python 3
Tom Clegg
01:52 am Revision a5957935 (arvados): 11308: Fix traceback.format_exc() usage.
Tom Clegg
01:52 am Revision d0b76e97 (arvados): 11308: Close socket after pycurl.
Tom Clegg
01:52 am Revision 3622c512 (arvados): 11308: Expect full class name in traceback.
Tom Clegg
01:52 am Revision fd548e89 (arvados): 11308: assertRegexpMatches -> assertRegex
Tom Clegg
01:52 am Revision 44a87969 (arvados): 11308: pep8
Tom Clegg

04/03/2017

09:46 pm Bug #11136: Nginx config should speak JSON when returning its own response for unproxyable API re...
One approach is to do something like: ... Nico César
09:25 pm Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
Updates at commit:313415e33
Test run: https://ci.curoverse.com/job/developer-run-tests/218/
Note: tests are pendi...
Lucas Di Pentima
09:16 pm Revision 313415e3 (arvados): 7824: Now a subdirectory can be asked to be listed. Also, enhanced e...
Lucas Di Pentima
08:45 pm Revision 2c0c6e6b (arvados): 7824: Several improvements:
* Added error reporting on cases where the user ask for downloading a nonexisting file/subdir inside a collection.
* ...
Lucas Di Pentima
08:39 pm Feature #11410 (New): [SDK] tool for copying CWL file + input collections
arv-copy can copy workflow records (+ dependencies). However, our current workflow is oriented around CWL files stor... Peter Amstutz
08:25 pm Bug #11407 (Resolved): [SDK] Go respects ARVADOS_DEBUG to enable debug logging
Applied in changeset arvados|commit:07c92074fe2b493dc8cd0abdad154e5b36d4adfc. Peter Amstutz
08:07 pm Bug #11407: [SDK] Go respects ARVADOS_DEBUG to enable debug logging
Might be good to do that in init() instead of re-installing debug mode each time a new KeepClient is created. LGTM ot... Tom Clegg
07:27 pm Bug #11407 (In Progress): [SDK] Go respects ARVADOS_DEBUG to enable debug logging
Peter Amstutz
06:27 pm Bug #11407 (Resolved): [SDK] Go respects ARVADOS_DEBUG to enable debug logging
keepclient.New() should check for ARVADOS_DEBUG in environment and enable DebugPrintf() Peter Amstutz
08:21 pm Revision 07c92074 (arvados): Merge branch '11407-go-sdk-debug' closes #11407
Peter Amstutz
08:20 pm Revision 1654f776 (arvados): 11407: keepclient checks for ARVADOS_DEBUG and enables debugging.
Peter Amstutz
07:45 pm Task #11409 (In Progress): Review 11407-go-sdk-debug
Tom Clegg
07:27 pm Task #11409 (Resolved): Review 11407-go-sdk-debug
Peter Amstutz
07:26 pm Revision 9595389f (arvados): 11407: keepclient checks for ARVADOS_DEBUG and enables debugging.
Peter Amstutz
06:07 pm Revision 988e9052 (arvados): 7824: Log errors when trying to read the collection specified.
Lucas Di Pentima
05:52 pm Revision 3cc77494 (arvados): 7824: Closing file when finished writing
Lucas Di Pentima
01:52 pm Revision 17f22a2b (arvados): 8567: Add workaround for "trap EXIT" in pid 1 bash process. Add com...
Peter Amstutz
12:59 pm Feature #8465 (In Progress): [Crunch2] Support stdin/stderr redirection
Radhika Chippada
06:42 am Bug #11405 (New): [arv-put] Lumpy / slow upload
I'm not 100% sure that this is a bug, but it seems weird, so wanted to capture the data.
The last hour of the uplo...
Tom Morris
05:07 am Revision 5901b749 (arvados): 11308: Fix string/bytes confusion for Python 3
Tom Clegg
05:07 am Revision fd8940c4 (arvados): 11308: Close socket after pycurl.
Tom Clegg
05:07 am Revision 2685e2ea (arvados): 11308: Expect full class name in traceback.
Tom Clegg
05:07 am Revision 63eac50b (arvados): 11308: assertRegexpMatches -> assertRegex
Tom Clegg
05:07 am Revision d39d59e9 (arvados): 11308: Fix traceback.format_exc() usage.
Tom Clegg

04/02/2017

11:15 pm Revision 82f763b6 (arvados): 11308: run-tests --skip python2
Tom Clegg
05:26 am Revision 76d9365a (arvados): more str vs bytes, for py3
Tom Clegg

04/01/2017

10:00 pm Revision 41e9f18d (arvados): 11308: Fix deprecated logger.warn -> logger.warning.
Tom Clegg
09:52 pm Revision a81a5f35 (arvados): 11308: md5 -> hashlib.
Tom Clegg
09:45 pm Revision 0e0c1400 (arvados): 11308: Fix bytes vs. str problems.
Tom Clegg
09:45 pm Revision 0199da87 (arvados): 11308: Fix variable name conflict.
Tom Clegg
09:45 pm Revision 7fde9a0a (arvados): 11308: Eliminate old_div().
Tom Clegg
09:45 pm Revision d1af87de (arvados): 11308: Import future module.
Tom Clegg
09:43 pm Revision 8e485674 (arvados): 11308: Update python-gflags.
Tom Clegg
09:43 pm Revision de259910 (arvados): 11308: md5 to hashlib.
Tom Clegg
09:43 pm Revision e61df483 (arvados): 11308: Futurize stage2.
Tom Clegg
09:43 pm Revision fbce7b14 (arvados): 11308: UserDict to Python 3.
Tom Clegg
09:43 pm Revision 9b823dc4 (arvados): 11308: Test sdk/python with Python 3.
Tom Clegg
09:43 pm Revision 5e46c19f (arvados): 11308: Fix futurize.
Tom Clegg
09:43 pm Revision f69e114c (arvados): 11308: Futurize.
Tom Clegg
09:43 pm Revision d8fef944 (arvados): 11308: fmt
Tom Clegg
02:18 am Feature #11100: [CWL] Intermediary collection handling can be specified
Peter Amstutz wrote:
> Documentation of the @output_ttl@ should specify units.
Indeed. Got it in [[Containers API...
Tom Clegg
02:07 am Revision ff3bb22d (arvados): 11100: Adjust delete_at in before_validation hook instead of validat...
Tom Clegg

03/31/2017

09:49 pm Revision 00161497 (arvados): 11100: a-c-r sets output_ttl and deletes intermediate collections on...
Peter Amstutz
09:26 pm Revision b651984f (arvados): 11100: Specify units in output_ttl docs.
Tom Clegg
08:25 pm Feature #11100: [CWL] Intermediary collection handling can be specified
Documentation of the @output_ttl@ should specify units.
Should the code to adjust delete_at to the earliest valid ...
Peter Amstutz
01:44 am Feature #11100: [CWL] Intermediary collection handling can be specified
With "earliest possible delete_at":
11100-cr-output-ttl @ commit:65121f8db54a1ed15207d050e1f48c5fc26d646b
Tom Clegg
01:36 am Feature #11100 (In Progress): [CWL] Intermediary collection handling can be specified
Tom Clegg
08:20 pm Bug #11402 (Resolved): Failure caching discovery document in crunch
Applied in changeset arvados|commit:55c34fb97edfba5a014eb43b965130cf11dbda8a. Tom Clegg
08:08 pm Bug #11402 (Resolved): Failure caching discovery document in crunch
... Peter Amstutz
08:17 pm Revision 55c34fb9 (arvados): 11402: Disable caching if $HOME/ is not writable, instead of returni...
closes #11402 Tom Clegg
07:14 pm Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
arv-get should get some test cases, as well. At minimum, something like:
* Get a single file from a collection
*...
Peter Amstutz
07:00 pm Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
One last thought. It seems a bit inefficient to iterate over every single file in the collection to only extract a s... Peter Amstutz
06:47 pm Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
get.py L170... Peter Amstutz
04:49 pm Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIs
Branch @7824-arvls-arvput-collection-api-usage@ at commit:c820bfc91be7635739bad0857ba3a385d1334b6a _(I've realized th... Lucas Di Pentima
06:08 pm Bug #6304 (In Progress): [API] Fix config file race condition and improve test cases for DNS upda...
Lucas Di Pentima
06:04 pm Revision 28e59edc (arvados): closes #11368
Merge branch '11368-collection-file-rename-warning' Radhika Chippada
06:03 pm Revision 4af850d6 (arvados): 11368: show error if user tries to rename a collection file with a d...
Radhika Chippada
06:00 pm Revision c0d251c4 (arvados): 11368: optimize coll.exist?(new_file_path) invocation
Radhika Chippada
05:54 pm Revision 736a629e (arvados): Merge branch '11398-nodes-items-available'
refs #11398 Tom Clegg
05:52 pm Revision 5ee2b960 (arvados): 11398: Fix missing items_available in nodes#index response.
Tom Clegg
05:45 pm Task #11272 (In Progress): Review 7824-arvls-arvput-collection-api-usage
Peter Amstutz
05:37 pm Story #10401: [CWL] Limit expansion of Directory inputs
Not sure what to make of this. Master (b9236fbe81426446e1b541a45e219bbe513fe8d0) passes.... Tom Clegg
05:10 pm Task #11194 (In Progress): Review 10401-limit-dir-expansion
Tom Clegg
04:40 pm Revision c820bfc9 (arvados): 7824: Use logging facility to show error messages.
Lucas Di Pentima
03:39 pm Revision 02a997da (arvados): 11368: show error if user tries to rename a collection file with a d...
Radhika Chippada
03:06 pm Task #11367 (Resolved): Need to pass CWL tests
Peter Amstutz
03:06 pm Task #11224 (Resolved): Update API usage
Peter Amstutz
03:06 pm Task #11192 (Resolved): Review 9132-dockerclient
Peter Amstutz
02:45 pm Story #9132 (Resolved): [Crunch2] crunch-run uses official docker client library
Applied in changeset arvados|commit:b9236fbe81426446e1b541a45e219bbe513fe8d0. Peter Amstutz
02:41 pm Revision b9236fbe (arvados): Merge branch '9132-dockerclient' closes #9132
Peter Amstutz
02:24 pm Revision fb1c8e81 (arvados): 7824: Updated code to use newer Collection APIs, leaving all_streams...
Lucas Di Pentima
02:17 pm Revision b351a877 (arvados): 9132: Fix test errors due to rebase.
Peter Amstutz
01:57 pm Revision 0b04efef (arvados): Fix package test for ubuntu1204.
refs #10797 Ward Vandewege
01:37 am Story #11308 (In Progress): Support Python3 for arvados-python-client & command line utilities
Tom Clegg

03/30/2017

10:47 pm Revision 65121f8d (arvados): 11100: If caller sets collection.delete_at too early, set it to the ...
Tom Clegg
10:46 pm Revision 7f881ce0 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
10:44 pm Revision 8a1d756a (arvados-dev): Make delete script a little more verbose. Update remote branches...
refs #11020 Javier Bértoli
10:02 pm Revision 1a4a2f32 (arvados): 9132: Fix ContainerCreate to include HostConfig, container UUID as n...
Peter Amstutz
09:19 pm Feature #11100: [CWL] Intermediary collection handling can be specified
Tom Clegg wrote:
> Currently it's an error to set delete_at to a time earlier than blob_signature_ttl. Some time pas...
Peter Amstutz
09:11 pm Feature #11100: [CWL] Intermediary collection handling can be specified
11100-cr-output-ttl @ commit:4437774e863465c0daa41dfd9716174e18d93122 Tom Clegg
09:08 pm Feature #11100: [CWL] Intermediary collection handling can be specified
Currently it's an error to set delete_at to a time earlier than blob_signature_ttl. Some time passes between when the... Tom Clegg
09:11 pm Revision 4437774e (arvados): 11100: Add comment about delete_at validation race.
Tom Clegg
08:56 pm Revision a12ed889 (arvados): 11100: Clean up permission checks.
Tom Clegg
08:50 pm Revision c507b0b0 (arvados): 11100: Add container_requests.output_ttl field. Fix validation of ou...
Tom Clegg
07:40 pm Revision 38141f72 (arvados): 8567: Log total bytes to migrate
Peter Amstutz
07:14 pm Revision acb0bd43 (arvados): 8567: Use arvados.config instead of os.environ. Tweak disk space ac...
Peter Amstutz
06:40 pm Revision 0568c2d4 (arvados): 7824: Moved arv-get code to arvados.commands and replaced bin/arv-ge...
Lucas Di Pentima
06:32 pm Revision e8e9262a (arvados): 9132: few more updates to use the new API
Peter Amstutz
06:32 pm Revision 6fe136c2 (arvados): 9132: WIP switching to using official Docker Go client.
Peter Amstutz
06:32 pm Revision bebac87e (arvados): 9132: Use ThinDockerClientProxy to overcome the import issue around ...
Perform most of the test updates needed to use the new API. Radhika Chippada
06:32 pm Revision d6703f16 (arvados): 9132: Fix after rebase
Peter Amstutz
06:32 pm Revision f34f20a9 (arvados): Merge branch '11332-fix-crunchscript' refs #11332
Peter Amstutz
06:15 pm Revision 154ae0d4 (arvados): 7824: Further ordering fixes. Now all arv-ls tests passes.
Lucas Di Pentima
05:41 pm Revision ff8d14ac (arvados): Merge branch '7709-api-rails4' (partial)
refs #7709
refs #11100
Tom Clegg
05:34 pm Bug #11397 (Resolved): [Ruby 2.3.3] Make json gem dependency > 1.7.7
arvados and arvados-cli gems accept 1.8.x.
New gems aren't published yet but as of commit:78ca1460d58172f455540929...
Tom Clegg
05:09 pm Bug #11397 (Resolved): [Ruby 2.3.3] Make json gem dependency > 1.7.7
gem json 1.7.7 doesn't compile in distros nowadays . If this is not a hard dependency for arvados, will be good to h... Nico César
05:18 pm Bug #11190: Containers seem to run more than once, which isn't supposed to happen
I wonder if we should move the state transition to "Running" as soon as crunch-run has starting doing anything substa... Peter Amstutz
05:12 pm Task #11225 (Resolved): Update cwltool
Peter Amstutz
04:44 pm Bug #11396: Network saturation
Possibly not relevant, but the example job that failed under saturation conditions ended up failing with the followin... Joshua Randall
04:42 pm Bug #11396: Network saturation
Might be useful in the context of running a squid proxy between keep clients and keepstores:
- https://github.com/f...
Joshua Randall
04:41 pm Bug #11396 (New): Network saturation
Some of our extremely I/O intensive jobs are able to completely saturate the network links. When that happens, jobs f... Joshua Randall
03:15 pm Revision 433c48ed (arvados): 8567: Improve disk space accounting. Default to use overlay storage...
Peter Amstutz
03:06 pm Revision ae7f5a9c (arvados): 7824: Fix upper/lower case ordering on arv-ls.
Lucas Di Pentima
02:45 pm Revision 37313363 (arvados): 7709: Do not set job_readable when it won't be used.
Tom Clegg
02:45 pm Revision 68287280 (arvados): 7709: De-duplicate "ensure unique name" implementations.
Tom Clegg
02:45 pm Revision d040869c (arvados): 7709: Fix tests (count existing links before adding new ones).
Tom Clegg
02:45 pm Revision de1e1bf2 (arvados): 7709: Leave container_request alone when applying default keep_cache...
Tom Clegg
02:29 pm Revision fbc867e0 (arvados): 11209: Restore missing import.
refs #11209 Tom Clegg
02:09 pm Revision c6a8839b (arvados): 11332-fix-crunchscript: Set ACCEPTLIST_RE because crunch_script alte...
point doesn't go through main. Peter Amstutz
01:23 am Revision 78ca1460 (arvados): 11355: Arvados gems accept json 2.x.
refs #11355 Tom Clegg

03/29/2017

09:18 pm Bug #4124: [Crunch] Socket timed out on send/recv operation causes pipeline failure
well i found this from 2016-04-12 :... Nico César
07:58 pm Bug #4124: [Crunch] Socket timed out on send/recv operation causes pipeline failure
Review production cluster logs to see if this specific timeout message has been seen. If not, close this. Otherwise, ... Tom Morris
08:10 pm Task #11392 (New): Review
Tom Morris
08:10 pm Task #11391 (New): Review
Tom Morris
08:09 pm Task #11390 (New): Review
Tom Morris
08:09 pm Task #11389 (New): Review a-c-r changes
Tom Clegg
08:09 pm Task #11388 (Resolved): Review 11100-cr-output-ttl
Tom Clegg
08:08 pm Task #11385 (Resolved): Review branch 8465-stdin-redirection
Radhika Chippada
08:07 pm Task #11384 (Resolved): Review branch 11365-collection-lock-button
Radhika Chippada
08:05 pm Task #11382 (Resolved): Review 6304-dns-update-fix
Lucas Di Pentima
08:05 pm Task #11381 (New): Review
Lucas Di Pentima
08:04 pm Task #11379 (In Progress): Review 11308-python3
Tom Clegg
08:03 pm Task #11378 (Resolved): Warn that most users don't want --unmount-all
Tom Clegg
08:03 pm Task #11377 (Resolved): Honor subtype arg
Tom Clegg
08:03 pm Task #11376 (Resolved): Review 11209-unmount-subtype
Tom Clegg
08:02 pm Task #11375 (Resolved): Review 11349-nodemanager-status-api
Tom Clegg
07:49 pm Task #11370 (New): Update cwl runner after API feature is merged
Peter Amstutz
07:40 pm Story #11365: [Workbench] Add lock button for collection display to enable rename & delete
The lock / unlock is at UI level only. For ex: if I unlock, and close the browser and reopen in a new browser, the co... Radhika Chippada
05:46 pm Story #11365 (Resolved): [Workbench] Add lock button for collection display to enable rename & de...
Because renaming or deleting a file changes the portable data hash of a collection, we want to warn the user of the c... Tom Morris
07:35 pm Bug #11283: duplicate key value violates unique constraint "index_nodes_on_slot_number"
We can do better than this in source:services/api/app/models/node.rb... Tom Clegg
07:16 pm Task #11367 (Resolved): Need to pass CWL tests
Peter Amstutz
07:06 pm Story #9132: [Crunch2] crunch-run uses official docker client library
Assigning to Peter to debug / fix any issues with docket client invocations. Thanks. Radhika Chippada
03:11 pm Story #9132: [Crunch2] crunch-run uses official docker client library
Yea, that might be the legitimate exit code. But the CWL test should pass, and it isn't, so it needs further investi... Peter Amstutz
12:57 am Story #9132: [Crunch2] crunch-run uses official docker client library
> I think maybe the return code from ContainerWait is not being interpreted properly
We are just returning what is...
Radhika Chippada
04:11 pm Task #11199 (Resolved): Review 10218-record-node-info
Radhika Chippada
04:10 pm Task #11265 (Resolved): Review branch 3821-collection-file-manage
Radhika Chippada
04:00 pm Revision 0708d4a0 (arvados): 10401: Bump cwltool for packing fix.
Peter Amstutz
04:00 pm Revision 25ab6beb (arvados): 10401: Bump schema salad for self-colliding id fix.
Peter Amstutz
03:59 pm Revision d7d00f51 (arvados): 10401: Only upload file:// identifiers.
Peter Amstutz
03:59 pm Revision 201ddbd8 (arvados): 10401: Update crunch script, no longer expands directories by default.
Peter Amstutz
03:59 pm Revision 692ef5fd (arvados): 10401: Fixup file/directory literal handling
Peter Amstutz
03:59 pm Revision 7bd9af41 (arvados): 10401: Bump arvados-python-client version for updates to
arvados.command.run.uploadfiles. Peter Amstutz
03:59 pm Revision 5cd99e52 (arvados): 10401: Use custom_schema_callback. Bump schema-salad and cwltool ve...
Peter Amstutz
03:58 pm Revision 820d3244 (arvados): 10401: Use use_custom_schema feature for extensions.
Peter Amstutz
03:58 pm Revision 6b0632d7 (arvados): 10401: Allow Arvados extensions to be used in "requirements" not jus...
Peter Amstutz
03:58 pm Revision 33b9f9bd (arvados): 10401: Rework support for uploading Directories (don't assume files are
enumerated). Peter Amstutz
03:51 pm Revision acefccc4 (arvados): Merge branch '11235-log-retryable'
refs #11235 Tom Clegg
03:51 pm Revision 20d1e6ac (arvados): 10401: Bump cwltool for packing fix.
Peter Amstutz
03:50 pm Story #3821 (Resolved): [Workbench] Delete and rename files in collections
Applied in changeset arvados|commit:2b27132d13fd72f29dbab2297f0d2dc1c110eed1. Radhika Chippada
03:41 pm Story #3821: [Workbench] Delete and rename files in collections
LGTM, please merge. Lucas Di Pentima
03:33 pm Story #3821: [Workbench] Delete and rename files in collections
> Could you update the rest of the routes so that instead of using the “:on => :member” style, use the newer “on: :me... Radhika Chippada
02:33 pm Story #3821: [Workbench] Delete and rename files in collections
Some comments:
* File @apps/workbench/config/routes.rb@:
** Related detail: Could you update the rest of the rout...
Lucas Di Pentima
03:50 pm Story #4594 (Resolved): [Workbench] In the collections page, provide the ability to select indivi...
Applied in changeset arvados|commit:2b27132d13fd72f29dbab2297f0d2dc1c110eed1. Radhika Chippada
03:49 pm Revision b3d286cd (arvados): 11235: Log a message when a job is interrupted by node failure.
...and say what's going to happen as a result, even if that is not
"giving up" yet.
Tom Clegg
03:44 pm Revision 2b27132d (arvados): closes #3821
Merge branch '3821-collection-file-manage' Radhika Chippada
03:43 pm Revision 46fac76c (arvados): Fix invalid records in API collections test fixture and improve erro...
to include both values. Incidentally revealed by adding PDH index to
collections table. refs #11361
Peter Amstutz
03:28 pm Revision 3306ff58 (arvados): 3821: improve the tooltip on rename file option.
Radhika Chippada
02:54 pm Revision 32430fd4 (arvados): Fix packaging for python-arvados-cwl-runner refs #11333
Peter Amstutz
04:26 am Revision f92654c5 (arvados): Merge branch '11361-pdh-index'
refs #11361 Tom Clegg

03/28/2017

10:01 pm Revision 01da808f (arvados): 11361: Add pdh index to collections table.
Tom Clegg
08:29 pm Story #9132: [Crunch2] crunch-run uses official docker client library
I'm running stuff and getting an exit code of "2" when it should be "0". I think maybe the return code from Containe... Peter Amstutz
08:17 pm Task #11353 (Resolved): use arv-mount --unmount-all in crunch-job
Tom Clegg
08:13 pm Revision fe0751fd (arvados): Merge branch '11209-unmount-replace'
refs #11209 Tom Clegg
08:07 pm Task #11265 (In Progress): Review branch 3821-collection-file-manage
Lucas Di Pentima
08:01 pm Revision 126a8a23 (arvados): Merge branch '11333-cwl-phone-home' closes #11333
Peter Amstutz
08:01 pm Revision 07e060db (arvados): 3821: update test assertion that randomly fails; in fact, I don't se...
Radhika Chippada
08:00 pm Revision 951ba69d (arvados): 10401: Bump schema salad for self-colliding id fix.
Peter Amstutz
07:52 pm Revision 71f0b2cc (arvados): 11332: Suppress phone-home behavior in check_exists by recognizing
http://arvados.org/cwl and returning True. Peter Amstutz
07:52 pm Revision 11e42199 (arvados): 11333: Add explicit version pin on 'typing'
Peter Amstutz
07:50 pm Revision fdc7164e (arvados): Merge branch '11332-cwl-filenames' closes #11332
Peter Amstutz
07:26 pm Revision 4b8e618c (arvados): Merge branch 'master' into 3821-collection-file-manage
Radhika Chippada
06:51 pm Story #11350 (In Progress): [Nodemanager] Merge cloud_environment support to upstream libcloud
Fix tests so this pull request can be merged:
https://github.com/apache/libcloud/pull/969
Peter Amstutz
06:43 pm Story #11349: [Node Manager] Add status URL for node manager
See source:sdk/python/tests/keepstub.py and source:sdk/python/tests/test_keep_client.py for example of starting up a ... Tom Clegg
06:33 pm Story #11349 (Resolved): [Node Manager] Add status URL for node manager
Implemented HTTP server which serves status URL with JSON format output
Configurable port #
Start with the data w...
Tom Morris
06:33 pm Story #3821: [Workbench] Delete and rename files in collections
Branch 3821-collection-file-manage @ commit:07e060d
* Added a "Remove selected files" option to the Selection drop...
Radhika Chippada
06:07 pm Revision 189495ad (arvados): 3821: collection viewed from another object
Radhika Chippada
05:57 pm Revision 1c4a828b (arvados): 10401: Bump arvados-python-client version for updates to
arvados.command.run.uploadfiles. Peter Amstutz
05:57 pm Revision 9b6019bc (arvados): 10401: Only upload file:// identifiers.
Peter Amstutz
05:57 pm Revision ea937330 (arvados): 10401: Fixup file/directory literal handling
Peter Amstutz
05:57 pm Revision 62b4f786 (arvados): 10401: Update crunch script, no longer expands directories by default.
Peter Amstutz
05:57 pm Revision 8065702d (arvados): 10401: Use custom_schema_callback. Bump schema-salad and cwltool ve...
Peter Amstutz
05:57 pm Revision c31a9cf4 (arvados): 10401: Allow Arvados extensions to be used in "requirements" not jus...
Peter Amstutz
05:57 pm Revision ca6de897 (arvados): 10401: Rework support for uploading Directories (don't assume files are
enumerated). Peter Amstutz
05:57 pm Revision dbb5d6ea (arvados): 10401: Use use_custom_schema feature for extensions.
Peter Amstutz
05:46 pm Feature #7709: [API] Upgrade API server to Rails 4.2
This LGTM, thanks! Lucas Di Pentima
05:36 pm Feature #7709: [API] Upgrade API server to Rails 4.2
Lucas Di Pentima wrote:
> * File @services/api/app/controllers/application_controller.rb@ - Line 491: Was @has_key?(...
Tom Clegg
04:59 pm Feature #7709: [API] Upgrade API server to Rails 4.2
Phew, lots of reading! Some minor comments:
* File @services/api/app/controllers/application_controller.rb@ - Line...
Lucas Di Pentima
05:34 pm Task #11292 (Resolved): Review 11209-unmount-replace
Lucas Di Pentima
05:33 pm Revision e1e05845 (arvados): 7709: Fix lazy splat.
Tom Clegg
05:33 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
LGTM. All tests passing now. Lucas Di Pentima
02:13 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
Fixed a race condition in the tests, and a problem with the refactored "standalone mode" code (evidently it's critica... Tom Clegg
05:32 pm Revision 78ff2a60 (arvados): 7709: Dispatch assert_equal to assert_nil when needed to accommodate...
Tom Clegg
05:24 pm Revision 7fd4aa96 (arvados): 7709: Use assert_nil instead of assert_equal nil.
Tom Clegg
04:57 pm Revision 5f260ce7 (arvados): 11333: Add explicit version pin on 'typing'
Peter Amstutz
04:54 pm Revision f14965fa (arvados): 3821: The "Remove selected files" option should only be presented if...
Radhika Chippada
04:44 pm Revision 00deca5e (arvados): 11332: Suppress phone-home behavior in check_exists by recognizing
http://arvados.org/cwl and returning True. Peter Amstutz
04:37 pm Revision a7748734 (arvados): 11332: Override relaxed path checks to be even more relaxed, and bum...
schema salad pin for hashes-in-filenames fix. Peter Amstutz
04:29 pm Story #10401: [CWL] Limit expansion of Directory inputs
10401-limit-dir-expansion
* Bump cwltool version, add support for cwltool:LoadListing hint which controls expansio...
Peter Amstutz
04:02 pm Bug #11236 (Resolved): Dependencies in Ubuntu 16.04 (xenial) offer to REMOVE arvados-node-manager
Run... Javier Bértoli
03:33 pm Revision 2b2c23bf (arvados): 3821: attr_accessor not needed on collection "filename"
Radhika Chippada
03:25 pm Revision dfa41e50 (arvados): 3821: add an integration test to rename a file in a collection
Radhika Chippada
02:48 pm Story #6777 (Rejected): [Documentation] recommend ruby2.1 package on debian 8
I don't think we can use the "system ruby" approach. There are some intervals when Debian ships with a version we can... Tom Clegg
02:43 pm Bug #8222 (Resolved): [SDK] "gem install arvados" fails on ruby <2.2.2 since activesupport 5.0.0 ...
arvados and arvados-cli gems decline activesupport 5 now Tom Clegg
02:30 pm Revision fb7bb4c8 (arvados): 7709: Avoid omniauth >= 1.5.0 to stay compatible with Ruby < 2.1.9
Tom Clegg
02:19 pm Task #11340 (Resolved): Review 10401-run-upload-dir
Tom Clegg
01:57 pm Revision 22a6d5ae (arvados): 10401: Update crunch script, no longer expands directories by default.
Peter Amstutz
01:56 pm Revision 86fec403 (arvados): 10401: Pin 'typing' dependency to resolve dependency conflict.
Peter Amstutz
01:30 pm Revision 262dfdbc (arvados): 10401: Override relaxed path checks to be even more relaxed.
Peter Amstutz
05:52 am Revision 8b4d5991 (arvados): 11209: Test using ./bin/arv-mount from source dir.
Tom Clegg

03/27/2017

11:37 pm Revision 823ef132 (arvados): 3821: integration test to remove files from a collection
Radhika Chippada
10:43 pm Revision 8cd68a56 (arvados): 11209: Replace "--unmount /path/..." with "--unmount-all /path".
Tom Clegg
10:36 pm Revision 387376e5 (arvados): 11209: Fix daemon mode.
DaemonContext cannot happen after listen_for_events(). Tom Clegg
10:35 pm Revision c33eb319 (arvados): 3821: collection controller tests for remove and rename files.
Radhika Chippada
08:29 pm Bug #11257: [CWL] Fails with $schemas referencing remote resources.
Fix pending in #10401 (as part of larger pathmapper refactor) Peter Amstutz
08:27 pm Revision c627686e (arvados): 10401: Fixup file/directory literal handling
Peter Amstutz
08:23 pm Revision 922e79be (arvados): Update cwl test_submit refs #10401
Peter Amstutz
08:18 pm Revision 0582da72 (arvados): 10401: Only upload file:// identifiers.
Peter Amstutz
08:04 pm Revision eafc2f38 (arvados): 10401: Bump arvados-python-client version for updates to
arvados.command.run.uploadfiles. Peter Amstutz
08:03 pm Revision a3d6e83a (arvados): 10401: Rework support for uploading Directories (don't assume files are
enumerated). Peter Amstutz
08:03 pm Revision 7563a5bd (arvados): 10401: Use custom_schema_callback. Bump schema-salad and cwltool ve...
Peter Amstutz
08:03 pm Revision 869a759f (arvados): 10401: Allow Arvados extensions to be used in "requirements" not jus...
Peter Amstutz
08:03 pm Revision acaa7f27 (arvados): 10401: Use use_custom_schema feature for extensions.
Peter Amstutz
07:54 pm Revision ca19a29f (arvados): Merge branch '10401-run-upload-dir' refs #10401
Peter Amstutz
07:33 pm Story #10401: [CWL] Limit expansion of Directory inputs
10401-run-upload-dir @ commit:7e9156a LGTM
Tom Clegg
07:17 pm Story #10401: [CWL] Limit expansion of Directory inputs
10401-run-upload-dir
Add support for uploading Directories in the "uploadfiles" method of arv-run (used by a-c-r)
Peter Amstutz
07:07 pm Revision d1a64a8b (arvados): 10401: Use custom_schema_callback. Bump schema-salad and cwltool ve...
Peter Amstutz
07:05 pm Task #11340 (Resolved): Review 10401-run-upload-dir
Peter Amstutz
07:03 pm Revision d32ca4db (arvados): 10401: Use use_custom_schema feature for extensions.
Peter Amstutz
07:03 pm Revision 54afede3 (arvados): 10401: Allow Arvados extensions to be used in "requirements" not jus...
Peter Amstutz
07:03 pm Revision bf7d4235 (arvados): 10401: Rework support for uploading Directories (don't assume files are
enumerated). Peter Amstutz
06:58 pm Revision 7e9156ae (arvados): 10401: Support in arv-run for uploading Directories.
Peter Amstutz
06:35 pm Task #11264 (In Progress): Review 7709-api-rails4
Lucas Di Pentima
06:29 pm Bug #11209: stuck keep fuse mounts not cleared by crunch-job
Tom Clegg wrote:
> I figure since we'll try to mount at the literal path "/path/..." we have to assume "/path/..."...
Lucas Di Pentima
06:18 pm Feature #7709: [API] Upgrade API server to Rails 4.2
> we should do 4.2.5 because of Ubuntu16's default pg is 9.5
(Done, and in the meantime we have a workaround for thi...
Tom Clegg
06:14 pm Task #11337 (Resolved): Check ruby warnings and Rails upgrade notes
Tom Clegg
06:12 pm Revision 8d9b12f2 (arvados): 7709: Fix Ruby warnings.
Tom Clegg
05:51 pm Revision d42ec212 (arvados): 7709: Fix warning while suppressing warnings.
Tom Clegg
05:49 pm Revision c066a2e6 (arvados): 7709: Suppress Ruby warnings while loading gems.
Tom Clegg
05:19 pm Revision c36272a5 (arvados): 7709: Fix some ruby warnings.
Tom Clegg
04:49 pm Revision 045bace6 (arvados): 7709: Fix state leaking between test suites.
Tom Clegg
04:49 pm Revision 84ad2157 (arvados): 7709: Update bundle
Tom Clegg
04:49 pm Revision c9a361f7 (arvados): 7709: De-duplicate "ensure unique name" implementations.
Tom Clegg
04:49 pm Revision 390af6a1 (arvados): 7709: Allow null for empty serialized fields.
Tom Clegg
04:49 pm Revision ab314b9e (arvados): 7709: Remove passenger/puma websocket tests.
Tom Clegg
04:49 pm Revision 30dbddd3 (arvados): 7709: Remove services/api/log/
Tom Clegg
04:49 pm Revision 126dd750 (arvados): 7709: Serialized fields reject wrong types with useful error messages.
Tom Clegg
04:49 pm Revision 9090c60b (arvados): 7709: Upgrade to rails4, fix some of the compatibility issues.
Tom Clegg
04:49 pm Revision dfd8c4bb (arvados): 7709: A scope that permits GET permits HEAD too.
Tom Clegg
04:49 pm Revision 7c6852e1 (arvados): 7709: Accommodate Rails3 or 4 error message.
Tom Clegg
04:49 pm Revision de4df7f8 (arvados): 7709: Fix full-text index check
PostgreSQL query from http://stackoverflow.com/a/6777904
(In Rails 4.2, ActiveRecord::Base.connection.indexes() no l...
Tom Clegg
04:49 pm Revision 05d453ec (arvados): 7709: Do not set job_readable when it won't be used.
Tom Clegg
04:49 pm Revision 09dcf71e (arvados): 7709: Leave container_request alone when applying default keep_cache...
Tom Clegg
04:49 pm Revision 099a8c62 (arvados): 7709: Fix tests (count existing links before adding new ones).
Tom Clegg
04:26 pm Revision 13c47e13 (arvados): Pin ws4py <0.4 (our websocket tests fail on 0.4.1)
No issue # Tom Clegg
02:42 pm Task #11266 (Resolved): Review 11065-rotate-logs
Tom Clegg
02:42 pm Task #11329 (Resolved): Review 10766-install-ws
Tom Clegg
02:41 pm Task #11328 (Resolved): update Upgrading wiki
Tom Clegg
01:49 pm Revision 04997f11 (arvados): Merge branch 'master' into 9132-dockerclient
Radhika Chippada
 

Also available in: Atom