Activity

From 02/07/2017 to 03/08/2017

03/08/2017

10:04 pm Revision c69d4a2b (arvados): 10401: Fix arvcontainer use of pathmapper.
Peter Amstutz
08:33 pm Revision c7143dc9 (arvados): Add ability to use "arvbox start test" to re-run tests without teari...
restarting whole container. no issue # Peter Amstutz
08:33 pm Revision 005595fb (arvados): Arvbox now uses Go websockets server instead of Puma. no issue #
Peter Amstutz
07:31 pm Bug #11226 (Resolved): [SDK] Python client tries to cache discovery document at the same local pa...
The Python SDK instructs httplib2 to cache the discovery document in @~/.cache/arvados/discovery@. However, the goog... Peter Amstutz
06:43 pm Task #11225 (In Progress): Update cwltool
Peter Amstutz
06:43 pm Task #11225 (Resolved): Update cwltool
Peter Amstutz
06:41 pm Revision 16180e3b (arvados): 10401: Passes unit tests.
Peter Amstutz
06:23 pm Revision 08bf82bb (arvados): 10401: Allow Arvados extensions to be used in "requirements" not jus...
Peter Amstutz
06:19 pm Revision dd7710a6 (arvados): 10401: Use use_custom_schema feature for extensions.
Peter Amstutz
06:13 pm Revision e05dddc4 (arvados): 10401: Bump cwltool & schema salad pins.
Peter Amstutz
04:18 pm Bug #4748 (Closed): [API] Explain why there are 21 jobs with log = ""
Tom Clegg
04:17 pm Bug #4451 (Closed): [Workbench] Fiddlesticks when rerunning pipeline with edited template
Tom Clegg
04:16 pm Bug #4416 (Closed): [Workbench] "Manage" permission should be sufficient to edit project's name/d...
Tom Clegg
04:15 pm Bug #4440 (Closed): [Documentation] "Writing a Crunch script" is mysterious
Tom Clegg
04:15 pm Bug #4529 (Closed): [SDKs] Ensure Python and Go SDKs in a given situation always choose either pr...
Tom Clegg
04:14 pm Bug #4488 (Closed): [SDKs] arvados.keep.KeepClient.KeepService should know whether it is a proxy
Tom Clegg
04:12 pm Bug #4335 (Closed): [Crunch] Improve node startup speed. 15 min is too slow.
Tom Clegg
04:11 pm Bug #4218 (Closed): [Keep] Improve speed of /index requests
Tom Clegg
04:11 pm Story #4238 (Closed): [Documentation] Show how to run a pipeline instance without using a template.
Tom Clegg
04:10 pm Story #4305 (Closed): [Node Manager] Investigate and try to address RAM use
Tom Clegg
04:10 pm Bug #3688 (Closed): [API] User should be able to delete a project even though dependent logs exist
Tom Clegg
04:09 pm Story #3820 (Closed): [Crunch] Jobs need code from multiple git repositories
Tom Clegg
04:09 pm Feature #4131 (Closed): [Workbench] Support incremental loading of long job logs, using Range req...
Tom Clegg
04:09 pm Feature #4090 (Closed): Add code coverage report for workbench javascript
Tom Clegg
04:08 pm Story #4424 (Closed): [Keep] [DRAFT] Use stronger hash algorithm (such as SHA-2) for Keep blocks
Moved to [[Changing Keep hash algorithm]] Tom Clegg
04:06 pm Story #11123: Add node manager to install guide
Question, in *doc/install/install-keepstore.html.textile.liquid*: does it make sense to document the runit process, a... Javier Bértoli
03:52 pm Task #11141 (In Progress): Review 6520-nodemanager-docs
Peter Amstutz
03:01 pm Task #11224 (In Progress): Update API usage
Peter Amstutz
03:01 pm Task #11224 (Resolved): Update API usage
Peter Amstutz
02:20 pm Feature #10218 (In Progress): [Crunch2] Gather and record cloud/physical node information for eac...
Lucas Di Pentima
02:14 pm Bug #3624: [API] Remove redundant "updated_at" column
You know, we still have a redundant update_at column... Peter Amstutz

03/07/2017

10:23 pm Story #3812 (Closed): [SDKs] Clients should warn when they're expecting a newer API server
Tom Clegg
10:22 pm Bug #4756 (Rejected): [API] Add migration to change collection uuids to portable_data_hash in old...
Tom Clegg
10:18 pm Revision 5eae182a (arvados): 3853: Do not try to look up collections by PDH for update/delete act...
Tom Clegg
10:03 pm Bug #3954 (Closed): [API] permission_to_update should prohibit all changes to content-addressable...
Tom Clegg
10:03 pm Bug #4003 (Closed): [Crunch] crunch-dispatch errors that relate to a specific job should be avail...
Tom Clegg
10:03 pm Bug #3831 (Closed): [Crunch] inconsistent find_or_create behavior in pipeline runner vs job
Tom Clegg
10:02 pm Bug #3556 (Closed): [Documentation] Avoid using the term "parallel" when referring to Crunch conc...
Tom Clegg
10:01 pm Feature #3768 (Closed): [SDKs] Speculatively pre-fetch data blocks while reading from a collection.
Tom Clegg
10:01 pm Bug #3789 (Closed): [API] Pipeline template (temporarily?) disappears after move to project
Tom Clegg
10:00 pm Bug #3810 (Closed): [Crunch] run-command should bail when $(file) expands to a nonexistent path
Tom Clegg
09:59 pm Story #3795 (Closed): [Crunch/SDKs] Tasks need more retry support
Tom Clegg
09:59 pm Feature #3633 (Closed): [Workbench] Compare pipeline templates, similar to the existing pipeline ...
Tom Clegg
09:59 pm Story #3347 (Closed): [Crunch] Run dev (and real) jobs using a syntax as close as possible to "py...
Tom Clegg
09:58 pm Bug #3470 (Closed): [Workbench] Ajax updated pages don't interact correctly with browser history.
Tom Clegg
09:58 pm Story #3157 (Closed): [Workbench] Admin button to download user list as CSV
Tom Clegg
09:57 pm Bug #3418 (Rejected): [Crunch] crunch-job ignores task sequence
Tom Clegg
09:57 pm Feature #3329 (Closed): [Docker] Loose ends in local docker install
Tom Clegg
09:56 pm Bug #3386 (Closed): [Documentation] Crunch script can exit 111 to indicate transient failure (i.e...
Tom Clegg
09:56 pm Support #2870 (Closed): Write up PGP data curation plan
Tom Clegg
09:56 pm Bug #3624 (Closed): [API] Remove redundant "updated_at" column
Tom Clegg
09:55 pm Feature #3095 (Closed): [Workbench] Add "run" button to each pipeline template in project view
Tom Clegg
09:43 pm Bug #10669: googleapiclient.errors.InvalidJsonError retrieving discovery document
Discovery doc retrieval is done by google-api-client. We ask it to use httplib2's built-in cache mechanism.
Unfort...
Tom Clegg
09:30 pm Bug #11222 (New): Docker load time on compute node is too long
using a-c-r --local, it took 10 minutes to load a 3GB Docker image... Bryan Cosca
09:16 pm Bug #11221: [systemd] Always restart exited services, even after 3 startup failures
11221-always-restart-services @ commit:273a233818ae39e843fab0276f9e381da6645d28 Tom Clegg
09:06 pm Bug #11221 (Resolved): [systemd] Always restart exited services, even after 3 startup failures
Currently, when postgresql is unavailable for ~300ms when arvados-ws is trying to start (which is normal for a bottin... Tom Clegg
09:15 pm Revision 273a2338 (arvados): 11221: Always restart systemd services, even after a few startup fai...
Tom Clegg
08:45 pm Bug #3115 (Resolved): [API] Retrieving a nonexistent keep disk as an admin should not create a ne...
Applied in changeset arvados|commit:533a50bbcee4ece451a65e87066563a5b33ef150. Tom Clegg
08:39 pm Revision 533a50bb (arvados): Merge branch '3115-keep-disk-create-surprise'
closes #3115 Tom Clegg
08:32 pm Revision c97408a5 (arvados): 3115: Create keep_disks implicitly only in #ping action, not #show.
Tom Clegg
08:03 pm Story #3043 (Resolved): [SDKs] Add Go SDK page to doc site with examples similar to Python etc
Tom Clegg
08:03 pm Story #3045 (Closed): [Workbench] Generalize "re-run this job" button
Tom Clegg
08:02 pm Bug #3097 (Closed): [Crunch] Pass script_parameters on stdin instead of JOB_PARAMETER_* environme...
Tom Clegg
08:02 pm Story #3017 (Closed): [Workbench] When comparing pipeline instances, show/highlight the differing...
Tom Clegg
08:02 pm Story #2932 (Closed): Pipeline has default name from pipeline template ("instance of X") and star...
Tom Clegg
08:02 pm Bug #3159 (Closed): Display of the "Add a new keep disk" button in the keep disks page in incorre...
Tom Clegg
08:00 pm Story #2661 (Closed): Security audit
Tom Clegg
08:00 pm Feature #3895 (Closed): set up dev ci server
Tom Clegg
07:59 pm Story #4599 (Closed): [Crunch] crunch-job should not retry tasks after its SLURM allocation is re...
Tom Clegg
07:53 pm Feature #6312 (Resolved): [Workbench] Use download service instead of passing Keep data through R...
Tom Clegg
07:52 pm Feature #7709: [API] Upgrade API server to Rails 4.2
Release notes: http://guides.rubyonrails.org/ Peter Amstutz
07:38 pm Feature #8801 (Duplicate): [Crunch] log free disk space before task starts
Tom Morris
07:37 pm Feature #8196 (Duplicate): [Crunch] Jobs should log the hardware of their compute node(s)
Tom Morris
07:31 pm Bug #4222 (Closed): [Keep] Rename -enforce-permissions to -bypass-read-permissions with default=f...
Tom Morris
07:30 pm Bug #3555 (Closed): [API] Rename nondeterministic to impure
Tom Morris
07:28 pm Feature #6191 (Resolved): Propagate stderr from all srun uses (e.g., clean-tmp and docker-install...
Tom Clegg
07:13 pm Story #4101 (Rejected): Workbench tests: app/controllers/projects_controller.rb
Tom Clegg
07:13 pm Story #4100 (Rejected): Workbench tests: app/helpers/application_helper.rb
Tom Clegg
07:12 pm Story #4099 (Rejected): Workbench tests: app/helpers/pipeline_instances_helper.rb
Tom Clegg
07:12 pm Story #4098 (Rejected): Workbench tests: app/controllers/pipeline_instances_controller.rb
Tom Clegg
07:12 pm Story #4097 (Rejected): Workbench tests: app/controllers/users_controller.rb
Tom Clegg
07:12 pm Story #4096 (Rejected): Workbench tests: app/helpers/provenance_helper.rb
Tom Clegg
07:12 pm Story #4095 (Rejected): Workbench tests: app/controllers/groups_controller.rb
Tom Clegg
07:12 pm Story #4094 (Rejected): Workbench tests: app/controllers/jobs_controller.rb
Tom Clegg
07:12 pm Story #4093 (Rejected): Workbench tests: app/helpers/vcf_pipeline_helper.rb
Tom Clegg
07:12 pm Story #4092 (Rejected): Workbench tests: app/helpers/jobs_helper.rb
Tom Clegg
07:11 pm Bug #6086 (Closed): [API] Creating new job sometimes takes too long
Tom Clegg
07:10 pm Story #6407 (Closed): [Workbench] Collection file trees are collapsible
Tom Clegg
06:59 pm Bug #10350 (Resolved): [Documentation] Update /api/authentication.html with correct information
Fixed in #10346 Peter Amstutz
06:57 pm Bug #9702 (Closed): [Crunch2] Five CWL conformance tests are failing with Crunch2
Closing this as it no longer reflects the current state of CWL on crunch v2.
The latest conformance test results c...
Peter Amstutz
06:55 pm Story #9132 (In Progress): [Crunch2] crunch-run uses official docker client library
Peter Amstutz
06:53 pm Task #8845 (Resolved): Diagnose
Parent ticket is resolved. Peter Amstutz
06:47 pm Task #7601 (Resolved): Review 6358-put-rendezvous
Parent ticket is resolved. Peter Amstutz
06:47 pm Task #6299 (Resolved): Review
Parent ticket is resolved. Peter Amstutz
06:46 pm Task #5680 (Resolved): Async dispatch hooks from CWL engine
CWL is supported an Arvados. Peter Amstutz
06:45 pm Task #5449: Update examples to use new Python Collection SDK and add deprecation notes to old APIs
Workflow examples using CWL don't involve the SDK.
Pipeline examples that use the Python SDK are already marked as...
Peter Amstutz
06:41 pm Feature #4223: [SDKs] arv keep docker --download to get image from Arvados
I still find myself wanting this feature occasionally, although we may want to abandon this particular branch. Peter Amstutz
06:39 pm Task #5681 (Closed): Task dispatch from CWL job runner
arvados-cwl-runner doesn't use job_tasks (which are specific to crunch v1).
Peter Amstutz
06:37 pm Task #5679 (Resolved): CWL job runner framework
CWL is supported on Arvados. Peter Amstutz
06:32 pm Task #2637 (Closed): Add support for filters to generic create API
Parent ticket closed. Peter Amstutz
06:31 pm Task #2628 (Closed): Write wiki page design for simple joins in REST API
Not currently planned. Peter Amstutz
06:31 pm Task #3884 (Closed): Review 3605-improved-dashboard
Revisit dashboard in future tickets. Peter Amstutz
06:30 pm Task #4703 (Resolved): Work on CWL draft 1 spec for tools
CWL supported on Arvados.
Peter Amstutz
06:30 pm Story #11139 (Resolved): [Node manager] Expected MemTotal for each cloud node size
Applied in changeset arvados|commit:4812b5639cfaf724540786fcd331aaa227635c77. Lucas Di Pentima
06:07 pm Story #11139: [Node manager] Expected MemTotal for each cloud node size
Thanks, LGTM Peter Amstutz
05:59 pm Story #11139: [Node manager] Expected MemTotal for each cloud node size
Updates & master branch merge at commit:101e3227b25c16874fa73660bfd7e338fbfe0da2
Tests: https://ci.curoverse.com/job...
Lucas Di Pentima
02:54 pm Story #11139: [Node manager] Expected MemTotal for each cloud node size
Can you add a test that shows that changing the configuration parameter changes the server calculator.
Can you add...
Peter Amstutz
06:29 pm Task #4686 (Resolved): Example wrapping bcbio sub-tool using CWL tool description
bcbio generates CWL that runs on Arvados. Peter Amstutz
06:29 pm Task #4613 (Resolved): Workbench template for generating UI
Workbench generates input form for CWL workflows. Peter Amstutz
06:28 pm Task #4612 (Resolved): Start writing user guide page
CWL support in Arvados is documented. Peter Amstutz
06:27 pm Task #4575 (Resolved): Review 4035-pipeline-support-cwl-tool
CWL is supported in Arvados. Peter Amstutz
06:27 pm Task #4420 (Resolved): Support user+pass database managed locally by SSO server
Implemented commit:49ab94b9d8d2c6e85c22b0e19171255af66db5e2 Peter Amstutz
06:25 pm Revision 4812b563 (arvados): Merge branch '11139-nodemanager-mem-scale-factor'
Closes #11139 Lucas Di Pentima
06:25 pm Task #4146 (Closed): Fix crunch-job
Parent ticket closed. Peter Amstutz
06:24 pm Bug #4028 (Closed): [Crunch] Task stdout appearing in job logs should be labeled "stdout" instead...
Seems unlikely this will be fixed.
Crunch v2 logs distinguish between stdout and stderr.
Peter Amstutz
06:23 pm Task #4144 (Resolved): Use num_retries on regular collection writer instead of robust_put
Was implemented in commit:1281ecab8f2396739ee9232c36796e46cd551426
Also run-command is nearing end-of-life as it i...
Peter Amstutz
06:20 pm Task #3887 (Closed): Fix tests
Revisit dashboard / workbench design in future tickets. Peter Amstutz
06:20 pm Feature #3605 (Closed): [Workbench] improved dashboard page
Will dashboard design in future tickets. Peter Amstutz
06:18 pm Task #3324 (Closed): Build new docker image, run on qr1hi with test data
Bcbio now generates CWL to run on Arvados. Uses upstream bcbio Docker image. Peter Amstutz
06:17 pm Feature #3319 (Closed): [Sample Pipelines] Run monolithic bcbio pipeline on tiny dataset. Package...
Bcbio now generates CWL to run on Arvados. Peter Amstutz
06:16 pm Task #2648 (Closed): Write tests
No longer applicable due to closing of original ticket. Peter Amstutz
06:15 pm Task #2649 (Closed): Assign collection to project
Obsolete by arv-put --project-uuid Peter Amstutz
06:15 pm Task #3230 (Closed): Support bcbio parallelization framework
bcbio generates CWL files which are runnable on Arvados. Peter Amstutz
06:14 pm Task #2647 (Closed): Put files from staging directory into Keep
Obsoleted by enabling arv-put to upload directly to Keep via keepproxy. Peter Amstutz
06:14 pm Task #2643 (Closed): "Ready to ingest" endpoint (in /actions controller?)
Obsoleted by enabling arv-put to write directly to Keep using keepproxy. Peter Amstutz
06:13 pm Feature #2176 (Closed): API server supports atomic find_or_create operation
Containers API uses a different approach. Does not have compelling use case in other APIs. Peter Amstutz
06:12 pm Story #2627 (Closed): Write design for joins in API
Currently no plans to do this. Peter Amstutz
06:07 pm Task #11200 (Resolved): Review 11139-nodemanager-mem-scale-factor
Peter Amstutz
02:46 pm Task #11200 (In Progress): Review 11139-nodemanager-mem-scale-factor
Peter Amstutz
05:54 pm Revision 101e3227 (arvados): 11139: Merge branch 'master' into 11139-nodemanager-mem-scale-factor
Lucas Di Pentima
05:52 pm Revision 9dae7131 (arvados): 11139: Added new test to check for non-default values. Updated examp...
Lucas Di Pentima
03:57 pm Bug #11220 (Resolved): [SDKs] Fix misleading arv-mount/pysdk error messages by removing obsolete ...
Current error message:... Tom Clegg
03:54 pm Bug #9435 (Resolved): keep-balance failing when a small page of collections has the same modified...
In #9998 keep-balance started using count="none" and accommodating multiple pages of identical timestamps, so this sh... Tom Clegg
03:48 pm Story #6830 (Resolved): [API] [keep-balance] Option to return unsigned manifests from collections...
This was implemented in #9998. Tom Clegg
03:39 pm Bug #11190: Containers seem to run more than once, which isn't supposed to happen
... Tom Clegg
03:15 pm Bug #11166 (Resolved): [Crunch2] crunchrun.go should avoid name collisions when creating log coll...
Applied in changeset arvados|commit:a1b0149879593df786807afbefb26d4687ebe161. Tom Clegg
03:01 pm Bug #11166: [Crunch2] crunchrun.go should avoid name collisions when creating log collections
11166-log-name-collision @ commit:cb0b666f86ce87d35667e8e23e3775b703a37841 Tom Clegg
02:45 pm Bug #11166: [Crunch2] crunchrun.go should avoid name collisions when creating log collections
LGTM. Peter Amstutz
03:11 pm Revision a1b01498 (arvados): Merge branch '11166-log-name-collision'
closes #11166 Tom Clegg
03:02 pm Revision fce072c7 (arvados): 8567: Add check for ARVADOS_API_HOST_INSECURE
Peter Amstutz
02:46 pm Task #11196 (Resolved): Review 11166-log-name-collision
Peter Amstutz
02:27 pm Task #11196 (In Progress): Review 11166-log-name-collision
Peter Amstutz
02:26 pm Revision 123251c6 (arvados): 8567: Rename docker19-migrate to migrate-docker19 for consistency with
arv-migrate-docker19. Add docstrict to migrate19() function. Peter Amstutz
02:18 pm Revision 57fec89d (arvados): 8567: Add status reporting to migrate script.
Peter Amstutz
02:18 pm Revision 09fc7249 (arvados): 8567: Move out of tools/ into sdk/python and docker/docker19-migrate.
Peter Amstutz
02:18 pm Revision 80d9b40b (arvados): 8567: Docker image migration WIP.
Peter Amstutz
02:18 pm Revision b5886da1 (arvados): 8567: Creates migration links.
Peter Amstutz
02:18 pm Revision 5780b539 (arvados): 9132: WIP switching to using official Docker Go client.
Peter Amstutz

03/06/2017

10:38 pm Story #11139: [Node manager] Expected MemTotal for each cloud node size
Updates on @11139-nodemanager-mem-scale-factor@ branch at commit:b6423b5
Test run: https://ci.curoverse.com/job/deve...
Lucas Di Pentima
10:04 pm Revision b6423b5e (arvados): 11139: Added default config parameter 'node_mem_scaling' to be appli...
Added test checking the default config and that the value is actually applied to the server calculations. Lucas Di Pentima
08:58 pm Bug #11166 (In Progress): [Crunch2] crunchrun.go should avoid name collisions when creating log c...
Tom Clegg
08:24 pm Task #11212 (Resolved): use ensure_unique_name flag or timestamp
Tom Clegg
03:37 pm Task #11212 (Resolved): use ensure_unique_name flag or timestamp
Tom Clegg
07:59 pm Revision cb0b666f (arvados): 11166: Use ensure_unique_name to avoid collisions when saving logs a...
Tom Clegg
07:52 pm Revision 58342141 (arvados): 8567: Move out of tools/ into sdk/python and docker/docker19-migrate.
Peter Amstutz
07:31 pm Revision f2c45e2c (arvados): 8567: Add status reporting to migrate script.
Peter Amstutz
04:20 pm Story #11171: [Nodemanager] Packaging for compute node setup scripts
relevant modules in puppet:
arvados-compute-and-crunch-dispatch-dependencies
arvados-compute-dependencies
arvado...
Nico César
03:39 pm Revision 93384273 (arvados): 8567: Creates migration links.
Peter Amstutz
03:35 pm Bug #11138 (Resolved): Protection against running new style docker 1.10+ image on old docker host
Applied in changeset arvados|commit:a4a3227c634d8cfd54a23205be727e910032cbe9. Tom Clegg
02:09 pm Bug #11138: Protection against running new style docker 1.10+ image on old docker host
Thanks! LGTM. Lucas Di Pentima
03:31 pm Revision a4a3227c (arvados): Merge branch '11138-docker-load-fail'
closes #11138 Tom Clegg
03:23 pm Revision cee1c18c (arvados): 8567: Docker image migration WIP.
Peter Amstutz
02:10 pm Task #11197 (Resolved): Review 11138-docker-load-fail
Lucas Di Pentima

03/03/2017

10:05 pm Revision 47c56297 (arvados): Add missing install step: add keep-balance token to keepstore configs.
No issue # Tom Clegg
09:53 pm Revision 2e5c44a6 (arvados): Remove obsolete GOMAXPROCS advice.
No issue # Tom Clegg
09:50 pm Bug #11168 (Resolved): [API] Use JSON instead of YAML for serialized fields in database
Applied in changeset arvados|commit:660a6143ecf1e777f33bd84183ba9e821e1d7a8e. Tom Clegg
09:23 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
commit:07e4083ea451913b988d77e8e4c926da8ad844a4
"Double-decode serialized fields if database was mangled by downgr...
Tom Clegg
06:22 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
... Peter Amstutz
05:02 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
... Peter Amstutz
04:59 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
... Peter Amstutz
04:45 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
Hold on, now I'm having trouble starting the API server in arvbox. I don't know if something is just corrupted/confu... Peter Amstutz
04:30 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
LGTM @ commit:594e00f Peter Amstutz
05:43 am Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
> I notice that a couple of places have been switched to use SafeJSON, but they still have @require 'oj'@. The speci... Tom Clegg
09:46 pm Revision 660a6143 (arvados): Merge branch '11168-serialize-json'
closes #11168 Tom Clegg
09:44 pm Bug #11138: Protection against running new style docker 1.10+ image on old docker host
Lucas Di Pentima wrote:
> On line 428: Could it be that "docker load" fails for some reason, and the printed message...
Tom Clegg
07:59 pm Bug #11138: Protection against running new style docker 1.10+ image on old docker host
On line 428: Could it be that "docker load" fails for some reason, and the printed message still say that it exited w... Lucas Di Pentima
06:06 am Bug #11138: Protection against running new style docker 1.10+ image on old docker host
11138-docker-load-fail uses @docker inspect@ (before and after @docker load@) to assure that @docker run aaa123@ will... Tom Clegg
09:41 pm Revision f7d32ebc (arvados): 11138: Show actual image ID when checking whether docker image is lo...
Tom Clegg
09:18 pm Revision 07e4083e (arvados): 11168: Double-decode serialized fields if database was mangled by do...
Tom Clegg
04:31 pm Task #11195 (Resolved): Review 11168-serialize-json
Peter Amstutz
04:22 pm Task #11197 (In Progress): Review 11138-docker-load-fail
Lucas Di Pentima
06:10 am Task #11207 (Resolved): Make docker_install_script more robust
Tom Clegg
05:52 am Revision dd08615e (arvados): 11138: Test for docker image after loading, in case docker-load erro...
Tom Clegg
05:26 am Revision 594e00f9 (arvados): 11168: Always deep-sort before comparing in where_serialized.
Tom Clegg
05:25 am Revision 6fe6fb4a (arvados): 11168: Remove unused import.
Tom Clegg
05:17 am Revision c7a44e53 (arvados): 11168: Revert serialization change in order to avoid breaking job re...
Tom Clegg

03/02/2017

11:12 pm Revision 2ab1b540 (arvados): Merge branch 'master' into 11071-fts-perf-test
Radhika Chippada
11:06 pm Bug #11071: [Workbench] qr1hi performance tests are broken
Branch 11071-fts-perf-test @ commit:cad4856 takes a stab at using count='none' for full text search in workbench.
...
Radhika Chippada
10:53 pm Revision cad4856b (arvados): 11017: Use count='none' for full text search on workbench.
Update groups.load_searchable_objects which is used by contents method
to work with multiple class types when count='...
Radhika Chippada
09:08 pm Bug #11209 (Resolved): stuck keep fuse mounts not cleared by crunch-job
crunch-job attempts to unmount any fuse filesystems that are mounted under $CRUNCH_TMP but it attempts to do so only ... Joshua Randall
08:48 pm Story #11177: [GA4GH] Participate in interoperability challenge
https://www.synapse.org/#!Synapse:syn8080249/wiki/415165 Peter Amstutz
05:43 pm Story #11177 (Resolved): [GA4GH] Participate in interoperability challenge
Peter Amstutz
05:43 pm Task #11193 (Resolved): Submit writeup
Peter Amstutz
03:48 pm Bug #11138 (In Progress): Protection against running new style docker 1.10+ image on old docker host
Tom Clegg
03:48 pm Task #11207 (In Progress): Make docker_install_script more robust
Tom Clegg
03:48 pm Task #11207 (Resolved): Make docker_install_script more robust
Tom Clegg
03:42 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
11168-serialize-json @ commit:766ddd6
I notice that a couple of places have been switched to use SafeJSON, but the...
Peter Amstutz
02:45 pm Story #11139 (In Progress): [Node manager] Expected MemTotal for each cloud node size
Lucas Di Pentima
02:33 pm Task #11195 (In Progress): Review 11168-serialize-json
Peter Amstutz
06:04 am Revision e9696655 (arvados): 11183: Add "setup -wait" flag
Tom Clegg

03/01/2017

11:07 pm Revision 345dfd5a (arvados): 11168: Add missing require.
Tom Clegg
10:59 pm Revision 766ddd6d (arvados): 11168: Prohibit down-migration to YAML-only codebase.
Tom Clegg
09:24 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
11168-serialize-json @ commit:d7c84d69bb62d61bc671b2d5e0ad4ed42dbeb7c0 Tom Clegg
09:06 pm Bug #11168 (In Progress): [API] Use JSON instead of YAML for serialized fields in database
Tom Clegg
09:09 pm Revision d7c84d69 (arvados): 11168: Change db serialize from YAML to JSON.
Tom Clegg
09:01 pm Task #11205 (Resolved): Review branch 11071-fts-perf-test
Radhika Chippada
08:51 pm Task #11200 (Resolved): Review 11139-nodemanager-mem-scale-factor
Lucas Di Pentima
08:51 pm Task #11199 (Resolved): Review 10218-record-node-info
Lucas Di Pentima
08:51 pm Task #11198 (New): Review
Peter Amstutz
08:51 pm Task #11197 (Resolved): Review 11138-docker-load-fail
Tom Clegg
08:51 pm Task #11196 (Resolved): Review 11166-log-name-collision
Tom Clegg
08:51 pm Task #11195 (Resolved): Review 11168-serialize-json
Tom Clegg
08:50 pm Task #11194 (Resolved): Review 10401-limit-dir-expansion
Peter Amstutz
08:49 pm Task #11193 (Resolved): Submit writeup
Peter Amstutz
08:49 pm Task #11192 (Resolved): Review 9132-dockerclient
Peter Amstutz
08:48 pm Task #11191 (Resolved): Review 11158-fuse-projects
Peter Amstutz
08:45 pm Bug #11071 (In Progress): [Workbench] qr1hi performance tests are broken
Tom Morris
08:43 pm Bug #10729: [Crunch2] Propagate error messages if sbatch command succeeds but crunch-run can't ru...
We can also try running "srun" with the same flags, since it is more likely to return useful errors from the subproce... Peter Amstutz
08:37 pm Bug #11190 (Resolved): Containers seem to run more than once, which isn't supposed to happen
Example: tb05z-dz642-eie1eal1059y9bb Tom Clegg
08:28 pm Bug #11166: [Crunch2] crunchrun.go should avoid name collisions when creating log collections
Should be a matter of using the ensure_unique_name flag. Tom Clegg
05:58 pm Bug #11166: [Crunch2] crunchrun.go should avoid name collisions when creating log collections
We've been seeing CR logs with messages like this:... Lucas Di Pentima
08:24 pm Task #11119 (Resolved): Review 7995-keep-balance-docs
Tom Clegg
08:22 pm Task #11164 (Resolved): Review 9277-container-output
Peter Amstutz
08:22 pm Bug #10585 (Resolved): crunch doesn't end jobs when their arv-mount dies
Tom Clegg
08:12 pm Revision 15b33da8 (arvados): 9216: Require size hints in manifest validation.
Tom Clegg
08:11 pm Bug #10035 (In Progress): [FUSE] Determine why bcl2fastq doesn't work with writable keep mount
Peter Amstutz
08:11 pm Bug #10035 (Resolved): [FUSE] Determine why bcl2fastq doesn't work with writable keep mount
Peter Amstutz
08:11 pm Feature #6520 (Resolved): [Node Manager] [Crunch2] Take queued containers into account when compu...
Peter Amstutz
08:10 pm Feature #10764 (Resolved): [arvados-ws] Write unit tests
Applied in changeset arvados|commit:2db0c3a44d8f12e3a566c796bdecaa1de150b155. Tom Clegg
02:14 pm Feature #10764: [arvados-ws] Write unit tests
10764-ws-tests
commit:f36162457a771824059fefa098a3ffb89c59263f
https://ci.curoverse.com/job/developer-run-tests/180/
Tom Clegg
08:09 pm Bug #9437 (Resolved): keep-balance doesn't seem to be reducing replication level of blocks
Tom Clegg
08:04 pm Revision 2db0c3a4 (arvados): Merge branch '10764-ws-tests'
closes #10764 Tom Clegg
08:03 pm Task #11112 (Resolved): unit tests
Tom Clegg
08:03 pm Bug #11186 (Duplicate): Priority 0 state shouldn't be labeled "Ready"
Peter Amstutz
07:28 pm Bug #8113 (Resolved): "Download the full log" in log tab should show the log in the browser, not ...
Tom Clegg
07:00 pm Story #10401 (In Progress): [CWL] Limit expansion of Directory inputs
Peter Amstutz
06:24 pm Story #11185 (Resolved): [Workbench] Add ability to disable job reuse when running a workflow
* Either "re-run container request" button should present a dialog box asking the user if they want job reuse or not,... Tom Morris
05:41 pm Bug #8685: [FUSE] Add timestamp to arv-mount --logfile option
Nico - can you please expand/update the description for this. Tom Morris
05:37 pm Bug #8624: [FUSE] arv-mount `by_tag` directory only shows 100 tags and tags that exist are not ac...
Assigning to Peter for triage/grooming Tom Morris
03:58 pm Revision a58e9ca2 (arvados): 10764: De-duplicate real/test server startup. Add test for broken co...
Tom Clegg
03:37 pm Task #11108 (Resolved): Review 11015-crunch-run-output-upload
Radhika Chippada
02:14 pm Task #11118 (Resolved): Review 10777-die-if-arv-mount-dies
Tom Clegg
07:52 am Revision f3616245 (arvados): 10764: Simplify test server shutdown.
Tom Clegg
07:36 am Revision b3fa9983 (arvados): 10764: Test v0 session.
Tom Clegg

02/28/2017

11:23 pm Feature #11184 (New): [Keep] Support multiple storage tiers
As an Arvados system administrator, I'd want the ability take advantage of the cool/cold storage classes offered by c... Tom Morris
10:20 pm Bug #10777 (Resolved): [Crunch2] crunch-run: stop the container and fail if arv-mount dies before...
Applied in changeset arvados|commit:a98594f82f1786b09d9d1e0f57a651d4eae3474a. Tom Clegg
12:52 am Bug #10777: [Crunch2] crunch-run: stop the container and fail if arv-mount dies before the contai...
LGTM Radhika Chippada
10:08 pm Revision a98594f8 (arvados): Merge branch '10777-die-if-arv-mount-dies'
closes #10777 Tom Clegg
08:54 pm Feature #11183: [Deployment] Use consul to report health of services/components
work in progress on source:/@11183-setup-agent Tom Clegg
08:41 pm Feature #11183 (In Progress): [Deployment] Use consul to report health of services/components
Tom Clegg
08:52 pm Revision c78289e5 (arvados): 11183: Add "arvados-admin setup"
Tom Clegg
07:58 pm Bug #11168: [API] Use JSON instead of YAML for serialized fields in database
This sounds like a nice improvement. That PG page has some warnings to which we should need to probably pay careful a... Javier Bértoli
07:08 pm Story #11139: [Node manager] Expected MemTotal for each cloud node size
Based on discussion: reduce the predicted RAM available for node by 5% Peter Amstutz
06:35 pm Feature #11015 (Resolved): Improve throughput of crunch-run output-uploading stage using multi-th...
Applied in changeset arvados|commit:0b125c52cd816e6a4120c414d3817f354cad1055. Radhika Chippada
06:18 pm Feature #11015: Improve throughput of crunch-run output-uploading stage using multi-threaded tran...
> Are the upload workers number configurable?
Number of writers >1 will help us as this let's us use multiple thre...
Radhika Chippada
04:44 pm Feature #11015: Improve throughput of crunch-run output-uploading stage using multi-threaded tran...
Sorry for taking so long to send comments, was trying to understand the code the best I could.
One question: Are the...
Lucas Di Pentima
06:29 pm Revision 0b125c52 (arvados): closes #11015
Merge branch '11015-crunch-run-output-upload' Radhika Chippada
06:23 pm Revision b7f4b12d (arvados): Merge branch 'master' into 11015-crunch-run-output-upload
Radhika Chippada
02:25 pm Task #11111 (Resolved): Review 11097-reuse-impure
Radhika Chippada
06:09 am Revision 784371c8 (arvados): 10764: Test v0 session. (WIP)
Tom Clegg
02:49 am Revision aea8596a (arvados): 10764: Permission tests. Support PDH permission check.
Tom Clegg

02/27/2017

10:51 pm Revision ca812f58 (arvados): 10764: Add unit tests
Tom Clegg
07:17 pm Story #11177 (Resolved): [GA4GH] Participate in interoperability challenge
https://www.synapse.org/#!Synapse:syn8080249/wiki/412159
# Get tool & test data
# Import / run it on Arvados
# D...
Peter Amstutz
07:07 pm Task #11118 (In Progress): Review 10777-die-if-arv-mount-dies
Radhika Chippada
06:39 pm Task #11108 (In Progress): Review 11015-crunch-run-output-upload
Lucas Di Pentima
06:25 pm Revision e86baeea (arvados): 11015: use multiple writers to increate throughput of goUpload.
Radhika Chippada
06:23 pm Task #11112 (In Progress): unit tests
Tom Clegg
04:54 pm Revision 58ed4ee3 (arvados): 10979: Add missing SqueueChecker initialization.
refs #10979 Tom Clegg
03:30 pm Story #11171 (New): [Nodemanager] Packaging for compute node setup scripts
All software installed into compute node images, including setup scripts, should be packaged so that documentation fo... Peter Amstutz
01:40 pm Bug #11002 (Resolved): [arv-put] crash in arvfile on upload NoneType object has no attribute 'clo...
Applied in changeset arvados|commit:9889021db676c152d5b62f399bd27024b5a51ae1. Lucas Di Pentima
01:37 pm Revision 9889021d (arvados): Merge branch '11002-arvput-crash-fix'
Closes #11002 Lucas Di Pentima
12:42 pm Revision 1130ddc2 (arvados): 11002: Merge branch 'master' into 11002-arvput-crash-fix
Lucas Di Pentima
12:41 pm Revision 954b101b (arvados): 11002: Added note explaining why we're expecting a SystemExit to cat...
Lucas Di Pentima
08:59 am Revision ceb95004 (arvados): more vault
Tom Clegg

02/26/2017

07:03 pm Bug #11170 (Resolved): Stale squeue processes on c97qk caused by "crunch-dispatch --jobs"
There were a total of 4694 of these processes, representing a significant resource leak.... Ward Vandewege
10:12 am Revision 307e7af0 (arvados): vault
Tom Clegg
12:35 am Revision a4512ef7 (arvados): Merge branch 'master' into deploy-agent
Conflicts:
build/run-tests.sh
Tom Clegg
12:32 am Revision cea3337f (arvados): multiple servers, add docker-compose test
Tom Clegg

02/25/2017

10:48 pm Revision 37972640 (arvados): gitignore
Tom Clegg
07:05 am Bug #10777: [Crunch2] crunch-run: stop the container and fail if arv-mount dies before the contai...
10777-die-if-arv-mount-dies @ commit:c106221bf8c90fd80ca2c19a578fd7faec985229 Tom Clegg
07:04 am Task #11169 (Resolved): Detect arv-mount death & error out
Tom Clegg
06:57 am Revision 2ac112e2 (arvados): 10777: Stop container if arv-mount dies before container exits.
Tom Clegg
06:57 am Revision c106221b (arvados): 10777: Close and flush logs right away instead of waiting for next t...
Tom Clegg
12:34 am Revision db371c57 (arvados-dev): arvados-version-server:
* style improvements: don't stutter
* test improvement: stop printing log output during tests
No issue #
Ward Vandewege

02/24/2017

10:46 pm Revision e47bb9c1 (arvados): run-tests.sh exit non-zero if gofmt fails
No issue # Tom Clegg
10:29 pm Revision 1042f200 (arvados): 10777: Stop container if arv-mount dies before container exits.
Tom Clegg
09:55 pm Task #11169 (In Progress): Detect arv-mount death & error out
Tom Clegg
09:55 pm Task #11169 (Resolved): Detect arv-mount death & error out
Tom Clegg
09:46 pm Revision 17f4a03d (arvados): 10979: Check for orphans only once at startup. Add missing Lock() in
squeue checker. Avoid holding mtx while waiting for API response.
Ensure RunContainer actually gets called in test ca...
Tom Clegg
08:56 pm Revision dccfd637 (arvados): build improvement: really include apps/workbench_functionals when
apps/workbench is specified.
No issue #
Ward Vandewege
07:43 pm Revision 353eb4f4 (arvados): Merge branch '6347-log-timestamps'
closes #6347 Tom Clegg
07:41 pm Bug #11168 (Resolved): [API] Use JSON instead of YAML for serialized fields in database
Currently hashes (like log properties) and arrays (like api_client_authorization scopes) are encoded in YAML, which i... Tom Clegg
07:30 pm Bug #10629 (Resolved): [FUSE] slow enumerating files by collection uuid
Applied in changeset arvados|commit:9b2aa42213a7d333bbe93e040c2d152a70e9b5af. Peter Amstutz
07:25 pm Story #9277 (Resolved): [Crunch2] System-owned container outputs should be garbage-collected
Applied in changeset arvados|commit:cca78a112d6a2aa4f76c1956cfe8ec2d43a68759. Peter Amstutz
06:30 pm Story #9277: [Crunch2] System-owned container outputs should be garbage-collected
It would be nice (if it doesn't already exist) to add a test such as "error setting output on running container if tr... Radhika Chippada
04:12 pm Story #9277: [Crunch2] System-owned container outputs should be garbage-collected
Additional fixes on 9277-container-output
Tests:
https://ci.curoverse.com/job/developer-run-tests/178/
Peter Amstutz
07:24 pm Task #11114 (Resolved): Review 10629-fuse-listing-perf
Peter Amstutz
07:24 pm Revision 9b2aa422 (arvados): Merge branch '10629-fuse-listing-perf' closes #10629
Peter Amstutz
07:22 pm Task #11041 (Resolved): Review 11002-arvput-crash-fix
Peter Amstutz
07:21 pm Revision cca78a11 (arvados): Merge branch '9277-container-output' closes #9277
Peter Amstutz
07:20 pm Revision b86728d2 (arvados): 9277: Add test that setting trashed, unreable collection is disallowed.
Peter Amstutz
06:53 pm Story #11167 (Resolved): [Workbench] Remove arv-get file download fallback
Currently, if keep-web isn't configured, Workbench will fallback to arv-get and Rails streaming to serve files. Howe... Peter Amstutz
06:51 pm Bug #11166 (Resolved): [Crunch2] crunchrun.go should avoid name collisions when creating log coll...
Lucas Di Pentima
06:45 pm Revision 00d95f0f (arvados): 6347: Use RFC3339Nano to render timestamps loaded from serialized fi...
Psych (YAML) serializes timestamps as ISO8601-with-space-separators,
and safe_load deserializes them to Time even wit...
Tom Clegg
06:28 pm Task #11164 (In Progress): Review 9277-container-output
Radhika Chippada
04:12 pm Task #11164 (Resolved): Review 9277-container-output
Peter Amstutz
06:17 pm Revision c702045c (arvados): refs #10979
Merge branch '10979-cancelled-job-nodes' Radhika Chippada
06:16 pm Revision 1a06dafa (arvados): 10979: fix failing test
Radhika Chippada
06:14 pm Bug #11002: [arv-put] crash in arvfile on upload NoneType object has no attribute 'closed'
Rest LGTM. Peter Amstutz
05:02 pm Bug #11002: [arv-put] crash in arvfile on upload NoneType object has no attribute 'closed'
From IRC: make a note around line L454 that because we catch signals, we expect a SystemExit, not KeyboardInterrupt.
...
Peter Amstutz
04:10 pm Story #11163 (In Progress): [GA4GH] Participate in workflow interoperability activities
Peter Amstutz
04:04 pm Story #11162 (New): [GA4GH] Support workflow execution API
Peter Amstutz
04:03 pm Story #11161 (New): [GA4GH] Support task execution API using crunch v2 containers
Peter Amstutz
04:02 pm Story #11160 (New): [GA4GH] Expose workflow records via tool registry API
Peter Amstutz
04:02 pm Story #11159 (New): [GA4GH] Import from tool registry API
Support import of tools from tool registry API.
https://github.com/ga4gh/tool-registry-schemas
Initial capabili...
Peter Amstutz
04:01 pm Revision a04b28a6 (arvados): 9277: arvados-cwl-runner sets "is_trashed" when directly setting out...
Peter Amstutz
03:53 pm Revision a6e4bccc (arvados): 9277: Container output check must be unscoped to include trashed col...
Peter Amstutz
09:17 am Revision c7df9c8c (arvados): 6347: Use RFC3339Nano to render timestamps loaded from serialized fi...
Psych (YAML) serializes timestamps as ISO8601-with-space-separators,
and safe_load deserializes them to Time even wit...
Tom Clegg
06:23 am Bug #11158 (Resolved): [FUSE] Hang on simple FUSE operation and when logging in again later
I changed directory on a shell node to a ~/keep/home/... project which each had a single sub project, which each had ... Tom Morris
05:54 am Revision 6029ffdc (arvados): closes #10979
Merge branch '10979-cancelled-job-nodes' Radhika Chippada
05:54 am Revision a6c7c8db (arvados): Merge branch 'master' into 10979-cancelled-job-nodes
Radhika Chippada
05:52 am Revision e2f2a9dd (arvados): 10979: ruby way of doing it
Radhika Chippada

02/23/2017

10:04 pm Bug #11071: [Workbench] qr1hi performance tests are broken
Looks like we need to use count=none for Workbench search... Tom Clegg
09:52 pm Bug #11002: [arv-put] crash in arvfile on upload NoneType object has no attribute 'closed'
Updates at commit:d2a202e
Added a specific exception class for this kind of errors, so it's not confused with othe...
Lucas Di Pentima
08:56 pm Revision d2a202ef (arvados): 11002: Don't save the state and log the stack trace before quitting ...
Updated tests to reflect this new behaviour. Lucas Di Pentima
07:10 pm Bug #11156 (Resolved): [Crunch2] 9tee4-xvhdp-0m3a4ys52od7dhs was stuck until c-d-s was restarted
Applied in changeset arvados|commit:19dcf1ed20214b8349c76037ba8679db07bf44fc. Tom Clegg
07:09 pm Bug #11156: [Crunch2] 9tee4-xvhdp-0m3a4ys52od7dhs was stuck until c-d-s was restarted
c-d-s was stuck in an infinite loop due to a bad condition in its "get next page" loop. Just kept repeating the same ... Tom Clegg
04:56 pm Bug #11156: [Crunch2] 9tee4-xvhdp-0m3a4ys52od7dhs was stuck until c-d-s was restarted
Looks like @9tee4-dz642-2a5iga7lyeg2rog@ was queued the whole time, but c-d-s didn't pick it up until after restart f... Tom Clegg
04:07 pm Bug #11156 (Resolved): [Crunch2] 9tee4-xvhdp-0m3a4ys52od7dhs was stuck until c-d-s was restarted

9tee4-xvhdp-0m3a4ys52od7dhs was stuck in the running state for almost 16 hours, with one container request (step1_4...
Ward Vandewege
07:04 pm Revision 19dcf1ed (arvados): 11156: Fix infinite loop condition.
closes #11156 Tom Clegg
06:55 pm Bug #11157 (New): [arv-put] When debug mode is active, report the cache file name being to be used
When trying to debug cache file locking problems, it's useful that arv-put report which file is going to use. Can be ... Lucas Di Pentima
06:18 pm Story #9277 (Feedback): [Crunch2] System-owned container outputs should be garbage-collected
... Peter Amstutz
04:37 pm Bug #10629: [FUSE] slow enumerating files by collection uuid
LGTM.
Local @sdk/python@ and @services/fuse@ tests ran without issues.
Tried to do some benchmarking using arvbox b...
Lucas Di Pentima
02:56 pm Task #11114 (In Progress): Review 10629-fuse-listing-perf
Lucas Di Pentima

02/22/2017

11:51 pm Bug #5901: [API] Improve performance of large requests in parallel
This has improved by over an order of magnitude(!) since 2015 which is great, but 20 seconds to fetch 75 MB still see... Tom Morris
11:23 pm Bug #9732: [FUSE] performance issues stat() in python
This seems like it could be related to #10629 Tom Morris
10:40 pm Bug #5713: [FUSE] File access sometimes takes too long on su92l
Has this problem been seen recently? It was reported over 2 years ago and a lot has changed since then, include a bun... Tom Morris
09:53 pm Bug #11071: [Workbench] qr1hi performance tests are broken
Another bit of evidence is this long API call from the logs on qr1hi which looks like it basically accounts for all o... Tom Morris
09:48 pm Bug #11155 (New): [Docs] Document keepstore S3 volume configuration
Add a "Configure S3 storage" page similar to the "Configure Azure Blob storage" page (http://doc.arvados.org/install/... Tom Clegg
09:48 pm Bug #11154 (New): [Docs] Migrate install docs from command line flags to YAML configs (keepstore,...
Currently, install docs recommend setting CLI flags, but these are deprecated.
Update install docs to use @/etc/ar...
Tom Clegg
09:45 pm Story #7995 (Resolved): [Documentation] Document Keep Balance setup in the Install Guide
Applied in changeset arvados|commit:1e6a756a10a1c0a77aeea5041844ba3a572bdd70. Tom Clegg
08:35 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
Great, it LGTM, then. Javier Bértoli
08:05 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
Javier Bértoli wrote:
> I notice you added these two flags and removed *-never-delete=true*. Is that correct, or jus...
Tom Clegg
05:27 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
Tom Clegg wrote:
> Updated the dry run instructions:
>
> To do this, edit your keep-balance startup script to use...
Javier Bértoli
04:45 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
Javier Bértoli wrote:
> Keepproxy is optional, as I understand it. If so, can I have more than one keep-balance, ins...
Tom Clegg
04:38 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
Updated the dry run instructions:
To do this, edit your keep-balance startup script to use the flags @-commit-pull...
Tom Clegg
04:19 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
Tom Morris,
* From this text:
bq. Keep-balance can be installed anywhere with network access to Keep services. ...
Javier Bértoli
04:18 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
Tom Morris wrote:
> I made a few copy edits and pushed them to the branch. Please review them to make sure that thin...
Tom Clegg
09:44 pm Feature #11153 (New): [API] create_token script should allow operator to provide token scopes and...
h3. Background
Currently the install guide relies on the @script/create_superuser_token.rb@ program in the apiserv...
Tom Clegg
09:41 pm Bug #10629: [FUSE] slow enumerating files by collection uuid
10629-fuse-listing-perf ready for review. Peter Amstutz
08:35 pm Bug #10629 (In Progress): [FUSE] slow enumerating files by collection uuid
... Peter Amstutz
04:16 pm Bug #10629: [FUSE] slow enumerating files by collection uuid
... Peter Amstutz
09:36 pm Revision 1e6a756a (arvados): Merge branch '7995-keep-balance-docs'
closes #7995 Tom Clegg
09:33 pm Revision 2a30eacc (arvados): Remove pidfiles after shutting down test servers.
No issue # Tom Clegg
09:28 pm Revision aedb62c7 (arvados): 10629: Don't flush dirhandles.
Peter Amstutz
09:08 pm Revision fe85406f (arvados): 10629: Make tracking and dirtying of _committed flag efficient.
Peter Amstutz
08:40 pm Revision 04fb9a14 (arvados): 10629: improve debug logging
--debug includes Keep logging.
--logfile includes timestamps.
Peter Amstutz
08:37 pm Task #11119 (Feedback): Review 7995-keep-balance-docs
Javier Bértoli
07:49 pm Revision 25af6c40 (arvados): Merge branch 'master' into 10979-cancelled-job-nodes
Radhika Chippada
07:48 pm Revision 3a31db02 (arvados): 10979: refactor squeue invocations
Radhika Chippada
06:26 pm Revision 56a99279 (arvados): 11002: Track this specific error with its own exception class, for f...
Lucas Di Pentima
06:25 pm Bug #11097 (Resolved): [API] Reuse containers even when multiple matching containers exist with d...
Applied in changeset arvados|commit:0c529ed05805507b4d2c903b9587e9b61cec5ee6. Tom Clegg
06:19 pm Revision 0c529ed0 (arvados): Merge branch '11097-reuse-impure'
closes #11097 Tom Clegg
04:45 pm Revision d8f1b7e8 (arvados): 7995: Add note about one keep-balance process at a time.
Tom Clegg
04:29 pm Revision 11f5e94c (arvados): 7995: Fix up dry-run instructions.
Tom Clegg
04:24 pm Revision ced8b355 (arvados): 7995: Fix up inconsistent "e.g." vs. "e.g.,".
Tom Clegg
04:02 pm Revision d2c9ca97 (arvados): 7995: Copy edits.
Tom Clegg
03:38 pm Feature #11015 (In Progress): Improve throughput of crunch-run output-uploading stage using multi...
Radhika Chippada
03:29 pm Revision 423d27d0 (arvados): Fix dispatch panic when processing an update after tracker has been ...
refs #11151 Tom Clegg

02/21/2017

11:46 pm Bug #11071: [Workbench] qr1hi performance tests are broken
Actually the commits are likely a red herring since it doesn't look like qr1hi was deployed on or immediately after D... Tom Morris
11:15 pm Bug #11071: [Workbench] qr1hi performance tests are broken
Here are the first few commits after the last successful pass (read from bottom up):
commit b27f1106beb0fb321ee123...
Tom Morris
09:53 pm Bug #10729: [Crunch2] Propagate error messages if sbatch command succeeds but crunch-run can't ru...
Thoughts about how to retrieve the error messages:
* If the cluster is set up with shared home directories (which is...
Tom Clegg
09:40 pm Revision fc742c91 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
09:39 pm Revision 1b617d62 (arvados-dev): Fix detection of version of arvados/jobs image to install.
No issue # Ward Vandewege
09:35 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
I made a few copy edits and pushed them to the branch. Please review them to make sure that things are still technica... Tom Morris
09:35 pm Revision fc54d2d3 (arvados): A few copy edits
Tom Morris
09:34 pm Revision 6159bebd (arvados): Document epydoc dependency
Tom Morris
09:08 pm Task #11119 (In Progress): Review 7995-keep-balance-docs
Tom Morris
08:57 pm Bug #11097: [API] Reuse containers even when multiple matching containers exist with differing ou...
> Yes, trashing the output or log collection would accomplish this ... "poking re-use in the eye" procedure seems wo... Radhika Chippada
08:36 pm Bug #11097: [API] Reuse containers even when multiple matching containers exist with differing ou...
Radhika Chippada wrote:
> * I think moving “select_readable_pdh” to the line above the declaration of “candidates” a...
Tom Clegg
07:58 pm Bug #11097: [API] Reuse containers even when multiple matching containers exist with differing ou...
* I think moving “select_readable_pdh” to the line above the declaration of “candidates” at line 85 would help improv... Radhika Chippada
08:27 pm Revision 802af81e (arvados): 11097: Clarify reuse query.
Tom Clegg
08:27 pm Revision 1732df24 (arvados): 11097: Update docs to reflect new container reuse behavior.
Tom Clegg
07:58 pm Task #11111 (In Progress): Review 11097-reuse-impure
Radhika Chippada
07:56 pm Bug #11148 (New): [Crunch2] Propagate dispatch error messages (e.g., sbatch fails) to user via lo...
h3. Background
If crunch-dispatch-slurm encounters an error while trying to submit a container to the slurm queue,...
Tom Clegg
07:44 pm Story #9619 (Rejected): [Crunch2] [API] Default vcpus value to fill in for ContainerRequests
Implemented by client a-c-r Tom Morris
07:44 pm Story #9620 (Rejected): [Crunch2] [API] Default ram value to fill in for ContainerRequests
Implemented by client cwl runner Tom Morris
07:41 pm Story #11147 (New): Write scalability test which can be used periodically as needed
Tom Morris
07:00 pm Feature #11146 (New): [Crunch2] [Workbench] Show slurm queue position of containers submitted to ...
h3. Background
From the user's perspective, it's hard to see what (if anything) is happening between the time a co...
Tom Clegg
06:56 pm Bug #10008 (New): [SDKs] [Python] When reading data through Collection et al., signatures should ...
Tom Clegg
06:49 pm Bug #11002: [arv-put] crash in arvfile on upload NoneType object has no attribute 'closed'
From our conversations on IRC, I think KeyboardInterrupt should skip checkpointing but shouldn't log anything. Howev... Peter Amstutz
06:21 pm Bug #11002: [arv-put] crash in arvfile on upload NoneType object has no attribute 'closed'
Updates: commit:a83e4de
Test run: https://ci.curoverse.com/job/developer-run-tests/175/
Changed so when catching ...
Lucas Di Pentima
04:07 pm Bug #11002: [arv-put] crash in arvfile on upload NoneType object has no attribute 'closed'
So if I'm following along, the problem is that it is in repack_small_blocks() which creates a new, un-owned bufferblo... Peter Amstutz
01:50 pm Bug #11002: [arv-put] crash in arvfile on upload NoneType object has no attribute 'closed'
Branch @11002-arvput-crash-fix@ at commit:e2ea7599812d12dd027930069ee92e8816ee9de8
Test run: https://ci.curoverse.co...
Lucas Di Pentima
06:44 pm Bug #11093: "arv keep_service create" should go to an editor.
You want "arv create keep_service"
or do you mean the default behavior should be more user friendly? :-)
Peter Amstutz
06:29 pm Revision 2b34839c (arvados): 10979: scancel orphaned job nodes in crunch1.
Radhika Chippada
05:41 pm Revision a83e4de2 (arvados): 11002: Do not try to save internal state when receiving a KeyboardIn...
Updated test accordingly. Lucas Di Pentima
04:33 pm Story #11123 (In Progress): Add node manager to install guide
Peter Amstutz
04:32 pm Task #11041 (In Progress): Review 11002-arvput-crash-fix
Peter Amstutz
04:29 pm Feature #11015: Improve throughput of crunch-run output-uploading stage using multi-threaded tran...
Possible implementation
In (*CollectionFileWriter)goUpload(), currently we have...
Tom Clegg
04:22 pm Revision 8ed521f7 (arvados): 11002: Merge branch 'master' into 11002-arvput-crash-fix
Lucas Di Pentima
01:00 pm Revision e2ea7599 (arvados): 11002: When trying to save the cache's state before quitting, if an ...
is caught because of a BlockManager problem induced by an interruption,
print a warning message and quit without savi...
Lucas Di Pentima
12:54 pm Revision f69196e8 (arvados): 11002: Added missing assertion to test.
Lucas Di Pentima
11:41 am Revision 3988d2c6 (arvados): 11002: Improved test mocking a more suitable method and catching the...
Lucas Di Pentima

02/20/2017

10:04 pm Story #7995: [Documentation] Document Keep Balance setup in the Install Guide
7995-keep-balance-docs @ commit:14304c7af0b0dd7bc9345b6c5aeb61a3bdc1d3b0 Tom Clegg
10:03 pm Task #11134 (Resolved): Explain limitations re shared volumes and service_type
Tom Clegg
08:50 pm Task #11134 (In Progress): Explain limitations re shared volumes and service_type
Tom Clegg
10:03 pm Revision 14304c7a (arvados): 7995: Add "dry run" note.
Tom Clegg
09:32 pm Task #11141 (Resolved): Review 6520-nodemanager-docs
Peter Amstutz
09:18 pm Revision 8b75947e (arvados): Merge branch '6520-pending-reason' refs #6520
Peter Amstutz
09:17 pm Revision 4bd54c4c (arvados): 6520: Add ReqNodeNotAvail to list of reasons (along with "Resources"...
Peter Amstutz
09:13 pm Revision 53887579 (arvados): 6520: Add information about setting up SLURM to crunchv2 documentation.
Peter Amstutz
08:49 pm Task #11140 (Resolved): Update tests
Tom Clegg
08:34 pm Task #11140 (In Progress): Update tests
Tom Clegg
08:34 pm Task #11140 (Resolved): Update tests
Tom Clegg
08:46 pm Bug #11097: [API] Reuse containers even when multiple matching containers exist with differing ou...
commit:4f0e07d462b7860bb10686c27fac16970220377f with updated test case. Tom Clegg
08:45 pm Revision 4f0e07d4 (arvados): 11097: Update test to match new behavior.
Tom Clegg
08:34 pm Revision 4cd89bd1 (arvados): 11097: Merge branch 'master' into 11097-reuse-impure
Tom Clegg
08:25 pm Revision 142a3c23 (arvados): 11002: Added test to make the bug happen.
Lucas Di Pentima
07:47 pm Story #11139: [Node manager] Expected MemTotal for each cloud node size
This can be fixed entirely through configuration by overriding the "ram" field for each size. However, we need deter... Peter Amstutz
07:07 pm Story #11139: [Node manager] Expected MemTotal for each cloud node size
GCP... Peter Amstutz
06:20 pm Story #11139 (Resolved): [Node manager] Expected MemTotal for each cloud node size
There's a discrepancy between the RAM of a VM used to choose what size node to boot for a job, and the actual amount ... Peter Amstutz
04:45 pm Task #11133 (Resolved): Set -enable-trash in keepstore docs
Tom Clegg
03:12 pm Task #11133 (In Progress): Set -enable-trash in keepstore docs
Tom Clegg
03:41 pm Revision 84bd6883 (arvados): 7995: Add keep-balance to install guide.
Tom Clegg
12:10 pm Bug #11138 (Resolved): Protection against running new style docker 1.10+ image on old docker host
I upgraded docker to 1.13.1 over the weekend (or so I thought). Since then, I've been struggling to get the system wo... Joshua Randall
09:28 am Revision fc33deae (arvados): cmd, lib
Tom Clegg

02/19/2017

12:06 pm Bug #11137 (New): apt-get upgrade arvados-api-server on Ubuntu 12.04 did not update crunch-job
My jobs were failing since I upgraded docker to 1.13.1 and installed v1.10+ versions of the docker images in keep.
...
Joshua Randall

02/17/2017

10:32 pm Revision d069de03 (arvados): Merge branch '11127-delete-trash-with-links'
refs #11127 Tom Clegg
10:30 pm Bug #11136 (New): Nginx config should speak JSON when returning its own response for unproxyable ...
Also update the install documentation to explain to customers how to do this. Tom Clegg
10:27 pm Story #7995 (In Progress): [Documentation] Document Keep Balance setup in the Install Guide
Tom Clegg
10:26 pm Task #11132 (Resolved): Add page to install guide
Tom Clegg
10:23 pm Task #11132 (In Progress): Add page to install guide
Tom Clegg
10:23 pm Task #11132 (Resolved): Add page to install guide
Tom Clegg
10:26 pm Task #11134 (Resolved): Explain limitations re shared volumes and service_type
Tom Clegg
10:26 pm Task #11133 (Resolved): Set -enable-trash in keepstore docs
Tom Clegg
10:25 pm Bug #11121 (Resolved): [Crunch] Set output collection owner_uuid to match job
Applied in changeset arvados|commit:d0ba6eaac1387ad817de1c2df2d7f4f00800aaa5. Lucas Di Pentima
09:45 pm Bug #11121: [Crunch] Set output collection owner_uuid to match job
LGTM, thanks! Tom Clegg
10:23 pm Revision ead2ecb2 (arvados): 7995: Add keep-balance to install guide.
Tom Clegg
10:20 pm Task #11125 (Resolved): Review 11121-crunch-output-collection-owner
Lucas Di Pentima
10:18 pm Revision d0ba6eaa (arvados): Merge branch '11121-crunch-output-collection-owner'
Closes #11121 Lucas Di Pentima
10:17 pm Revision b1fffbeb (arvados): 11121: Merge branch 'master' into 11121-crunch-output-collection-owner
Lucas Di Pentima
09:17 pm Revision 484370cb (arvados): 11127: Delete dependent links too when emptying trash.
Tom Clegg
08:28 pm Feature #10218: [Crunch2] Gather and record cloud/physical node information for each container
Things to consider logging:
$ uname -a
$ cat /proc/cpuinfo
$ cat /proc/meminfo
Peter Amstutz
08:15 pm Revision f29e958f (arvados): 10979: TrackContainer implementation and test
Radhika Chippada
07:49 pm Revision 85887cd7 (arvados): 11127: Do not crash server if SweepTrashedCollections thread has an ...
Tom Clegg
12:30 pm Bug #9998: Improve performance of API server when keep-balance retrieves collections
Using the arv cli, I timed the performance of collection retrieval on our system using a query very similar to what k... Joshua Randall
08:09 am Revision 016a995d (arvados): more vault
Tom Clegg

02/16/2017

11:22 pm Revision f6071ef7 (arvados): Merge branch 'master' into 10979-cancelled-job-nodes
Radhika Chippada
10:20 pm Revision 6f93ea89 (arvados): 6520: Node manager docs WIP
Peter Amstutz
09:46 pm Task #11031 (Resolved): Review 6520-nodemanager-crunchv2
Peter Amstutz
09:08 pm Task #11031 (In Progress): Review 6520-nodemanager-crunchv2
Tom Clegg
09:46 pm Task #11106 (Resolved): Review 6520-skip-compute0
Peter Amstutz
09:08 pm Task #11106 (In Progress): Review 6520-skip-compute0
Tom Clegg
09:44 pm Revision 3fa4a2b6 (arvados): Merge branch '6520-nodemanager-crunchv2' refs #6520
Peter Amstutz
09:43 pm Revision ed1acd6f (arvados): 6520: Crunch v1 recognizes 'mix' and 'drng' as busy states for
consistency (won't affect node manager behavior going forward, however.) Peter Amstutz
09:35 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Tom Clegg wrote:
> in ArvadosNodeListMonitorActor it looks like "alloc*" will be reported as "down" -- instead of "a...
Peter Amstutz
09:18 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
... Peter Amstutz
09:07 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
in ArvadosNodeListMonitorActor it looks like "alloc*" will be reported as "down" -- instead of "alloc", which is how ... Tom Clegg
09:34 pm Revision 24b1aecb (arvados): Merge branch '6520-skip-compute0' refs #6520
Peter Amstutz
09:31 pm Revision d5ddfd9d (arvados): 6520: Expand recognized 'busy' states.
Peter Amstutz
09:17 pm Task #11125 (Resolved): Review 11121-crunch-output-collection-owner
Lucas Di Pentima
09:17 pm Bug #11121: [Crunch] Set output collection owner_uuid to match job
Test added at commit:3a6966e
Test run: https://ci.curoverse.com/job/developer-run-tests/170/
Lucas Di Pentima
05:03 pm Bug #11121: [Crunch] Set output collection owner_uuid to match job
Branch @11121-crunch-output-collection-owner@ - Update at: commit:0403a0c
Running tests: https://ci.curoverse.com/jo...
Lucas Di Pentima
04:04 pm Bug #11121 (In Progress): [Crunch] Set output collection owner_uuid to match job
Lucas Di Pentima
03:50 pm Bug #11121 (Resolved): [Crunch] Set output collection owner_uuid to match job
Currently, if a job is started with owner_uuid set to a shared project, the job and its log are shared, but its outpu... Tom Clegg
09:11 pm Revision 3a6966e5 (arvados): 11121: Added test.
Lucas Di Pentima
09:07 pm Revision f6bdb550 (arvados): Merge branch 'master' into 10979-cancelled-job-nodes
Radhika Chippada
08:16 pm Task #11107 (Resolved): Review 9277-trash-container-outputs
Radhika Chippada
07:45 pm Task #11107 (In Progress): Review 9277-trash-container-outputs
Radhika Chippada
08:15 pm Story #9277 (Resolved): [Crunch2] System-owned container outputs should be garbage-collected
Applied in changeset arvados|commit:21598295f38998d8028aaa117f192de6b5758808. Peter Amstutz
08:02 pm Story #9277: [Crunch2] System-owned container outputs should be garbage-collected
LGTM Radhika Chippada
07:16 pm Story #9277 (In Progress): [Crunch2] System-owned container outputs should be garbage-collected
Peter Amstutz
06:15 pm Story #9277: [Crunch2] System-owned container outputs should be garbage-collected
Even better than @trash_at=now@, it looks like we can just set @is_trashed: true@
9277-trash-container-outputs @ c...
Peter Amstutz
08:13 pm Revision 21598295 (arvados): Merge branch '9277-trash-container-outputs' closes #9277
Peter Amstutz
08:10 pm Revision 425f9a75 (arvados): 6520: Explicitly mark node as 'down' before shutting down.
Peter Amstutz
08:04 pm Revision be33e85d (arvados): closes #11104
Merge branch '11104-work-unit-expand-on-caret' Radhika Chippada
07:08 pm Revision 73c2bd55 (arvados): 11104: add fa-caret-down class to hyperlink rather than adding a spa...
Radhika Chippada
06:37 pm Story #11123 (Resolved): Add node manager to install guide
Peter Amstutz
05:51 pm Revision 72873aff (arvados): 9277: Include trashed collections when creating copy collection for ...
request. Peter Amstutz
05:49 pm Revision dc086bed (arvados): closes #11103
Merge branch '11103-fts-crs' Radhika Chippada
05:48 pm Revision 197dd558 (arvados): 11103: ensure the index exists before attempting to drop it.
Radhika Chippada
05:28 pm Revision 2d80bb7d (arvados): 9277: crunch-run Log and output collections set is_trashed to true i...
setting trash_at. Peter Amstutz
05:27 pm Revision 1aa3f436 (arvados): 11103: Omit mounts from container_requests and definition from workf...
Radhika Chippada
05:09 pm Task #11122 (New): log partial job output to keep while job is running
Joshua Randall
05:01 pm Revision aaa53694 (arvados): Merge branch 'master' into deploy-agent
Conflicts:
build/package-build-dockerfiles/centos7/Dockerfile
build/package-build-dockerfiles/debian8/Dockerfile
b...
Tom Clegg
05:00 pm Revision 0403a0c8 (arvados): 11121: Add owner_uuid to output collection on crunch-job
Lucas Di Pentima
03:42 pm Bug #9232 (Resolved): datamanager outputs replication level distribution histogram in unsorted order
(incorporated into keep-balance) Tom Clegg
03:25 pm Story #11017 (Resolved): [API Server] Implement Docker version compatibility fallback support
Applied in changeset arvados|commit:5624fec61db977d386ce03ca333241c74ca251b5. Tom Clegg
03:17 pm Revision 5624fec6 (arvados): Merge branch '11017-docker-migration'
closes #11017 Tom Clegg

02/15/2017

10:41 pm Revision 4edf67ae (arvados): Merge branch '10797-ruby-2.3'
closes #10797 Tom Clegg
10:40 pm Revision 31d76600 (arvados): 10797: Merge branch 'master' into 10797-ruby-2.3
Tom Clegg
10:40 pm Revision 03306e21 (arvados): 10797: Ruby 2.3.3
Tom Clegg
10:35 pm Revision 6873eca0 (arvados): Update golang version to 1.7.5. Also make the equivalent changes to the
(new) ubuntu1604 build scripts.
refs #10797
Ward Vandewege
10:25 pm Revision 6b775ec4 (arvados): Merge branch 'master' into 10797-ruby-2.3
refs #10797 Ward Vandewege
09:38 pm Task #11119 (Resolved): Review 7995-keep-balance-docs
Tom Morris
09:37 pm Task #11118 (Resolved): Review 10777-die-if-arv-mount-dies
Tom Clegg
09:35 pm Task #11115 (New): Review
Peter Amstutz
09:35 pm Task #11114 (Resolved): Review 10629-fuse-listing-perf
Peter Amstutz
09:35 pm Task #11112 (Resolved): unit tests
Tom Clegg
09:35 pm Task #11111 (Resolved): Review 11097-reuse-impure
Tom Clegg
09:30 pm Task #11108 (Resolved): Review 11015-crunch-run-output-upload
Radhika Chippada
09:28 pm Bug #7853 (Closed): [Data Manager] Behave appropriately when multiple keepstore nodes share a sin...
Tom Clegg
08:33 pm Task #11107 (Resolved): Review 9277-trash-container-outputs
Peter Amstutz
08:29 pm Feature #10267 (Rejected): Permission-monitoring service keeps a flattened permissions table up t...
Tom Clegg
08:29 pm Feature #10683 (Rejected): [Crunch2] Separate service to poll sinfo and update nodes table
Node manager checks sinfo directly in #6520 Peter Amstutz
08:17 pm Task #11106 (Resolved): Review 6520-skip-compute0
Peter Amstutz
08:15 pm Revision fb60dbc9 (arvados): 6520: Don't ever assign compute0, start at slot 1 when assigning hos...
Peter Amstutz
08:08 pm Revision 314f3c34 (arvados): 6520: Check that hostname is set before setting weight.
Peter Amstutz
07:56 pm Revision 15c13298 (arvados): 6520: Apply node size fixup in CloudNodeListMonitorActor
Peter Amstutz
07:47 pm Revision 264ffa31 (arvados): 11097: Drop check for conflicting outputs when reusing completed con...
Tom Clegg
07:31 pm Revision 47476fa1 (arvados): 9849: Fix expensive "all readable collections" query.
Tom Clegg
05:57 pm Task #11035 (Resolved): Review 11017-docker-migration
Radhika Chippada
04:50 pm Task #11035 (In Progress): Review 11017-docker-migration
Radhika Chippada
05:56 pm Story #11017: [API Server] Implement Docker version compatibility fallback support
LGTM Radhika Chippada
05:28 pm Revision 0e6c1a28 (arvados): 6520: Update cloud_node.size to match server calculator configuration.
Peter Amstutz
05:14 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
Skipping compute0 ensures that even if every other node compute1-compute255 in slurm are updated to smaller actual no... Peter Amstutz
05:10 pm Feature #6520: [Node Manager] [Crunch2] Take queued containers into account when computing how ma...
It turns out you can't reconfigure hardware from "scontrol". You can only set the hardware configuration in slurm.co... Peter Amstutz
04:30 pm Revision e86743ca (arvados): 11017: Add arv-migrate-docker19 to package.
Tom Clegg
04:29 pm Revision 908959ea (arvados): 6520: Assign weight to node based on price to preferentially schedul...
cheaper nodes. Peter Amstutz
06:19 am Revision bee30106 (arvados): consul acl
Tom Clegg

02/14/2017

09:26 pm Revision 83c8f168 (arvados): 10979: trackContainers func in crunch-dispatch-slurm.go
Radhika Chippada
09:22 pm Revision f14ce113 (arvados): 6520: Add testcase using sinfo to set crunch_worker_state
Peter Amstutz
09:01 pm Revision bf9af31f (arvados): 6520: Add testcase getting wishlist from squeue.
Peter Amstutz
08:21 pm Revision 4440f049 (arvados): 6520: Call squeue to include slurm jobs in wishlist. Call sinfo to ...
crunch_worker_state in arvados node records. Peter Amstutz
08:20 pm Task #11086 (Resolved): Review 10701-refactor-dispatch
Tom Clegg
08:20 pm Task #11090 (Resolved): Add arv-migrate-docker19 program
Tom Clegg
08:19 pm Story #11017: [API Server] Implement Docker version compatibility fallback support
11017-docker-migration @ commit:f782a2505422ad9c853c4c416640c41f3b1e7e79 Tom Clegg
08:16 pm Revision f782a250 (arvados): 11017: When compute nodes use image format v2, prefer migrated docke...
Tom Clegg
08:06 pm Feature #2660 (Resolved): [Tests] [Draft] Make entire test suite run faster by eliminating excess...
Tom Clegg
08:03 pm Feature #3807 (Closed): [DRAFT] [Crunch] Tasks should be reused if marked "pure"
Tom Clegg
08:02 pm Revision 3f303422 (arvados): Merge branch 'arvbox-docker-imgfmt' no issue #
Peter Amstutz
08:02 pm Revision c90f6525 (arvados): Update test_with_arvbox script.
Peter Amstutz
08:00 pm Story #3090 (Closed): [Crunch] Pipeline template has a "parameters" section whose values can be r...
Tom Clegg
07:22 pm Feature #11100 (Resolved): [CWL] Intermediary collection handling can be specified
h1. Background
Workflows produce a lot of intermediate collections. For production workflows that are rarely re-r...
Tom Morris
07:11 pm Revision 6e8c06b8 (arvados): Set docker_image_formats to v1.
Peter Amstutz
06:30 pm Revision 347160e3 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
06:27 pm Revision e62a76da (arvados-dev): Add a new component, arvados-version-server.
No issue # Ward Vandewege
06:25 pm Revision 9bd113ee (arvados-dev): Add copyright notices to (almost) all files in this repository.
No issue # Ward Vandewege
06:19 pm Revision fd4b4bb1 (arvados-dev): Remove unused easy-docker-install.sh script.
No issue # Ward Vandewege
05:09 pm Revision 72b6a299 (arvados): Merge branch 'fix-arvbox' no issue #
Peter Amstutz
05:08 pm Revision 38bf86b7 (arvados): Fix arvbox build.
* Set up fuse groups separately.
* Separate generating superuser token from writing file so as not to write file on f...
Peter Amstutz
04:16 pm Revision 713a9030 (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
04:15 pm Revision 739ade8f (arvados-dev): Docker 1.13 doesn't support tag --force anymore. Detect that.
No issue # Ward Vandewege
04:15 pm Bug #10701 (Resolved): [Crunch2] crunch-dispatch-slurm leaks file descriptors
Applied in changeset arvados|commit:c86cbaa6f286e50900dae3203a42044449e042f7. Tom Clegg
04:00 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
LGTM. Thanks. Radhika Chippada
03:46 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
Radhika Chippada wrote:
> * can we rename Dispatcher.running as “trackers” ?
Good call -- done.
> * In running...
Tom Clegg
12:12 am Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
In dispatch.go:
* can we rename Dispatcher.running as “trackers” ?
* In runningUUIDs, this comment is confusing...
Radhika Chippada
04:08 pm Revision c86cbaa6 (arvados): Merge branch '10701-refactor-dispatch'
closes #10701 Tom Clegg
03:50 pm Revision 0030acee (arvados): 10701: Update error text in test.
Tom Clegg
03:42 pm Revision 70f47bfd (arvados): 10701: Improve comments and identifiers.
Tom Clegg
03:36 pm Revision cf91723a (arvados-dev): Merge branch 'master' of git.curoverse.com:arvados-dev
Git sync user
03:35 pm Revision 1680a689 (arvados-dev): The docker tag -f flag is now --force.
No issue # Ward Vandewege

02/13/2017

10:18 pm Revision 7c667d89 (arvados): 11017: Add arv-migrate-docker19
Tom Clegg
09:25 pm Story #9397 (Resolved): [Crunch2] Support prepopulating the output directory - CWL InitialWorkDir...
Applied in changeset arvados|commit:5763c67176e8e34656cd96881074777b14b2dc4a. Peter Amstutz
09:23 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Radhika Chippada wrote:
> Branch 9397-cwl-initialworkdir-crunchv2 @ commit:8c286d1
>
> * In test_container.py, s...
Peter Amstutz
09:24 pm Task #11018 (Resolved): Add support for InitialWorkDirRequirement to arvados-cwl-runner
Peter Amstutz
09:24 pm Task #11028 (Resolved): Update documentation
Peter Amstutz
09:24 pm Task #11051 (Resolved): Review 9397-cwl-initialworkdir-crunchv2
Peter Amstutz
09:19 pm Revision 5763c671 (arvados): Merge branch '9397-cwl-initialworkdir-crunchv2' closes #9397
Peter Amstutz
08:20 pm Revision 53b24cf4 (arvados): 9397: Improve container InitialWorkDir test.
Peter Amstutz
07:53 pm Revision 232a69ce (arvados): 9397: Fix keep URI splitting. Fix error when container has error log
collection with single file. Peter Amstutz
07:53 pm Revision 10b3fe2a (arvados): 9397: arvados-cwl-runner implementation of InitialWorkDir on contain...
Peter Amstutz
07:34 pm Revision 90982bd4 (arvados): Update schema-salad pin. refs #10993
Peter Amstutz
06:45 pm Bug #11097 (In Progress): [API] Reuse containers even when multiple matching containers exist wit...
Tom Clegg
06:45 pm Bug #11097: [API] Reuse containers even when multiple matching containers exist with differing ou...
11097-reuse-impure @ commit:264ffa31bae106bb6c36643e13186289b6cd0e18
...fails a few tests -- but perhaps only beca...
Tom Clegg
04:39 pm Bug #11097 (Resolved): [API] Reuse containers even when multiple matching containers exist with d...
h2. Background
Sometimes, running the same container twice on the same inputs can result in two successes with two...
Tom Clegg
06:40 pm Revision adf03c32 (arvados): 11097: Reuse containers even if we can detect they are impure. Fix e...
Tom Clegg
06:03 pm Revision 01779cbb (arvados): Update cwltool pin. closes #10993
Peter Amstutz
05:29 pm Revision 2a469c48 (arvados): Merge branch '9397-prepopulate-output-directory' refs #9397
Peter Amstutz
05:03 pm Task #10817 (Resolved): Review 9397-prepopulate-output-directory-pa
Peter Amstutz
05:01 pm Revision 091c92ae (arvados): Merge branch '10846-nodemanager-errors' closes #10846
Peter Amstutz
04:25 pm Feature #11095 (Resolved): [CWL] Hint to control reuse of individual workflow steps
Through the use of a CWL hint, enable the user to enable/disable job reuse for individual tools or workflow steps.
...
Peter Amstutz
06:51 am Revision 15907692 (arvados): gitignore
Tom Clegg
06:50 am Revision 5eace5e8 (arvados): consul 0.7.4, go tests
Tom Clegg

02/12/2017

05:19 pm Bug #11093 (New): "arv keep_service create" should go to an editor.
reading the docs, http://doc.arvados.org/install/install-keepstore.html at the very end:... Nico César
05:01 pm Revision 07f620f0 (arvados): gateway conf
Tom Clegg
05:01 pm Revision dde508ae (arvados): go vet
Tom Clegg

02/11/2017

07:25 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Branch 9397-cwl-initialworkdir-crunchv2 @ commit:8c286d1
* In test_container.py, should the path be “bar”, not “f...
Radhika Chippada
08:48 am Revision 816f3bfd (arvados): dev privileges, db
Tom Clegg

02/10/2017

11:23 pm Task #11072 (Resolved): Review 9397-go-manifest
Radhika Chippada
11:23 pm Task #11051 (In Progress): Review 9397-cwl-initialworkdir-crunchv2
Radhika Chippada
11:23 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Peter: branch 9397-prepopulate-output-directory-pa LGTM. Please make sure to review and make any amendments to the do... Radhika Chippada
07:43 pm Story #9397 (In Progress): [Crunch2] Support prepopulating the output directory - CWL InitialWork...
Peter Amstutz
11:10 pm Story #11017 (In Progress): [API Server] Implement Docker version compatibility fallback support
Tom Clegg
11:10 pm Task #11090 (In Progress): Add arv-migrate-docker19 program
Tom Clegg
11:10 pm Task #11090 (Resolved): Add arv-migrate-docker19 program
Tom Clegg
11:03 pm Revision e2f3f8e7 (arvados): Merge branch '11088-keep-cache-units'
closes #11088 Tom Clegg
10:15 pm Bug #11070 (Resolved): [arvados-ws] job logs are not being updated in real time
Applied in changeset arvados|commit:08c06bc813d96634b362e8c5736341fb2d874f59. Tom Clegg
10:08 pm Revision 08c06bc8 (arvados): Merge branch '11070-ws-listener-problem'
closes #11070 Tom Clegg
09:52 pm Task #10817 (In Progress): Review 9397-prepopulate-output-directory-pa
Radhika Chippada
06:54 pm Task #10817 (New): Review 9397-prepopulate-output-directory-pa
Peter Amstutz
09:42 pm Task #11086 (In Progress): Review 10701-refactor-dispatch
Radhika Chippada
07:19 pm Task #11086 (Resolved): Review 10701-refactor-dispatch
Tom Clegg
09:40 pm Task #11036 (Resolved): Review 10903-api-cancel-cascade
Radhika Chippada
09:40 pm Bug #10903 (Resolved): Cancel button on jobs & pipeline instances should cancel all child jobs too
Applied in changeset arvados|commit:d6aab18f9688d46bc1f86f021d68e02e5601cfe7. Radhika Chippada
08:20 pm Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
LGTM! Lucas Di Pentima
07:10 pm Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
Branch 10903-wb-cancel-cascade at commit:2e9cb97 addresses the suggestions from note 9 (need_transaction added).
T...
Radhika Chippada
09:39 pm Task #11044 (Resolved): Review 3900-trash-collections-delete-api
Radhika Chippada
09:36 pm Revision d6aab18f (arvados): closes #10903
Merge branch '10903-wb-cancel-cascade' Radhika Chippada
09:29 pm Revision 8c286d1b (arvados): 9397: Fix keep URI splitting. Fix error when container has error log
collection with single file. Peter Amstutz
09:21 pm Revision 5dfc48c1 (arvados): 11088: Fix keep_cache_ram units.
Tom Clegg
09:15 pm Revision 6e09079c (arvados): 11088: Fix keep_cache_ram units.
Tom Clegg
09:00 pm Revision dc82c1c1 (arvados): Merge branch '10956-arvput-pdh-fix'
Closes #10956 Lucas Di Pentima
08:31 pm Task #11073 (Resolved): Review 10903-wb-cancel-cascade
Lucas Di Pentima
07:26 pm Task #11073 (In Progress): Review 10903-wb-cancel-cascade
Lucas Di Pentima
07:44 pm Feature #6520 (In Progress): [Node Manager] [Crunch2] Take queued containers into account when co...
Peter Amstutz
07:25 pm Feature #3900 (Resolved): [Workbench] Trash button on collection uses "delete" API instead of set...
Applied in changeset arvados|commit:d3719c6ba29daec66eb1dd4003f2580f97da8e41. Lucas Di Pentima
04:24 pm Feature #3900: [Workbench] Trash button on collection uses "delete" API instead of setting expire...
* In the following, you might want to invoke "item.destroy" before updating "@removed_uuids" just for semantics
<pre...
Radhika Chippada
07:22 pm Revision f119109e (arvados): 9397: arvados-cwl-runner implementation of InitialWorkDir on contain...
Peter Amstutz
07:20 pm Bug #10945 (Resolved): [Workbench] No way to navigate from CWL job to its parent
Applied in changeset arvados|commit:35ae0e8d87dde84dd3a0ab41cef9567ab93b042a. Radhika Chippada
06:51 pm Bug #10945: [Workbench] No way to navigate from CWL job to its parent
LGTM, please merge. Lucas Di Pentima
05:19 pm Bug #10945: [Workbench] No way to navigate from CWL job to its parent
> In _show_status.html.erb Line 27: Maybe the word “the” should be eliminated from this title.
Updated
> In _sh...
Radhika Chippada
04:36 pm Bug #10945: [Workbench] No way to navigate from CWL job to its parent
Some small details:
* File @apps/workbench/app/views/jobs/_show_status.html.erb@
** Line 27: Maybe the word “the”...
Lucas Di Pentima
04:04 pm Bug #10945 (In Progress): [Workbench] No way to navigate from CWL job to its parent
Lucas Di Pentima
12:13 am Bug #10945: [Workbench] No way to navigate from CWL job to its parent
Branch 10945-used-in-section-for-cwl-jobs @ commit:78b94fd adds a "used in jobs" section to job#show page. I noticed ... Radhika Chippada
12:08 am Bug #10945: [Workbench] No way to navigate from CWL job to its parent
From IRC:
radhika
3:51 tom: just one quick question about 10945. looking at the code, the resolution for this iss...
Radhika Chippada
07:19 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
10701-refactor-dispatch @ commit:a2fe6f9367de3ee93064fbee3f2df78ce84aa318 Tom Clegg
07:18 pm Revision d3719c6b (arvados): Merge branch '3900-trash-collections-delete-api'
Closes #3900 Lucas Di Pentima
07:15 pm Revision 35ae0e8d (arvados): closes #10945
Merge branch '10945-used-in-section-for-cwl-jobs' Radhika Chippada
07:08 pm Revision 2e9cb970 (arvados): 10903: support need_transaction for job and pipeline_instance cancel...
Radhika Chippada
06:51 pm Task #11039 (Resolved): Review branch 10945-used-in-section-for-cwl-jobs
Lucas Di Pentima
04:07 pm Task #11039 (In Progress): Review branch 10945-used-in-section-for-cwl-jobs
Lucas Di Pentima
06:51 pm Revision fbc6c71e (arvados): 9397: Fix bind behavior to include Path. Update tests to check cont...
as well as output manifest. Peter Amstutz
06:40 pm Revision 397a7476 (arvados): 9397: Adjust stream normalization to use just digest (no hints) to d...
block locators. Peter Amstutz
06:31 pm Revision a2fe6f93 (arvados): 10701: Get multiple pages of queue if necessary.
Tom Clegg
06:16 pm Revision 9333c9f6 (arvados): 10701: Fix chan type.
Tom Clegg
06:11 pm Revision cff6cd57 (arvados): 10701: Add back MinRetryPeriod throttle. Update comments/identifiers.
Tom Clegg
06:04 pm Revision 6eded76b (arvados): 3900: Merge branch 'master' into 3900-trash-collections-delete-api
Lucas Di Pentima
06:01 pm Revision d6cc6f74 (arvados): 3900: Flipped item.destroy call for semantics reasons
Lucas Di Pentima
05:13 pm Revision ef9bbef0 (arvados): 10945: "Used in jobs" panel placement
Radhika Chippada
04:04 pm Revision ad464d41 (arvados): Merge branch 'master' into origin-9397-prepopulate-output-directory
Peter Amstutz
04:03 pm Revision 80689aac (arvados): Merge branch '9397-go-manifest' refs #9397
Peter Amstutz
04:02 pm Revision 4ee34d21 (arvados): 9397: Update comment & tests for CollectionFileReader to reflect it ...
lenient in the paths it accepts as a result of updates to implementation of
manifest.FileSegmentIterByName.
Peter Amstutz
02:31 pm Bug #11002 (In Progress): [arv-put] crash in arvfile on upload NoneType object has no attribute '...
Lucas Di Pentima
09:56 am Revision 6b1bf777 (arvados): 10701: cleanup
Tom Clegg
06:25 am Revision ddee3839 (arvados): 10701: Refactor dispatch library.
Tom Clegg
12:06 am Revision 78b94fd0 (arvados): 10945: show jobs in which this job is used in job#show page.
Radhika Chippada

02/09/2017

10:00 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
commit:d95e6df LGTM thanks! Tom Clegg
09:50 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Tom Clegg wrote:
> I think this would be an easy change, and it would make it possible to fix the "ignored error" ...
Peter Amstutz
08:25 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
If we have Extract() then we don't especially need NormalizedText() at all -- @m.Extract(...).Text@ would give us wha... Tom Clegg
08:19 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
ManifestTextForPath does two things at once, one of which we already have a separate function for. How about:
<pre...
Tom Clegg
06:22 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Tom Clegg wrote:
> If there's an edge case where it's expected to happen, can we check for that case explicitly be...
Peter Amstutz
09:58 pm Revision d95e6df5 (arvados): 9397: Tweak method documentation for Extract()
Peter Amstutz
09:55 pm Revision 7b4c1b70 (arvados): 9397: Update method documentation for Extract()
Peter Amstutz
09:47 pm Revision 4fa9baf6 (arvados): 10956: Fixed cwl test mocks to match behaviour change on collection ...
Lucas Di Pentima
09:46 pm Revision 4c081dd0 (arvados): 9397: Fix major bug in firstBlock(). Refactor ManifestTextForPath() to
Extract(). Test returning errors from Extract(). Peter Amstutz
09:14 pm Task #11036 (In Progress): Review 10903-api-cancel-cascade
Tom Clegg
08:39 pm Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
10903-api-cancel-cascade @ commit:3f83368
I think this needs to be protected by a transaction. Suggest new signatu...
Tom Clegg
12:59 am Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
Tom: branch 10903-wb-cancel-cascade @ commit:210af1cc7ede88914026fde078e45ef84c187a0c has both API and workbench upda... Radhika Chippada
07:30 pm Bug #10701: [Crunch2] crunch-dispatch-slurm leaks file descriptors
Problems:
* sdk/go/dispatch assumes the dispatcher's RunContainer method is done as soon as the channel closes while...
Tom Clegg
05:15 pm Bug #10701 (In Progress): [Crunch2] crunch-dispatch-slurm leaks file descriptors
Tom Clegg
07:28 pm Revision 2b297df8 (arvados): 9397: Fix collection.go to use blockdigest.LocatorPattern
Peter Amstutz
07:28 pm Revision a0195a50 (arvados): 9397: Fix collection.go to use blockdigest.LocatorPattern
Peter Amstutz
07:25 pm Revision 85c68412 (arvados): 9397: Use ManifestTextForPath
Peter Amstutz
07:11 pm Revision 1c434a9f (arvados): Merge branch '9397-go-manifest' into origin-9397-prepopulate-output-...
Peter Amstutz
07:11 pm Revision 310f7b02 (arvados): 9397: Remove conflicting code for easier merge
Peter Amstutz
06:46 pm Revision 15b49783 (arvados): 10846: Specify whether to try to resume the node when cancelling shu...
Peter Amstutz
06:25 pm Revision 5bf9312a (arvados): 9397: Use idomatic loop form.
Peter Amstutz
06:21 pm Revision 2b783402 (arvados): 9397: Move most functions into private API.
* Document ManifestTextForPath().
* Check for invalid file segments, and fix tests that were using invalid file segme...
Peter Amstutz
05:24 pm Revision 4fd89ed7 (arvados): 10701: Tidy up error handling.
Tom Clegg
05:15 pm Revision 8ee2c7c4 (arvados): 10701: Fix edge case: scheduling_parameters.partitions=[]
Tom Clegg
05:11 pm Revision d5effac0 (arvados): 11070: Fix startup race: clients can safely connect before/during pq...
Tom Clegg
09:25 am Bug #11070 (In Progress): [arvados-ws] job logs are not being updated in real time
Tom Clegg
09:24 am Bug #11070: [arvados-ws] job logs are not being updated in real time
11070-ws-listener-problem @ commit:0fc6eaead0bf7691e99d19e74ec33636909001a7
testing on 9tee4 now.
to test behav...
Tom Clegg
09:04 am Revision 0fc6eaea (arvados): 11070: Add some logs, ensure shutdown after a pq connection problem.
Tom Clegg
08:35 am Revision 8e7c3b36 (arvados): Merge branch '10700-dispatch'
refs #10700 Tom Clegg
07:28 am Revision 4c1ca6b3 (arvados): 10700: Fix error message.
Tom Clegg
12:16 am Revision 210af1cc (arvados): 10903: use cancel with cascade to cancel jobs and pause pipelines.
Radhika Chippada

02/08/2017

11:03 pm Revision bf0945d1 (arvados): 10956: When asked for the recently uploaded collection's pdh, arv-pu...
Lucas Di Pentima
10:37 pm Revision ca566236 (arvados): 10956: Get PDH from API server's response when saving a collection s...
Updated tests to reflect this change. Lucas Di Pentima
09:52 pm Bug #11070: [arvados-ws] job logs are not being updated in real time
I think I have it: when we hit a connection problem, we stopped our own event loop but
* didn't tell the pq library'...
Tom Clegg
06:30 pm Bug #11070: [arvados-ws] job logs are not being updated in real time
I ran tcpdump on the active connection, and I could see the once-per-minute "ping" between arvados-ws and postgresql.... Tom Clegg
05:55 am Bug #11070: [arvados-ws] job logs are not being updated in real time
Lots of connections from arvados-ws to postgresql sitting in CLOSE_WAIT, which doesn't seem right:... Tom Clegg
09:34 pm Revision 2565719b (arvados): 11070: Add some logs, ensure shutdown after a pq connection problem.
Tom Clegg
08:56 pm Revision c99fd72a (arvados): 11070: Add some logs, ensure shutdown after a pq connection problem.
Tom Clegg
07:33 pm Feature #3900: [Workbench] Trash button on collection uses "delete" API instead of setting expire...
Updates: commit:4dbaf8c
Test run: https://ci.curoverse.com/job/developer-run-tests/159/
Reverted changes so that ...
Lucas Di Pentima
04:30 pm Feature #3900: [Workbench] Trash button on collection uses "delete" API instead of setting expire...
Branch @3900-trash-collections-delete-api@ at commit:108467e
Test run: https://ci.curoverse.com/job/developer-run-te...
Lucas Di Pentima
07:28 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Peter Amstutz wrote:
> It can happen if @wantPos@ refers to a location past the end of the stream. @sendFileSegment...
Tom Clegg
06:44 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
On further consultation, the Python SDK does turn it into @0:0:zero@9@ so that does count as a Go SDK bug. Either @s... Peter Amstutz
06:38 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
Tom Clegg wrote:
> 9397-go-manifest @ commit:ab6a70e86dd041f3b4da167c59e3e91309f14365
>
> Functional
> * Is this...
Peter Amstutz
04:43 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
9397-go-manifest @ commit:ab6a70e86dd041f3b4da167c59e3e91309f14365
Functional
* Is this a panic (i.e., could only...
Tom Clegg
04:12 pm Story #9397: [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequirement
arvados-cwl-runner support in 9397-cwl-initialworkdir-crunchv2 Peter Amstutz
03:08 am Story #9397 (New): [Crunch2] Support prepopulating the output directory - CWL InitialWorkDirRequi...
Okay, I think this will unblock you:
# Merge 9397-go-manifest (it is a separate branch, I'm going to ask Tom for a...
Peter Amstutz
06:59 pm Revision 4dbaf8c3 (arvados): 3900: Reverted changes so that only Collection deletion from a proje...
Lucas Di Pentima
04:43 pm Bug #11075: Trying to copy a pipeline template to 9tee4 fails
9tee4 is a docker 1.13 version only cluster, see #10901
maybe this ticket will be relevant once #11017 gets resolv...
Nico César
03:33 pm Bug #11075: Trying to copy a pipeline template to 9tee4 fails
... Bryan Cosca
03:29 pm Bug #11075 (New): Trying to copy a pipeline template to 9tee4 fails
trying to copy a pipeline template to 9tee4... Bryan Cosca
04:18 pm Revision 108467e2 (arvados): 3900: When deleting items inside a project, use the delete API. In t...
Lucas Di Pentima
04:14 pm Task #11018 (In Progress): Add support for InitialWorkDirRequirement to arvados-cwl-runner
Peter Amstutz
04:11 pm Revision 8a1a96b6 (arvados): 9397: arvados-cwl-runner implementation of InitialWorkDir on contain...
Peter Amstutz
03:28 pm Task #11072 (In Progress): Review 9397-go-manifest
Tom Clegg
12:14 am Task #11072 (Resolved): Review 9397-go-manifest
Peter Amstutz
02:00 am Task #11073 (Resolved): Review 10903-wb-cancel-cascade
Radhika Chippada
02:00 am Bug #10903: Cancel button on jobs & pipeline instances should cancel all child jobs too
Branch 10903-api-cancel-cascade @ commit:3f833687 addresses the API server updates: (1) enhance job.cancel method to ... Radhika Chippada
12:14 am Revision ab6a70e8 (arvados): 9397: Add manifest normalization and sub-manifest extraction by path.
Introduces "SegmentedManifest" which stores streams -> files -> file segments.
Enables reexport of manifest in normal...
Peter Amstutz

02/07/2017

11:47 pm Revision b3fb8f09 (arvados): 9397: Make FileSegmentIterByName channel buffered so it isn't necess...
goroutine context switch on every segment. Peter Amstutz
10:51 pm Revision 3f833687 (arvados): 10903: Add cancel method with cascade to pipeline_instance and updat...
Radhika Chippada
10:47 pm Revision da3ed065 (arvados): 9397: Use binary search to find the first block. Store BlockOffsets...
Peter Amstutz
10:46 pm Bug #11071 (Resolved): [Workbench] qr1hi performance tests are broken
Oldest 'OK' run: https://ci.curoverse.com/job/qr1hi-automated-performance-suite/3162/
Since then (ie Dec. 26, 2016...
Ward Vandewege
09:59 pm Bug #11070 (Resolved): [arvados-ws] job logs are not being updated in real time
In workbench, job logs are updated in real time via websockets. On our test installations with arvados-ws, that is no... Ward Vandewege
09:44 pm Revision 559f3e22 (arvados): 9397: Improve manifest normalization efficiency
Peter Amstutz
07:50 pm Bug #9154 (Closed): data manager sometimes refuses to continue when collections are being written...
Tom Clegg
07:30 pm Feature #11064 (Duplicate): [Crunch2] crunch-run expires output collection immediately
Peter Amstutz
06:47 pm Revision e79722b0 (arvados): 9397: Syntax/logic fixes
Peter Amstutz
05:51 pm Bug #11068 (Resolved): [Arvados-CWL-runner] need better error message when there are issues getti...

The 9tee4 diagnostics job (e.g. 9tee4-xvhdp-yuv7hvb80fbb3lk) failed with this message:...
Ward Vandewege
02:26 pm Bug #10903 (In Progress): Cancel button on jobs & pipeline instances should cancel all child jobs...
Radhika Chippada
06:40 am Revision 61476528 (arvados): postgresql
Tom Clegg
06:19 am Revision 668b2ae2 (arvados): new webpack
Tom Clegg
 

Also available in: Atom