Arvados

Story

Resolved

Closed

Rejected

Sprint Impediments

3381 3381 Peter Amstutz Peter Amstutz 47 Workbench 56 Workbench [Workbench] "jobs" view in Workbench shows failed job job but green progress bar Resolved Resolved 3 1.0 1.0 1.0 3381 36 47 76 1 1.0	4112 4112 Update pipeline/job page display of job status Peter Amstutz 47 0 3381 3 36 -c-a 5 3950 3950 Review branch: 3381-job-progress-bar-bug Peter Amstutz 47 0 3381 3 36 -c-a 5
4042 4042 Peter Amstutz Peter Amstutz 47 Crunch 52 Crunch [Crunch] run-command script supports running MxN tasks (e.g., M directories x N chromosomes) Resolved Resolved 3 1.0 1.0 1.0 4042 36 47 76 2 1.0	4103 4103 Review 4042-run-command-MxN Peter Amstutz 47 0 4042 3 36 -c-a 5
2881 2881 Brett Smith Brett Smith 79 [OPS] Basic node manager that can start/stop compute nodes based on demand Resolved Resolved 3 5.0 5.0 5.0 2881 36 79 76 2 5.0	3239 3239 fix update_node_attributes script and/or merge its remaining functionality into crunch-dispatch this script updates the node objects with information about what is running on them, which us information used by node manager Ward Vandewege 1 0 2881 3 36 -c-a 5 3944 3944 Review 2881-node-has-job-wip This branch implements the work in task #3239, with amendments discussed on IRC. Nodes' job assignments are stored on the API server with a Rails association. crunch-dispatch updates this information when it makes assignments. Tom Clegg 3 0 2881 3 36 -c-a 5 3248 3248 implement basic functionality Ward Vandewege 1 0 2881 3 36 -c-a 5 4106 4106 Review 2881-node-manager-wip Brett Smith 79 0 2881 3 36 -c-a 5	3245 3245 deploy node manager Ward Vandewege 1 0 2881 5 36 -c-a 5
4085 4085 [Workbench] On pipeline instance page, queued jobs should have Cancel buttons Resolved Resolved 3 0.5 0.5 0.5 4085 36 76 1 0.5
3825 3825 Tim Pierce Tim Pierce 44 Crunch 52 Crunch [Crunch] crunch-job should save (very) large log files without filling up tmp space Resolved Resolved 3 1.0 1.0 1.0 3825 36 44 76 1 1.0	3913 3913 Test crunch-job log output Tim Pierce 44 0 3825 3 36 -c-a 5 3912 3912 Rewrite crunch_job::Log to pipe output to arv-put Tim Pierce 44 0 3825 3 36 -c-a 5 4078 4078 Review 3825-crunch-pipe-to-arv-put-final Review at commit:8f6d2653d Tim Pierce 44 0 3825 3 36 -c-a 5
3902 3902 Radhika Chippada Radhika Chippada 72 [Support] Support: fix bugs and write tests second half Resolved Resolved 3 5.0 5.0 5.0 3902 36 72 76 3 5.0	4115 4115 Review 3782-stub-io-pipe commit:c76954b Tom Clegg 3 0 3902 3 36 -c-a 5 4063 4063 Clean up 3782 test by stubbing "read from arv-get pipe" instead of using condemned KEEP_LOCAL_STORE Stub should send the arranged content, say, 4 bytes at a time. Tom Clegg 3 0 3902 3 36 -c-a 5
3261 3261 Ward Vandewege Ward Vandewege 1 Keep 53 Keep [Keep] arv-put fails if permission system is enabled Resolved Resolved 3 1.0 1.0 1.0 3261 36 1 76 1 1.0	3274 3274 Confirm bug still exists by reproducing somewhere Ward Vandewege 1 0 3261 3 36 -c-a 5 3429 3429 Review 3261-keep-docker-config at commit:5ce0e0fe Ward Vandewege 1 0 3261 3 36 -c-a 5 3626 3626 Update docker images, esp. base and java-bwa-samtools Ward Vandewege 1 0 3261 3 36 -c-a 5 3707 3707 Ensure crunch-job task output collation can survive with all permission features enabled 0 3261 3 36 -c-a 5
3894 3894 Tom Clegg Tom Clegg 3 Documentation 58 Documentation [Documentation] Document the run-tests.sh program on the Hacking pages on the wiki Resolved Resolved 3 1.0 1.0 1.0 3894 36 3 76 2 1.0	3930 3930 Fix bugs in run-tests script and make it a bit less hostile Tom Clegg 3 0 3894 3 36 -c-a 5 3936 3936 Review 3894-gem-version Use last-time-gem-source-dir-changed in gem version string, not last-time-anything-in-arvados-changed Tom Clegg 3 0 3894 3 36 -c-a 5 3933 3933 Fix pkg_resources.VersionConflict in fuse tests <pre> Installed /home/tom/src/arvados/services/fuse/llfuse-0.40-py2.7-linux-x86_64.egg Traceback (most recent call last): File "setup.py", line 29, in <module> zip_safe=False) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "build/bdist.linux-i686/egg/setuptools/command/test.py", line 111, in run File "build/bdist.linux-i686/egg/setuptools/dist.py", line 284, in fetch_build_eggs File "build/bdist.linux-i686/egg/pkg_resources.py", line 569, in resolve pkg_resources.VersionConflict: (arvados-python-client 0.1.1411069908.8ba7f94 (/tmp/tmp.rnjhBE5vpq/lib/python2.7/site-packages), Requirement.parse('arvados-python-client>=0.1.1411070090')) </pre> Tom Clegg 3 0 3894 3 36 -c-a 5 3924 3924 Write "Running tests" wiki page Tom Clegg 3 0 3894 3 36 -c-a 5 3946 3946 Review 3894-improve-run-tests on arvados-dev at commit:07b0b88 Tom Clegg 3 0 3894 3 36 -c-a 5 3970 3970 Update "making ruby gems" wiki Tom Clegg 3 0 3894 3 36 -c-a 5 3945 3945 Review 3894-gem-version (take 2) Use --first-parent to cope with commit order != merge order Tom Clegg 3 0 3894 3 36 -c-a 5
3990 3990 Radhika Chippada Radhika Chippada 72 [Workbench] "Re-run pipeline" buttons should put the new pipeline-instance in the same project as the original Resolved Resolved 3 0.5 0.5 0.5 3990 36 72 76 1 0.5	4125 4125 Review branch: 3990-owner-when-rerunning-pipeline Radhika Chippada 72 0 3990 3 36 -c-a 5
3882 3882 Radhika Chippada Radhika Chippada 72 [Workbench] fiddlesticks trying to cancel job that is already cancelled Resolved Resolved 3 0.5 0.5 0.5 3882 36 72 76 1 0.5	4104 4104 Add test(s) to verify a cancelled job stays cancelled on further cancel actions. Radhika Chippada 72 0 3882 3 36 -c-a 5 4105 4105 Review branch: 3882-cancel-already-cancelled-job Radhika Chippada 72 0 3882 3 36 -c-a 5
3775 3775 Tom Clegg Tom Clegg 3 Crunch 52 Crunch [Crunch] arv-crunch-job should work in local mode with an arvados-hosted repository Resolved Resolved 3 1.0 1.0 1.0 3775 36 3 76 2 1.0	4059 4059 Accept arbitrary git url as repository if --git-dir not given (If --git-dir given, do not do any git stuff anywhere else.) Tom Clegg 3 0 3775 3 36 -c-a 5 4060 4060 Look up fetch-url of repo via arvados API if bare repo name given in repository attr (if no --git-dir given and repository is not a path) Tom Clegg 3 0 3775 3 36 -c-a 5 4082 4082 Review 3775-fetch-git-repo at commit:5c1f03e Tom Clegg 3 0 3775 3 36 -c-a 5
3828 3828 Tom Clegg Tom Clegg 3 Tests 59 Tests [Keepproxy] insufficient error checking on "defer listener.Close()" lines in the tests, cf. jenkins run 846 where the keepproxy tests failed like that Resolved Resolved 3 0.5 0.5 0.5 3828 36 3 76 1 0.5	4119 4119 Review 3828-keepproxy-race Tim Pierce 44 0 3828 3 36 -c-a 5
4012 4012 Brett Smith Brett Smith 79 [Crunch] crunch-job bandaid: use eval/retry to wrap api calls that are likely to fail a long-running job due to a transient error condition Resolved Resolved 3 1.0 1.0 1.0 4012 36 79 76 1 1.0	4109 4109 Review 4012-crunch-job-api-retries-wip Peter Amstutz 47 0 4012 3 36 -c-a 5
4046 4046 Radhika Chippada Radhika Chippada 72 [API] All jobs claim to be "next in queue". Bug in queue_position calculation in services/api/app/models/job.rb. It needs to increment index. Resolved Resolved 3 0.5 0.5 0.5 4046 36 72 76 1 0.5	4117 4117 Review branch: 4046-job-queue-position Radhika Chippada 72 0 4046 3 36 -c-a 5
3634 3634 Phil Hodgson Phil Hodgson 80 [Workbench] Page content is the same as where you left it when navigating (back) to it with browser history Resolved Resolved 3 1.0 1.0 1.0 3634 36 80 76 1 1.0	3881 3881 Review 3634-tab-state Phil Hodgson 80 0 3634 3 36 -c-a 5 4080 4080 Deliver all pages with an expired cache To force a reload on browser-back. Phil Hodgson 80 0 3634 3 36 -c-a 5 3880 3880 Remember active tab panel on browser back navigation Phil Hodgson 80 0 3634 3 36 -c-a 5
4025 4025 Radhika Chippada Radhika Chippada 72 [Workbench] Make it clear that "Move to project" button in the upper right of the project page moves the current project, not the current selection. Resolved Resolved 3 4025 36 72 76 1	4079 4079 Review branch: 4025-move-project-button-label Radhika Chippada 72 0 4025 3 36 -c-a 5
4036 4036 Radhika Chippada Radhika Chippada 72 [Workbench] When combining collections in a project, the new collection is created in Home project, but the user is still in current project. Resolved Resolved 3 4036 36 72 76 1	4066 4066 Review branch: 4036-combine-collections-in-project-issue Peter Amstutz 47 0 4036 3 36 -c-a 5
4016 4016 [Workbench] "Make sharing link" missing from collection pages. Resolved Resolved 3 4016 36 76 1
4045 4045 [Workbench] Provenance graph display regressed (possibly with #3036) and does not connect jobs/components via input/output pdh Rejected Rejected 6 0.5 0.5 0.5 4045 36 76 1 0.5
3769 3769 Peter Amstutz Peter Amstutz 47 Crunch 52 Crunch [API] In crunch-dispatch, throttle by bytes_per_minute or _node_minute Resolved Resolved 3 1.0 1.0 1.0 3769 36 47 76 1 1.0	4061 4061 Review 3769-throttle-logs Ward Vandewege 1 0 3769 3 36 -c-a 5 3909 3909 Add throttle logic Peter Amstutz 47 0 3769 3 36 -c-a 5
3782 3782 Tim Pierce Tim Pierce 44 Workbench 56 Workbench [Workbench] Browser prompts about long-running JavaScript when trying to view a large Job log in viewer Resolved Resolved 3 0.5 0.5 0.5 3782 36 44 76 1 0.5	4034 4034 Review 3782-stub-file-content commit:5a341ad Tom Clegg 3 0 3782 3 36 -c-a 5 4017 4017 Use ?size= URL parameter to limit job log data Tim Pierce 44 0 3782 3 36 -c-a 5 3975 3975 Review 3782-large-job-logs Tom Clegg 3 0 3782 3 35 -c-a 5
3052 3052 Peter Amstutz Peter Amstutz 47 [Crunch] stdout from non-task programs invoked by crunch-job should propagate to job log. Resolved Resolved 3 0.5 0.5 0.5 3052 36 47 76 1 0.5	4039 4039 Review 3052-crunch-log-stdout Tim Pierce 44 0 3052 3 36 -c-a 5
4000 4000 Tim Pierce Tim Pierce 44 API 57 API [API] "re-run with latest" yields fiddlesticks if pipeline template has been updated with new components Resolved Resolved 3 1.0 1.0 1.0 4000 36 44 76 1 1.0	4047 4047 Review 4000-rerun-pipeline-changed-template Review at commit:5233f1d Tim Pierce 44 0 4000 3 36 -c-a 5 4021 4021 make PipelineInstance.copy handle exceptions gracefully Tim Pierce 44 0 4000 3 36 -c-a 5
4004 4004 Peter Amstutz Peter Amstutz 47 Restore "All jobs/pipelines/collections" buttons to dashboard Resolved Resolved 3 0.5 0.5 0.5 4004 36 47 76 1 0.5	4005 4005 Review 4004-restore-dashboard-buttons Peter Amstutz 47 0 4004 3 36 -c-a 5
4038 4038 Tim Pierce Tim Pierce 44 merges to master may use "fixes #" in addition to "closes #" Resolved Resolved 3 0.5 0.5 0.5 4038 36 44 76 2 0.5
3961 3961 Tim Pierce Tim Pierce 44 API 57 API [API] 9tee4: cannot create subproject under Home Resolved Resolved 3 0.5 0.5 0.5 3961 36 44 76 1 0.5	4023 4023 Review 3961-uuid-assignment-not-permitted Radhika Chippada 72 0 3961 3 36 -c-a 5 4020 4020 make ensure_unique_name to work without admin privileges Tim Pierce 44 0 3961 3 36 -c-a 5
3866 3866 Brett Smith Brett Smith 79 SDKs 54 SDKs [SDK] Python KeepClient needs to construct URLs smarter Resolved Resolved 3 0.5 0.5 0.5 3866 36 79 76 1 0.5
3848 3848 Brett Smith Brett Smith 79 SDKs 54 SDKs [SDK] Python SDK should always set proper version when built from git Resolved Resolved 3 0.5 0.5 0.5 3848 36 79 76 1 0.5	3952 3952 Review 3848-egg-default-git-tag-wip Brett Smith 79 0 3848 3 36 -c-a 5 4110 4110 Review arvados-dev branch 3848-build-packages-pyversion The Jenkins build-packages script is overriding the version information in our Python packages. Now that setup.py generates build tags the way we want, this branch removes the override, treating our Python packages more like our Ruby gems. Ward Vandewege 1 0 3848 3 36 -c-a 5
3889 3889 Radhika Chippada Radhika Chippada 72 API 57 API [API] [Workbench] Refactor any functional tests that call multiple actions into either integration tests or independent functional tests. Resolved Resolved 3 1.0 1.0 1.0 3889 36 72 76 1 1.0	3940 3940 Raise exception when multiple actions are performed on the controller in functional testing Radhika Chippada 72 0 3889 3 36 -c-a 5 3941 3941 Identify API functional tests that are performing multiple actions on a single controller. Either fix them or convert them into integration tests. Radhika Chippada 72 0 3889 3 36 -c-a 5 3942 3942 Identify Workbench functional tests that are performing multiple actions on a single controller. Either fix them or convert them into integration tests. Radhika Chippada 72 0 3889 3 36 -c-a 5 3943 3943 Review branch: 3889-functional-testing (raising exception on controller reuse only) This is review task for work towards raising an exception when multiple actions on performed on a single controller in a functional test. Radhika Chippada 72 0 3889 3 36 -c-a 5 4001 4001 Review branch: 3889-functional-testing All tests are fixed and branch is ready for a review. Tom Clegg 3 0 3889 3 36 -c-a 5
3616 3616 Peter Amstutz Peter Amstutz 47 Workbench 56 Workbench [Workbench] Workbench uses huge amounts of RAM when user downloads a large (> 1 GiB) file. Resolved Resolved 3 1.0 1.0 1.0 3616 36 47 76 1 1.0	4022 4022 Use rails "live" feature Tom Clegg 3 0 3616 3 36 -c-a 5 4014 4014 Review 3616-live-stream Quick fix for memory leak. Peter Amstutz 47 0 3616 3 36 -c-a 5 3919 3919 Debug streaming Peter Amstutz 47 0 3616 3 36 -c-a 5
3871 3871 Peter Amstutz Peter Amstutz 47 Keep 53 Keep [FUSE] Crashes on non-ASCII folder names Resolved Resolved 3 1.0 1.0 1.0 3871 36 47 76 1 1.0	3904 3904 Encode/decode filenames as utf-8 Peter Amstutz 47 0 3871 3 36 -c-a 5 4007 4007 Review 3871-non-ascii-project-names Peter Amstutz 47 0 3871 3 36 -c-a 5
3038 3038 Radhika Chippada Radhika Chippada 72 Clean up recommended/default port numbers for API server and Workbench Resolved Resolved 3 1.0 1.0 1.0 3038 36 72 76 1 1.0	3958 3958 Update all the port numbers as specified Radhika Chippada 72 0 3038 3 36 -c-a 5 3959 3959 Review branch: 3038-default-port-numbers Ready for review Radhika Chippada 72 0 3038 3 36 -c-a 5 3995 3995 Review 3991-docker-workbench-address Ward Vandewege 1 0 3038 3 36 -c-a 5 3991 3991 Update docker scripts to set workbench_address, reset it to nil in default config Tom Clegg 3 0 3038 3 36 -c-a 5
3899 3899 Peter Amstutz Peter Amstutz 47 [Crunch] crunch-job and Workbench use the new job state attribute (see #3898) instead of paying attention to (and updating) running and success flags. Resolved Resolved 3 2.0 2.0 2.0 3899 36 47 76 2 2.0	3962 3962 Update crunch-dispatch Peter Amstutz 47 0 3899 3 36 -c-a 5 3963 3963 Update arv-run-pipeline-instance Peter Amstutz 47 0 3899 3 36 -c-a 5 3982 3982 Review 3988-crunch-use-job-state Tom Clegg 3 0 3899 3 36 -c-a 5 3921 3921 Update crunch-job Peter Amstutz 47 0 3899 3 36 -c-a 5
3168 3168 Peter Amstutz Peter Amstutz 47 Crunch 52 Crunch [Crunch] Source git repository for a job should not be fetched into internal arvados repository over and over again even when requested node profile is not available Resolved Resolved 3 0.5 0.5 0.5 3168 36 47 76 1 0.5	3986 3986 Review 3168-crunch-git-fetch Peter Amstutz 47 0 3168 3 36 -c-a 5
3949 3949 SDKs 54 SDKs [SDK] arv-get consumes excessive memory on very large collections Resolved Resolved 3 0.5 0.5 0.5 3949 36 76 1 0.5
3413 3413 Tim Pierce Tim Pierce 44 Keep 53 Keep [Keep] Keep Server accepts list of trashable blocks Resolved Resolved 3 1.0 1.0 1.0 3413 36 44 76 6 1.0	3679 3679 Implement TrashHandler A "PUT /trash" HTTP request should be routed to a TrashHandler: * The TrashHandler should make sure the request comes from Data Manager, and return 401 Unauthorized if not. * If the request is valid, it should parse the request body as a JSON message and make sure it has a valid structure. If the JSON unmarshalling fails, TrashHandler should return 400 Bad Request. * Otherwise, if the request is syntactically valid and comes from Data Manager, TrashHandler should return 200 OK and send the parsed list to the TrashCollector, which runs asynchronously in a goroutine. Tim Pierce 44 0 3413 3 36 -c-a 5
3890 3890 [API] dots (and probably other valid characters) are being stripped from git repository names. The name field in the output of arv repository get_all_permissions is wrong (it strips characters that are valid) Resolved Resolved 3 0.5 0.5 0.5 3890 36 76 1 0.5
3187 3187 Peter Amstutz Peter Amstutz 47 Workbench 56 Workbench [Workbench] Better information and features on "show pipeline instance" page Resolved Resolved 3 3.0 3.0 3.0 3187 36 47 76 6 3.0	3850 3850 Add started_at and finished_at to tasks and pipelines Peter Amstutz 47 0 3187 3 36 -c-a 5 3862 3862 Review 3187-pipeline-instance-page Peter Amstutz 47 0 3187 3 36 -c-a 5 3888 3888 Fix tests Peter Amstutz 47 0 3187 3 36 -c-a 5 3766 3766 Send screen capture to Tom O'Keefe to get design/layout advice Peter Amstutz 47 0 3187 3 36 -c-a 5 3487 3487 Show all component and job details, and overall timing stats Peter Amstutz 47 0 3187 3 36 -c-a 5 3849 3849 Review 3187-start-finish-timestamps-tasks-pipelines Ward Vandewege 1 0 3187 3 36 -c-a 5
3893 3893 Brett Smith Brett Smith 79 [Documentation] Explain in public wiki how to submit good/safe support requests and bug reports without accidentally leaking private information Resolved Resolved 3 0.5 0.5 0.5 3893 36 79 76 3 0.5	3925 3925 Review wiki page https://arvados.org/projects/arvados/wiki/How_to_Report_Issues_in_Arvados Tom Clegg 3 0 3893 3 36 -c-a 5
3898 3898 Radhika Chippada Radhika Chippada 72 [API] Job model has a single state attribute that can be updated and read instead of paying attention to running and success flags. Resolved Resolved 3 0.5 0.5 0.5 3898 36 72 76 6 0.5	3906 3906 Add state column Radhika Chippada 72 0 3898 3 36 -c-a 5 3939 3939 review 3898-job-state-attr-fixes Ward Vandewege 1 0 3898 3 36 -c-a 5 3934 3934 Review branch: 3898-job-state-attr Branch ready for review Peter Amstutz 47 0 3898 3 36 -c-a 5 3951 3951 review 3898-job-state-attr-TC At commit:d70e5d0 0 3898 3 36 -c-a 5
3661 3661 Phil Hodgson Phil Hodgson 80 Workbench 56 Workbench [Workbench] Add "Move" and "Copy" buttons to top of #show page for every kind of object that goes_in_folders Resolved Resolved 3 0.5 0.5 0.5 3661 36 80 76 6 0.5	3845 3845 Add the buttons and make them work Phil Hodgson 80 0 3661 3 36 -c-a 5 3851 3851 Add one or two new integration tests for the new buttons Phil Hodgson 80 0 3661 3 36 -c-a 5 3852 3852 Review 3661-copy-move-from-show Phil Hodgson 80 0 3661 3 36 -c-a 5
3147 3147 Brett Smith Brett Smith 79 SDKs 54 SDKs [SDKs] Python clients should automatically retry failed API and Keep requests (including timeouts), in order to survive temporary outages like server restarts and network blips. Resolved Resolved 3 1.0 1.0 1.0 3147 36 79 76 1 1.0	3735 3735 Make crunch-job retry when it calls arv-put Brett Smith 79 0 3147 3 36 -c-a 5 3802 3802 arv-mount exposes retry support Brett Smith 79 0 3147 3 36 -c-a 5 3658 3658 PySDK Collection classes expose retry support Brett Smith 79 0 3147 3 36 -c-a 5 3659 3659 arv-put exposes retry support Probably through a simple @--retries@ switch. Brett Smith 79 0 3147 3 36 -c-a 5 3657 3657 Review 3147-pysdk-retries-wip Tim Pierce 44 0 3147 3 36 -c-a 5 3872 3872 Review 3147-py-collection-retries-wip2 Please see "review notes on the story":https://arvados.org/issues/3147#note-25. Tim Pierce 44 0 3147 3 36 -c-a 5
3338 3338 Phil Hodgson Phil Hodgson 80 Workbench 56 Workbench [Workbench] Show number of items in each tab of the project page. Resolved Resolved 3 0.5 0.5 0.5 3338 36 80 76 2 0.5	3854 3854 Implement simple, generic way to get #count for each pane Phil Hodgson 80 0 3338 3 36 -c-a 5 3860 3860 Review 3338-tab-counts Phil Hodgson 80 0 3338 3 36 -c-a 5
3927 3927 Tim Pierce Tim Pierce 44 API 57 API [API] Db migration for User.owner_uuid NOT NULL constraint Resolved Resolved 3 0.5 0.5 0.5 3927 36 44 76 1 0.5
3384 3384 Tim Pierce Tim Pierce 44 Crunch 52 Crunch [Crunch] Termination of jobs due to 'Connection timed out'? Closed Closed 5 3384 36 44 76 1