Roadmap

2019-03-27 Sprint

Due in 8 days (03/27/2019)

16%

59 issues   (8 closed — 51 open)

2019-03-27 Sprint

Arvados Contributor Wiki

This wiki has information related to the open source project. It's a work in progress, so come back often for updates and feel free to become a contributor.

Platform Overview

Using Arvados

Community

Contributing and hacking

Key Components

Related issues
Feature #14322: [CWL] Accept collection uuid in input
Feature #14573: [Spike] [API] Fully functional filename search
Feature #14947: [Tests] run-tests.sh --interactive
Task #14289: Review
Task #14663: Review 14645-fuse-operations-reporting
Task #14747: Review
Task #14838: Review
Task #14868: Review 14807-dispatch-cloud-fixes
Task #14893: Review 14322-cwl-uuid-input
Task #14897: Review
Task #14898: Review
Task #14935: Review 14670-new-java-sdk-docs
Task #14940: Review 14939-crunchstat-summary-python3 -> 14939-crunchstat-summary-python3
Task #14948: Update install docs
Task #14949: Ops deploy multi-API server configuration on test cluster
Task #14950: Add to run-tests
Task #14953: Review
Task #14954: Review
Task #14955: Review
Task #14956: Review 14451-crunchstat-summary-cache-ram
Task #14957: Review
Task #14958: Review 14451-crunchstat-summary-cache-ram
Task #14959: Review 14947-run-tests-interactive
Task #14960: Review
Task #14961: Review 14451-crunchstat-summary-cache-ram
Task #14962: Review 14807-escalate-sigterm
Task #14963: Review
Task #14970: Review 14966-api-test-hang-fix
Task #14978: Review
Task #14981: Review
Task #14982: Review 14977-kill-if-not-in-queue
Story #13484: [API] Support multiple load-balanced API server nodes
Story #13996: [API] Migrate RailsAPI to new cluster config file
Story #14287: [Controller] Refactor API code / routing
Story #14484: [API Server] Return collection size and number of files in collection record
Story #14606: [CWL] CWL v1.1 spec work and support
Story #14645: [arv-mount] Report FUSE operation latency
Story #14669: [Java] Merge into main Arvados repo
Story #14670: [Java] Write / merge Java SDK documentation
Story #14796: [crunch-dispatch-cloud] Document installation / migration from c-d-slurm + node manager
Story #14807: [crunch-dispatch-cloud] Features/fixes needed before first production deploy
Story #14835: Finish cluster config key design
Story #14873: [API] Update to Rails 5
Story #14939: Update crunchstat-summary to Python 3
Story #14946: Update to Ruby 2.4 - 2.3 is going EOL
Story #14988: [Workbench] Upgrade to Rails 5
Story #14989: [SSO] Upgrade to Rails 5

Arvados Future Sprints

Due in about 10 months (01/02/2020)

A pseudo-version to collect tickets from multiple projects

5%

862 issues   (39 closed — 823 open)

Future sprints

Arvados Contributor Wiki

This wiki has information related to the open source project. It's a work in progress, so come back often for updates and feel free to become a contributor.

Platform Overview

Release Notes

Key Components

Using Arvados

Contributing and hacking

Community

Legal

Related issues
Feature #3101: [Tests] Websockets should work out of the box in test/development modes, even when using an ssh tunnel and a self-signed certificate.
Feature #3125: [Workbench] Add a "docker images" page to Workbench that lists available docker images
Feature #3223: [Documentation] Tutorial for adding a crunch script wrapper for an existing tool that is already hosted on github.
Feature #3226: [Workbench] Project chooser dialog should have a "create new project" button.
Feature #3284: [Workbench] Project show page should have buttons to add objects besides Collections
Feature #3311: [SDKs] Improve metadata created by arv-put
Feature #3445: [Workbench] Improve chooser preview for collections, and add previews for objects that don't have any yet (pipeline instances, templates, jobs, projects)
Feature #3734: [Keep] Keepstore and keepproxy support HTTP Range requests
Feature #3818: [API] Discovery document schema should include whether field on resource is writable or read-only.
Feature #3892: [SDKs] Install man pages for CLI tools (auto generate from the --help content)
Feature #3928: [Workbench] Reveal log collection ID in pipeline/job summary and job Log tab
Feature #3964: [SDKs] Command-line diagnostic suite reveals problems on a shell VM (e.g., arv-mount, SDK version skew)
Feature #3980: [SDKs] CLI tool to view real time logs from running jobs/pipelines, much like the Log tab of the Workbench PipelineInstance#show page
Feature #4193: [SDK] Make arv keep docker smarter about storing docker layers, thus making uploads of changed images (much) faster
Feature #4223: [SDKs] arv keep docker --download to get image from Arvados
Feature #4302: [Crunch] Pipelines should not fail immediately after one job failure, but continue running as much as possible
Feature #4343: [SDKs] arv create needs --project-uuid and --name parameters for appropriate classes
Feature #4349: [SDKs] When editing a template in arv-edit and faced with a parse error, when trying again, it should automatically take you to the line/column of the error
Feature #4393: [Workbench] Display portable data hash, parent project, file count in Collection tables
Feature #4404: [API] [SDKs] Include provenance information for uploaded Collections
Feature #4446: [Workbench] Provide feedback on dashboard to indicate that NodeManager is booting a node.
Feature #4456: [Workbench] Provide more feedback about when a queued job is likely to start.
Feature #4502: [SDKs] "arv recent [objectType]" lists recently modified objects
Feature #4503: [SDKs] Option to watch logs (instead of exiting) when queuing a job or pipeline instance using arv job create, arv pipeline run, etc.
Feature #4508: [Documentation] Best practices for getting the most out of job re-use
Feature #4516: [Workbench] Don't show empty projects/collections when using the chooser dialog to select pipelines, files, etc.
Feature #4530: [Workbench] Show more (earlier) log entries for a running job
Feature #4531: [Documentation] Set of introductory tutorials each focused on accomplishing a simple task
Feature #4539: [Workbench] Job detail page should show the job status when the job is not running or has completed running (button that shows 'queued', 'not ready', 'failed', 'complete', etc)
Feature #4561: [SDKs] Refactor run-command so it can be used as an SDK by scripts in a git tree
Feature #4579: [Documentation] Run-command docs should remind user how & why to exit non-zero on failure.
Feature #4642: [Workbench] "Download" / "Save as" button for provenance graph
Feature #4647: [Workbench] Projects dropdown should have a free text filter box
Feature #4650: [API] API method and CLI shortcut for refreshing the signatures on some block locators (without creating a collection)
Feature #4808: [Workbench] "Add data from another project" modal supports regex filtering
Feature #4823: [SDKs] Good Collection API for Python SDK
Feature #4835: [Workbench] Log Graph - Feature requests
Feature #4906: [Documentation/FUSE] Document '.arvados#collection' in shell VM keep mount
Feature #4935: [Crunch] Choose worker nodes where the desired docker image is already available.
Feature #5033: [SDKs] `arv keep docker` should pipe the data directly to Keep instead of store-and-forward, and provide some progress feedback up front in any case.
Feature #5035: [Documentation] Document the minimum docker image / Dockerfile that can be used to run a job.
Feature #5036: [arv-mount] Change default mount type from "fuse" to "fuse.arvados"
Feature #5064: [Crunch] Automatically restart jobs after internal/intermittent errors
Feature #5097: [SDKs] CLI tools support project paths in addition to uuids
Feature #5133: [Crunch] Use uncommitted code in a git work tree to run a dockerized crunch job in a VM (using crunch-job or arv-run-pipeline-instance --run-jobs-here)
Feature #5138: [SDKs] arv-run-pipeline-instance should accept project and title parameters
Feature #5139: [SDKs] arv-get should support getting multiple files by shell glob
Feature #5193: [Workbench] Public project page should encourage visitors to try using Arvados themselves
Feature #5201: [SDKs] arv-put --description X --properties P should set respective attrs on new collection.
Feature #5237: [Workbench] Pipeline re-run dialog should offer docker image choices (latest/same), similar to script version choices
Feature #5245: [Documentation] Show footer with date last updated
Feature #5259: [Tests] Reduce dependency on integration tests
Feature #5260: [Tests] Implement long-running keepstore services in run-tests, similar to what api got in #3021
Feature #5284: [Docker] Auto setup as much as possible for new users
Feature #5296: [Workbench] Extend Sharing tab to collections
Feature #5297: [Workbench] Enable users to share with any other user
Feature #5316: [DRAFT] [Workbench] Browse change history for records
Feature #5375: [workbench] Enable in-browser preview for text files in collections
Feature #5382: [Documentation] arvados VM simulator, demonstrating how it looks to use Arvados from CLI
Feature #5384: [DRAFT] [Workbench] Improve display of pipelines and jobs
Feature #5387: [DRAFT] [Workbench] Improve provenance/used-by graphs so they are legible even for large pipelines / long provenance trails
Feature #5391: [SDK] Go KeepClient offers a "read block" API that recovers from transmission errors by attempting alternate servers.
Feature #5413: [DRAFT] [API/SSO] Use externally-defined (e.g., LDAP) groups as permission groups
Feature #5415: [Workbench] Improve presentation of anonymously-viewed collections (via both "sharing link" and regular copy-and-pasted Location)
Feature #5439: [Tests] Provide a docker image, based on puppet/packer process, that can run the entire test suite quickly.
Feature #5459: [Crunch] Job logs include node-level resource usage statistics (cpu, ram, swap, etc)
Feature #5494: [Documentation] Run an uncommitted script straight from a working directory using arv-run-pipeline-instance --run-here
Feature #5495: [Workbench] Change "recent pipelines" to curated selection of "public pipelines"
Feature #5510: [DRAFT] [Crunch] User can configure task retries
Feature #5511: [Workbench] Workbench should support viewing VCF files
Feature #5582: [API] [Workbench] Add read-only ("locked") flag to projects
Feature #5646: [Workbench] Display duration on pipeline instance listings
Feature #5668: [Workbench] Favorite projects, pipelines, collections
Feature #5676: [SDK] arv-copy automatically creates new repository(ies) on the destination cluster
Feature #5767: [API] Websockets subscribe API should accept a start timestamp, so clients don't have to miss any events logged between the given start time and websocket connection success.
Feature #5778: [FUSE] Support efficient copy at command line
Feature #5782: [Workbench] Support entering boolean script_parameters when editing a pipeline instance
Feature #5840: [API] Relax repository name constraints to permit [-_.] -- ideally allow anything github allows
Feature #5846: [Workbench] [Fiddlesticks] Live job log should show recent history
Feature #5962: [Keep] Use bufferpool in keepproxy to limit memory use when there are many concurrent PUTs
Feature #6015: [SSOs] Support authentication against multiple backend LDAP servers
Feature #6310: [FUSE] Support scaling the internal block cache based on number of open files
Feature #6348: [Deployment] [Documentation] Minimize system-wide dependencies for compute node setup
Feature #6442: [Workbench] [API] Support "world-readable, but not world-searchable" permission on projects: "anyone with the link can view"
Feature #6517: Port arvados-libpam to centos
Feature #6574: [Workbench] [Webshell] Rough edges in web shell
Feature #6782: [API] [Workbench] Generate debconf template from config yaml
Feature #7024: [Keep] keepstore status.json should report runtime environment
Feature #7127: [API] Job log cleaner should not clean logs when the job's log is the empty collection
Feature #7328: [Crunch] [UX] Standard excepthook to help debug job problems
Feature #7411: [Workbench] Pipeline template listings should include links to the scripts
Feature #7694: [API] [Workbench] Check for nonexistent config params during rake config:check
Feature #7818: [SDKs] Document that arv-copy accepts collection PDHs (ie content addresses) as arguments
Feature #7845: [Crunch] Crunchstat and arv-mount print final stats before exiting
Feature #7847: [SDKs] Update run-command to use arv-mount --mount-tmp instead of staging directory
Feature #8163: [FUSE] arv-mount should detect and log any files/dirs that are still open after unmounting
Feature #8212: [SDKs] arv CLI program accepts "-" and "/path/to/file" as values for parameters that expect JSON (most notably the "body" parameter)
Feature #8213: [SDKs] arv get (and edit) should support a flag that suppresses display of non-writable attributes like "status" and "writable_by"
Feature #8221: [Workbench] Add an "invert selection" button to checklists alongside "un/select all"
Feature #8228: [SDKs] [FUSE] Python SDK and arv-mount use Range requests when a caller requests part of a block that has been ejected from the cache
Feature #8318: [git] Should be able to clone git repositories that are shared with "anonymouspublic" without authentication
Feature #8344: [login-sync] Sets up settings.conf
Feature #8363: [FUSE] arv-mount exits non-zero if any I/O errors were returned
Feature #8379: [SDKs] Recursive arv-copy should report progress of the entire copy, in addition to individual objects
Feature #8415: [SDKs] arv-put's default output includes both collection UUID and PDH
Feature #8486: [Workbench] Change project "Jobs and pipelines" tab to just "Pipelines"
Feature #8498: [SDKs] GoSDK arvadosclient parameters are more transparent
Feature #8501: [SDKs] Add client-side join switches to `arv foo list`
Feature #8502: [SDKs] Fetch all results with the CLI `list` command
Feature #8515: [Node Manager] Detect and shut down nodes that fail provisioning in Azure
Feature #8551: [Node Manager] Minimum idle time before node is eligible for shutdown
Feature #8641: add key/value support for tags
Feature #9017: [Tests] Make slow tests optional (e.g., run-tests.sh --short)
Feature #9245: [FUSE] Make .arvados#collection available in collection subdirectories
Feature #9364: [keep-balance] "Expedited delete" tool: perform garbage collection on some specific (recently deleted) collections, bypassing usual GC race protections
Feature #9747: [CWL] Secondary files entering into Arvados automatically
Feature #9802: [Workbench UI] Pipeline instances should have a link to view the CWL file used.
Feature #9968: [Crunch2] [API] Support container request expiry (expires_at)
Feature #10123: [Workbench] [UNC] Copying collections also copies metadata
Feature #10141: [Packaging] Include sample config file (/etc/arvados/foo/foo.example.yml) with each configurable package
Feature #10164: Additional Crunch job logging controls
Feature #10183: [Node Manager] Support ephemeral additional storage for Azure node types that do not have sufficient instance storage
Feature #10191: Arvados should be able to cat/zcat files in collections easily
Feature #10217: [Crunch1] Log the node properties at the start of a job
Feature #10268: [SDK] Convenience method for resolving project/file paths
Feature #10309: [Workbench] add a configuration flag to hide compute nodes
Feature #10328: [CWL] a-c-r option to resubmit failed jobs
Feature #10403: Node manager should fetch pricing information from cloud APIs
Feature #10493: [Python SDK] Provide more control over pre-fetch behavior
Feature #10541: [Keep] Share buffers between overlapping/consecutive GET requests for the same block
Feature #10594: [API] [Workbench] Error out when unrecognized keys appear in config files
Feature #10830: [Workbench] Support array type input parameters to Workflows
Feature #10998: [keep-web] configurable block cache size
Feature #11146: [Crunch2] [Workbench] Show slurm queue position of containers submitted to slurm but not yet running
Feature #11153: [API] create_token script should allow operator to provide token scopes and opaque comment
Feature #11184: [Keep] Support multiple storage classes
Feature #11410: [SDK] tool for copying CWL file + input collections
Feature #11508: HTTP 503 status for planned downtime
Feature #11515: [CWL] [Workbench] Support for copying/moving pipelines
Feature #11555: [API] Webhook callbacks for server-to-server communication
Feature #11582: [API] Delete old job/container logs in main server process, instead of rake tasks invoked from cron jobs
Feature #11605: arv-put detects symlinks to common files and directories to avoid duplicated uploads
Feature #11668: [SDKs] Exceptions raised by Python SDK should include the "error token" if provided in the API response
Feature #11745: Support schema.org/Google structured markup for datasets
Feature #12009: Find arvados-cwl-runner version easier in a container request
Feature #12028: Be able to create a new collection based on a subdirectory in workbench
Feature #13382: [keepstore] Write new blocks to appropriate storage class
Feature #13403: [crunch-run] Cancel container on FUSE error
Feature #13611: Nodemanager reads instance sizes from the system config file to avoid duplication
Feature #13647: [keepstore] Load cluster configuration file
Feature #14712: [git] arv-git-httpd uses cluster config file
Feature #14713: crunch-dispatch-slurm uses cluster config file
Feature #14714: [keep] keep-balance uses cluster config file
Feature #14715: [keepproxy] Use cluster config file
Feature #14716: [keep-web] Use cluster config file
Feature #14717: [websockets] Use cluster config
Feature #14912: [Crunch2] Azure driver supports attaching extra storage
Feature #14922: [crunch-dispatch-cloud] Run multiple containers on a single VM
Task #2822: [SDKs] Use Ruby SDK instead of google-api-client in arv-run-pipeline-instance
Task #3190: [Blocker] Clarify design/requirements
Task #3591: Make chooser html partial DRYer
Task #3592: Handle display of any user's home page with projects#show
Task #3737: Include workbench URI in discovery document
Task #3738: arv-run-pipeline-instance: display workbench URI instead of the bare uuid
Task #3920: Use .utc-date data-utc-date javascript
Task #4211: a-r-p-i sets pipeline instance to 'Paused' in a catchall exception
Task #4212: [Workbench] Pipeline instance viewer should not omit paused pipelines
Task #4815: Make it highlight the differences once again
Task #4816: Review 4358
Task #4837: [SDKs] Define API and in-memory data structure for collections in Python SDK
Task #4874: Test that comparisons display correctly
Task #4893: Review 4823-python-sdk-writable-collection-api
Task #4987: Refactor Python tests to use fewer servers as able
Task #4993: Implement test server service, and use it across test suites
Task #5103: Implement live sync, merge
Task #5376: [workbench] wget link or download-all link in public collections
Task #5497: Review branch: 5375-preview-collection-text-files
Task #5649: [Workbench] We need better job graphs, less spaghetti
Task #5651: Fix reported scaling factor for failed jobs (see notes)
Task #5686: Review 5614-workbench-optimize-combine-collections-wip
Task #5689: Review 4223-arv-keepdocker-download
Task #5791: [Workbench] Job log memory and swap lines should use the same scale
Task #5860: Provide detail about expected information/presentation
Task #5918: Review 5523-stats-error
Task #5938: Handle normal container startup and shutdown races without logging an error/notice or missing the first interval
Task #5995: Review 5736-node-manager-easy-slot-cleanup-wip
Task #6109: Review
Task #6124: Add "goal mode" infrastructure
Task #6125: Enter goal mode when hitting Run on a pipeline template
Task #6126: Decide URL and advice for appropriate next step when in "pipeline goal" mode
Task #6127: Preserve template goal through login->UA->profile->back-to-work process, and then convert template to instance goal
Task #7339: Update the "Using arv-copy" user guide to include pipeline instances as one of the object types that can be copied over.
Task #7749: Draw UI Implementation
Task #8126: Review
Task #8210: review 8189-handle-large-collections-better
Task #8367: Review
Task #8368: Update docs
Task #8369: Test process
Task #8462: Review 8388-events-terminate
Task #8719: Review PR #40
Task #8976: Review 8912-missing-blocks-report
Task #9052: Review 9017-skip-slow-tests
Task #9101: Communicate --short flag to Rails test suites
Task #9102: Skip collection performance tests in API suite
Task #9103: Skip collection performance tests in Python SDK suite
Task #9158: Skip expensive setup/teardown, too
Task #9293: Review docs/spec [[Expiring collections]]
Task #9294: Document desired behavior and interpretation
Task #9296: Update API to desired behavior
Task #9297: Update Workbench to desired behavior
Task #9298: Update arv-mount to desired behavior
Task #9302: Review 9278-expiring-collections
Task #10046: rails systemd unit or nginx config
Task #10066: Review
Task #10073: Transition plan
Task #10330: Review 10008-check-token-exp-on-open
Task #10740: Review 10705-state-transition-error
Task #11198: Review
Task #11390: Review
Task #11391: Review
Task #11392: Review
Task #11571: Review
Task #11890: Review 10584-fuse-stop-threads
Task #12003: Review
Task #12005: Review
Task #12067: Instrumentation
Task #12336: Investigate Slurm API
Task #12965: Merge Composer 1.0 see if it fixes it
Task #13289: Investigate
Task #13290: Review
Task #13293: Review 13142-dispatch-nfiles
Task #13354: Review
Task #13355: Review
Task #13358: Review
Task #13572: Update developer docs with dependency version pinning requirements
Task #13732: Review
Task #13769: Review 13766-libyaml-workaround
Task #13846: Review 11942-tagging-support
Task #13848: Monitor dependencies for release of update
Task #13962: Review 13961-separate-polling
Task #14294: Review
Task #14662: Review
Task #14964: Port PAM to Python 3
Task #14965: Port arv-mount to Python 3
Story #2679: [API] If enabled by user via "preferences" checkbox in Workbench, notification service sends email to owner when a workflow finishes, fails, or is failing
Story #3022: Improvements to sortable log viewer
Story #3163: [Tests] Non-destructive test script suitable for running on a production VM/server in order to verify correct operation of basic features.
Story #3179: "Dump core" option when running a job (or every job in a pipeline): save the state of a failed task's container before tearing it down.
Story #3199: [SDK] Keep clients (e.g., arv-put and crunch-job) ask API to refresh blob signatures if expiry times are sooner than ETA of collection.create
Story #3279: [Workbench] Link to OFTC web chat on version/bugreport modal
Story #3298: [Workbench] Rearrange collection chooser to show a preview (file list) inline instead of in a separate preview pane. See [[Workbench UI images]]
Story #3355: [SDK] Show the API server URI for the new pipeline instance when running a pipeline from the command line.
Story #3365: [SDK] Show the Workbench URI for the job when running a job or pipeline from the command line
Story #3640: [SDKs] Add runtime option to SDKs (esp Python and arv-mount) to use a filesystem directory block cache as an alternative to RAM cache.
Story #3815: [SDK] arv-put supports --ignore-symlinks
Story #3897: [SDKs] Clients can specify which auth config file to load.
Story #4030: [SDK] Command line tool to manipulate collection manifests
Story #4128: [Workbench] [refactor] JavaScript refactoring
Story #4237: [Documentation] Update and expand user/tutorials/tutorial-keep-mount.html to show practical ways of finding data via FUSE mount.
Story #4255: [SDKs] Implement "arv keep less" and "arv keep check" natively instead of hoping whless and whcheck are installed.
Story #4350: [Workbench] Order by Collections "Name" column puts "folders" (collections containing files) at the top
Story #4566: [Documentation] Introduce compute workflow / pipeline concepts in user guide
Story #4744: [Workbench] Merge "user agreement" and "required profile fields" into a single page
Story #4833: [Crunch] Quick turnaround time when updating a docker image in order to re-run a job
Story #4919: [API] Arvados clients can use standard OAuth2 protocol instead of custom token handling mechanism
Story #5007: [Workbench] Pilot material design theme
Story #5094: [Workbench] Make pipeline rerun UI more like jobs'
Story #5142: [DRAFT] [API] Enrich supported dataclass values for pipeline templates
Story #5428: [Tests] Enhance keep testing to support signed locators.
Story #5468: [SDKs] Refactor arv-get/put/copy into the Python "arv" wrapper using common exception-handling and argument parsing
Story #5621: [Documentation] Improve SDK documentation
Story #5624: [API] [Workbench] [Performance] Use object caching to improvement performance (such as memcache).
Story #5781: [API] [DRAFT] Provide API methods for manipulating and combining collections
Story #5788: [SDKs] [refactor] Dry up KeepService get() and put() in arvados/keep.py
Story #5793: [Workbench] Add search filter to "/links" page
Story #5865: [Crunch] Run a local job using a docker image directly from the shell VM (without writing the docker image to Keep)
Story #5914: [DRAFT] Provide one clear way for users to get data from an external source into Arvados
Story #6058: [Workbench] Provide a guided workflow to new users who want to upload data and run a public pipeline
Story #6307: [Workbench] Public project listing should exclude subprojects
Story #6313: [Node Manager] Booting nodes shouldn't satisfy min_nodes
Story #6387: [SDKs] arv-put progress meter scales based on file size, rather than being fixed to MiB
Story #6405: [Workbench] Allow additional curation of the public projects listing
Story #6570: [DRAFT] [Workbench] More encouragement for users to send feedback
Story #6642: [API] Include a mechanism to automatically trust some clients
Story #6783: [Performance] Compare Collection performance benchmarks across components
Story #6840: [SDKs] Teach arv-copy to copy whole projects
Story #7394: [SDKs] arv-copy copies all metadata associated with an object
Story #7439: [SDKs] CLI tools support Mac
Story #7468: [Workbench] Displays content at the top of the dashboard, set by the sysadmin
Story #7469: [Workbench] Provide an interface for the sysadmin to set Dashboard banner content
Story #7470: [Workbench] Provide an interface to delete files from collections
Story #7471: [SDKs] Provide a CLI tool to add files to an existing collection
Story #7472: [SDKs] Provide a CLI tool to remove files from a collection
Story #7473: [FUSE] [SDKs] Ensure FUSE and other SDK clients work with a default replication level != 2
Story #7474: [FUSE] Add a --replication switch
Story #7476: [SDKs] Ensure `arv keep put` replicates properly
Story #7477: [SDKs] Go Keep client enforces a minimum transfer rate for proxy connections
Story #7542: [Documentation] Write a Crunch wiki page
Story #7543: [Documentation] Write a Crunch2 wiki page
Story #7621: [API] Job model validates that the script exists in the repository at script_version
Story #7830: [Workbench] User admin interface should show which VM(s) the user can login to
Story #7831: [FUSE] Add --umask, --fmask, --dmask options to control file permissions
Story #7973: [FUSE] Reset mount-tmp to the state of another collection
Story #7980: [Node Manager] Broken nodes always get shut down, exempt from all node-counting considerations
Story #7990: [FUSE] With --exec, expose the mount point in an environment variable
Story #8022: [SDKs] CLI tools install tab completion aids
Story #8058: [Documentation] Add Gitolite sanity check to install guide
Story #8091: [Workbench] Show all known runtime_constraints in job status pane
Story #8179: [keep-balance] Refuse to act when planning to delete data above an admin-specified threshold
Story #8181: [Keepstore] Support a portable data challenge API
Story #8313: [SDK] arv-copy better Git repository error reporting
Story #8314: [SDKs] arv-copy continue on error
Story #8328: [Deployment] .yml.example files should be consistent
Story #8329: [Docs] SSO install guide documents default_link_*
Story #8331: [Documentation] Clarify PostgreSQL setup commands for Software Collections users
Story #8332: [Docs] Fix flow of Rails servers' install guides
Story #8552: [Crunch] Log throttling shouldn't throttle logs from crunchstat, crunch-job
Story #8560: [Workbench] Ability to create a new project when creating a new pipeline instance
Story #8562: [Workbench] Project listings act like file manager directory trees, with expanding+collapsing trees
Story #8565: [API] Go code to connect to the API server database and make basic sense of it
Story #8566: [API] Go code to implement the Arvados API permissions model
Story #8774: [Crunch] Limit maximum wait time between retries
Story #8775: [Docs/Workbench] Improve SSH agent documentation
Story #8778: [Docs] Document Git 1.7.10+ dependency for shell node and HTTPS Git cloning
Story #8780: [SDKs] Git credentials helper in CLI tools
Story #8781: [Docs/Workbench] Document how to clone repositories over HTTPS
Story #8838: [SDKs] PySDK Collection class lets you iterate files and subcollections separately
Story #8885: [PySDK] API client logs a warning when you create an object without an owner_uuid
Story #8894: [Node Manager] Less logging of tracebacks for routine exceptions
Story #8912: [Data manager] Verbose reporting
Story #9045: [SDKs] `arv keep docker` can upload an image already saved as .tar
Story #9046: [Crunch2] Can issue container request to pull and import Docker images
Story #9147: [SDKs] `arv keep docker` can make links for an image already in a collection
Story #9148: [API] Finalize and document the collections/provenance and collections/used_by API calls
Story #9150: [API] Synthetic collections include the "kind" field
Story #9278: [Crunch2] Document/fix handling of collections with non-nil expires_at field
Story #9428: [SDKs] Port arv-keepdocker to use the docker-py library
Story #9441: [Workbench] Improve user page group memberships management [to be groomed]
Story #9584: [FUSE] Don't display expiring collections in regular collection listings
Story #9590: [FUSE] Trash directory to list, inspect, and un-trash trashed collections
Story #9591: [FUSE] Undelete collections by moving them out of the TrashDirectory
Story #9592: [FUSE] rmdir on CollectionDirectory sets expires_at
Story #9839: [Crunch2][Workbench] List both pipeline_templates and workflows on both pages
Story #9937: [Workbench] file upload should provide progress bar for overall upload, not just individual files
Story #9952: crunch-dispatch-local config file and systemd unit
Story #9958: login-sync config file & systemd unit
Story #9959: nodemanager systemd unit
Story #9960: Automate starting/enabling services during Workbench and API package installation
Story #10027: Move Container related documentation from wiki to code
Story #10213: Efficient queries for notification menu items
Story #10242: [Performance] Fetching the pipeline instance qr1hi-d1hrv-jl9fvzy3x8sdj8v seems to take too long
Story #10244: [Performance][Workbench] Computing notifications in topnav is expensive. Can the users properties be used instead?
Story #10245: [Performance][Workbench] Computing user notifications in topnav is expensive. Can the user's properties be used instead?
Story #10344: [Workbench] Import CWL workflow
Story #10354: Add varchar_pattern_ops to all relevant PostgreSQL UUID indexes
Story #10355: [Workbench] Enable refresh for "small" pipelines
Story #10387: Faster downloading using arv-get
Story #10388: Request collections that don't (yet) exist via fuse interface
Story #10402: [CWL] Reduce redundant work with subworkflow RunInSingleContainer
Story #10405: [API] Remove unimplemented fields from api_client_authorization
Story #10477: [keepstore] switch from goamz to a more actively maintained client library
Story #10511: [Crunch2] [API] Specify which user's credentials should be used by a container
Story #10679: [Workbench] Update job stats to be more useful to end user
Story #10717: Deprecate arv-run
Story #10849: [API] [SDKs] When client and server versions differ, the server must either implement the API the client expects, or return an error.
Story #10904: [Spike] virtualenv arvados-cwl-runner
Story #11016: Document how to choose a suitable blob signature TTL
Story #11147: Write scalability test which can be used periodically as needed
Story #11159: [GA4GH] Import from tool registry API
Story #11160: [GA4GH] Expose workflow records via tool registry API
Story #11161: [GA4GH] Support task execution API using crunch v2 containers
Story #11163: [GA4GH] Participate in DREAM workflow interoperability activities
Story #11171: [Nodemanager] Packaging for compute node setup scripts
Story #11251: [Tests] Make workbench integration tests finish faster (at least in Jenkins)
Story #11285: Update Pipeline Optimization wiki with CWL/Crunchv2
Story #11421: [Workbench] UUIDs & PDHs should be hyperlinked
Story #11452: Support rendering of CWL description fields with embedded markup
Story #11524: Implement disk usage reporting for the administrator
Story #11558: [FUSE] Integration test using real programs with complex write patterns.
Story #11705: [API] Remove async permissions code
Story #11740: [Workbench] Support display & reseting of expiration dates for collections
Story #11741: [Crunch2] Create workflow object for CWL runs which are initiated from the command line
Story #11871: [Workbench] Add a "Browse" tab that links to tree/master in repository#Show page
Story #11875: User accounts optionally have read-only access
Story #11942: [CWL] arvados-cwl-runner should support tagging output collection using properties
Story #11983: On the collections page indicate expiring collections
Story #12089: Javascript SDK
Story #12191: Support SLURM parameter passthrough from CWL jobs
Story #12239: Allow templating of the collection sharing web page
Story #12289: Migrate to Slurm API
Story #12409: [cwl] cwl-tool conditional implementation
Story #12917: Users should be able to see if a container failed from the container_request method
Story #13046: Provide configuration knob to disallow external access through keep-proxy
Story #13048: Refactor crunch2 logging
Story #13077: R SDK submit to Bioconductor
Story #13096: Investigate packrat to lock down R package dependencies
Story #13522: [keepproxy] document limiting maximum number of HTTP connections
Story #13571: Review dependencies to make sure that they meet requirements
Story #13731: Design new nodemanager config data organization for cluster-wide config
Story #13790: Add metrics endpoint to arvados-controller
Story #14374: Multi-site object search
Story #14532: Prepare for Python 2 sunsetting in December 2019
Story #14659: Merge Workbench2 repo into Arvados repo

OPS - Kanban

Due in about 10 months (01/03/2020)

Ops does kanban please see https://dev.arvados.org/projects/ops/issues/kanbans

64%

574 issues   (361 closed — 213 open)

Related issues
Feature #7331: [Deployment] Add systemd unit definition to the keepproxy package
Feature #7407: [Deployment] Add sysvinit script to the keepstore package
Task #7408: Review
Task #14390: [OPS][JENKINS] slaves are not launching anymore on gcloud
Task #14394: [OPS][Jenkins] upgrade Jenkins version and plugins
Task #14427: [OPS][JENKINS] create Federation Jenkins Slave

Curoverse Support - Pipeline Future Sprints

Due in about 10 months (01/05/2020)

19%

37 issues   (7 closed — 30 open)

Related issues
Task #8974: Review
Story #8563: [Docs] Pipeline author guide gives a basic demonstration of including a third-party tool
Story #8676: [Documentation] Promote pipeline porting pages
Story #8770: [Documentation] Integrate Sally's "Porting a pipeline" training presentation with Arvados documentation

Deferred

Due in about 22 months (12/31/2020)

54%

44 issues   (23 closed — 21 open)

Deferred

Arvados Contributor Wiki

This wiki has information related to the open source project. It's a work in progress, so come back often for updates and feel free to become a contributor.

Platform Overview

Release Notes

Key Components

Using Arvados

Contributing and hacking

Community

Legal

Related issues
Feature #2706: [SDKs] Python SDK Keep client can enable "HEAD before PUT" mode when writing. Expose this as an arv-put option.
Feature #3162: [API] Client can request signed locators for specific blobs from API server
Feature #3402: If a pipeline/job is in a project, it would be nice if the breadcrumbs reflected this for easy maneuvering back and forth
Feature #3498: Counting the number your pipelines were used by unique users
Feature #3528: Machine-readable API schema that indicates which fields of each object type permit updating by regular (non-admin) users who have can_write or can_manage permission.
Feature #4011: [Crunch] port crunch-job to python
Feature #5768: [Node Manager] Add max_node_per_user config
Task #2455: Fill in missing story description - goals / examples
Task #2565: Check out 2028 notes and sync up with Tom
Task #2633: Implement TLS support internally
Task #2636: Test SSL support
Task #2643: "Ready to ingest" endpoint (in /actions controller?)
Task #2647: Put files from staging directory into Keep
Task #2648: Write tests
Task #2649: Assign collection to project
Task #3189: [Blocker] Clarify design/requirements
Task #3459: Add documentation on how to specify a UserAgreement
Task #3460: Add a link and make it possible to see signed user agreements
Task #3461: Review 3257
Task #3523: Use fancy editor component for Textile textareas
Task #5503: Deploy "proxy_buffer off;" on /collections/download
Story #1558: [Lightning] Lightning Memory-Mapped Database
Story #2028: [SDKs] Python SDK supports Django model/manager style usage
Story #2293: [Keep] Keep server supports SSL
Story #2523: [Sample clients] Ingestor utility responds to "ready to ingest" events by copying the specified files from staging directory to Keep and storing the collection in the specified group.
Story #3093: [Crunch] Pipeline can use other pipelines templates as components
Story #3180: [Crunch] When a job fails in a pipeline instance, should the other queued/running jobs get cancelled automatically?
Story #3299: [Workbench] Render textile markup in description fields
Story #3304: [Workbench] Add checkboxes to multi-type chooser modal to filter by object type
Story #3473: Ability to mark one's own projects as read-only to avoid changing it by accident.
Story #3491: [Keep] Support transparent compression of blocks in Keep
Story #3519: Move permission links to their own table in API server.
Story #5471: [Workbench] Remove ActiveRecord dependency

To Be Groomed

Due in about 22 months (01/01/2021)

A fake sprint to hold stories to be groomed

1%

158 issues   (1 closed — 157 open)

To Be Groomed

Arvados Contributor Wiki

This wiki has information related to the open source project. It's a work in progress, so come back often for updates and feel free to become a contributor.

Platform Overview

Release Notes

Please see our blog

Key Components

Using Arvados

Contributing and hacking

Community

Legal

Related issues
Feature #5214: [SDKs] Use sphinx to render for Python SDK docs
Feature #6865: [Documentation] Higher Level Python SDK Reference Page
Feature #8064: [Keep-web] Support CORS requests with Authorization headers
Feature #9136: [Python SDK] Add a Collection method to efficiently concatenate files
Feature #10767: [API] [arvados-ws] remove all websockets and puma related code from the api server codebase
Feature #11183: [Deployment] Use consul to report health of services/components
Feature #11442: [CWL] Resolve symbolic names to collections
Feature #12090: Collections/data access API
Feature #12517: [Workbench] page with a view of all the top level container requests
Feature #12692: [keep-web] Accept /t=token/c=collection/path (currently only /c=collection/t=token/path)
Feature #13062: [SDK] Reduce collection class memory footprint
Feature #13126: [keep] Investigate using signed URLs to delegate access to cloud buckets
Feature #13272: [keep-web] read/write support for /by_id and /users
Feature #13388: [Epic] Zero-downtime upgrade
Feature #13389: [Epic] Easy cluster install
Feature #13557: Copying a container request should link to the original container request
Feature #13643: [CWL] Requirement for storage classes of data
Feature #13675: [Composer/Workbench] Import of public git repo to a new Arvados repository
Feature #13913: Crunchstat-summary graphs tmpdir usage
Feature #14016: [API] Container request can provide existing collection UUID that will accept CR output
Feature #14018: [CWL] arvados-cwl-runner should support tagging container requests using properties
Feature #14021: [crunch-dispatch-slurm] option to set job priority directly instead of using nice values
Feature #14030: [CWL] Use streamable flag as a hint to use keep output
Feature #14168: [Running a workflow] Input presets
Feature #14200: [API] Reduce privilege exposure via API tokens in multi-cluster workflows
Feature #14257: [controller] refresh manifest API
Feature #14457: [api] send e-mail notification to user when their arvados account is activated
Feature #14706: [Crunch2] Retain references + permissions to earlier containers when retrying a container request
Feature #14811: [SSO] Use cluster config
Feature #14812: [Workbench1] Use cluster config
Feature #14813: [Workbench2, Composer] Use cluster config
Feature #14874: [API] Protected collection property -- preserves original project owner through copy/move/modify operations
Feature #14930: Add flag to arv-put to set a trash_at / delete_at date
Feature #14974: [Workbench2] Linking to collections in input object display, recording uuids of input collections
Task #12588: Review
Task #13160: Review
Task #13839: Review 13804-no-shutdown-wanted-nodes
Task #14202: Test libcloud PR 1202 with Azure fixes
Task #14588: Review 14584-container-reuse-perm
Story #8573: crunchstat-summary should calculate averages of all metrics
Story #9425: [API?/Workbench] Search for a link tag value returns linked objects
Story #9653: [keepstore] Limit entire process to configured amount or % of system memory (replace -max-buffers=X)
Story #10525: Integrate CI::Reporter to provide Ruby test output in JUnit XML format
Story #11724: Pull Docker images without requiring Docker on client
Story #11748: Standard convention for Nagios-compatible health check scripts kept alongside source for services
Story #11770: [Python SDK] Implement support for universal newline mode in Collections API
Story #12288: Status endpoint aggregator
Story #12527: [Composer] Refactor to separate vendor-specific pieces and all them to be replaced
Story #12571: arv-mount by_tag directory support for properties
Story #12666: --ignore-docker-for-reuse not passed to crunch v2 jobs started via --submit --no-wait
Story #12703: [Workbench] Self serve account merge
Story #12704: [API] Associate multiple identity_url from SSO with same user
Story #12961: Support Bitbucket git repo access from Composer
Story #13074: Notify Composer user when file changed
Story #13110: [WebDAV] WebDAV performance enhancements
Story #13218: Support browsing of projects shared with me in WebDAV
Story #13325: As a CWL learner, I would like to be able to run CWL workflows easily on cloud.curoverse.com
Story #13326: As a data provider, I would like the ability to provide metered access to my data on Arvados, including start/end dates, bandwidth caps, and % of file accessed
Story #13384: [Workbench] Search users by email address and shell account
Story #13390: Document API in API discovery document
Story #13494: [Workbench2] View/copy/expunge previous versions of a collection
Story #13648: [Epic] Use one cluster configuration file for all components
Story #13697: Prevent the API server and database from continuing to serve requests to clients after timeout
Story #13760: Provide more information to SLURM to make scheduling decisions on HPC
Story #13792: Document strategies on how to use existing resources to save user preferences
Story #13794: Build Docker images without requiring Docker on client
Story #13908: Replace SLURM for cloud job scheduling/dispatching
Story #13921: Log stats for the entire node lifecycle
Story #13925: Default keep cache scales with machine size
Story #13954: [API server] Implement request throttling
Story #13995: [Controller] Port "collection update" functional tests from Rails API to controller test suite
Story #13997: [Controller] Port "collection update" API from Rails
Story #14001: [Spike] [Controller] Port "update workflow" API to controller
Story #14004: [Controller] Keep database in tune
Story #14151: Extend vocabulary support for properties to support strong identifiers and multiple labels
Story #14284: Send real time container logs to a suitable log distribution system (instead of adding rows to the postgres logs table)
Story #14467: [SDK] Merge Go ArvadosClient and arvados.Client and migrate to arvados.Client
Story #14803: CWL v1.1 specifies directory listings are ordered
Story #14824: postgresql 10.x compatibility
Story #14888: [CWL] running expression tool on arvados doesn't return proper output
Story #14923: [CWL] Reduce memory footprint by using webdav to get file/dir listings in collections (instead of Python SDK)