Proposals and design documents¶

Note: these are work in process documents used to facilitate discussion among the engineering team. They may be of historical interest to learn the thinking when a feature was original developed. Do not assume any of these are comprehensive, reflect the eventual actual implementation, describe a feature that actually exists, or is any guarantee that a feature will be implemented in the future. Visit the documentation site for official documentation of fully implemented and supported features.

Current/pending proposals¶

• Multi-pass mode to reduce keep-balance memory footprint
• Efficient block packing for small WebDAV uploads
• Workbench responsiveness
• Credential storage
• Objects as pseudo-blocks in Keep
• FUSE v2
• Concurrent writes to a single collection
• Admin CLI for managing dispatcher / cloud VMs
• Glacier
• Auto-discovery
• Changing Keep hash algorithm

Design documents¶

• Groups, Projects, Ownership and Permissions Specification
• Workbench authentication process implementation notes
• Keep Design Doc
• Authentication
• Containers API
• Container dispatch
• Audit logs and other events/logs
• Crunch2 run
• Crunch runner
• Keep-web flow
• Websocket server

Archived¶

• WebDAV performance metrics
• Optimal runtime constraints API
• Scaling things
• Efficient live access to container logs
• Diagnostics command
• Accessing Keep using S3 protocol
• All-in-one package
• Accepting OpenID access tokens
• API Historical/Forcasting data for CR
• Multi-cluster user database
• Native login implementation
• Index of files in collections
• Browsing shared projects
• Controller architecture transition
• Controller architecture
• Cluster-wide favorites
• Golang API server
• Dispatching containers to cloud VMs (Fixing cloud scheduling)
• Build docker images as part of a workflow
• Use a docker image from a remote registry
• Container status / outcome reporting
• API token format
• Source tree layout
• Installing controller service
• Collection version history
• Declarative test framework
• Container secret mounts
• Cluster configuration
• Vault
• Message queue
• Keep storage groups
• Keep storage tiers
• Keep service hints
• Federation
  • Federation use cases
  • Federated identity
  • Distributed workflows
  • Routing multi cluster requests
• Websocket v0 shim
• Websocket server
• Events API
• S3 bucket volume implementation
• Crunch2 installation
• Jobs API, Crunch1-in-Crunch2
• Content addressed records
• File splits
• Separating files from collections
• Collection API - Performance enhancements
• Performance profiling
• Collection update API
• Git strategy for pipeline development
• CLI client
• Node manager policy matrix
• Expiring collections
• Docker security
• Metadata
• Package versioning
• Consul
• Crunch v2 cloud scheduling
• Keep cache policy
• Privileged containers

• Keep Proxy Specification
• Collections as regular Arvados objects
• Reusable tasks
• Everything is a task
• Tasks as jobs
• Pipelines as jobs
• Better git integration
• Writable FUSE mount
• Data Manager Design Doc