Storing and Organizing Data¶

Rough demo outline

1. Automatic ingest from a POSIX directory to Keep
   • Access to existing staging area (e.g., remote NFS share) is arranged ahead of time as an admin/setup task
   • Optional(?) User can manage staging areas hosted inside Arvados
   • Someone (possibly 3rd party) uploads some files to the staging area via SFTP or whatever
   • Uploader does an API call to {something - ingestor app? directly to arvados api endpoint?}. In the API call, the uploader provides a tag (e.g., a sample ID) and a list of files, checksums, etc.
   • Ingestor daemon reads the data from the staging area and writes it into Keep; creates one collection per API call made by uploader
   • In Workbench the imported Datasets appear as Collections in the designated project
2. My data gets into the right project as specified by the uploader (API call)
3. Subscribe to notifications (by email and/or Workbench dashboard): when files start/finish uploading; when files are shared with customer; when files are downloaded by third party
4. Move/copy collections between projects (Project RX1234, or Customer X’s files), tag them in destination project with the appropriate string (e.g., sample ID) -- defaulting to existing tag used in source project (e.g., provided at time of upload).
   • UI for presenting Groups as Projects/Folders
5. “Anyone with this secret link can view/download” mode. Enable, disable, change magic link. Use cases: browser + “wget -r”.
6. See log/overview of who has accessed your shared data (incl. “anonymous user” if using secret-link-to-share); when shared/unshared; when each upload started/finished -- for a single project, and across all projects