Storing and Organizing Data¶

Rough demo outline

1. Automatic ingest from a POSIX directory to Keep
   • Access to existing staging area (e.g., remote NFS share) is arranged ahead of time as an admin/setup task
   • Optional(?) User can manage staging areas hosted inside Arvados
   • Someone (possibly 3rd party) uploads some files to the staging area via SFTP or whatever
   • Uploader does an API call to {something - ingestor app? directly to arvados api endpoint?}. In the API call, the uploader provides a tag (e.g., a sample ID) and a list of files, checksums, etc.
   • Ingestor daemon reads the data from the staging area and writes it into Keep; creates one collection per API call made by uploader
   • In Workbench the imported Datasets appear as Collections in the designated project
   • After data has been copied into Keep, ingestor deletes the files from the staging area (this had better be configurable!).
2. My data gets into the right project as specified by the uploader (API call)
3. Subscribe to notifications (by email and/or Workbench dashboard): when files start/finish uploading; when files are shared with customer; when files are downloaded by third party
4. Move/copy collections between projects (Project RX1234, or Customer X’s files), tag them in destination project with the appropriate string (e.g., sample ID) -- defaulting to existing tag used in source project (e.g., provided at time of upload).
   • UI for presenting Groups as Projects/Folders
5. “Anyone with this secret link can view/download” mode. Enable, disable, change magic link. Use cases: browser + “wget -r”.
   • Perhaps the secret in the secret link is an ApiClientAuthorization token, belonging to the person creating the link, scoped to a single project/collection
6. See log/overview of who has accessed your shared data (incl. “anonymous user” if using secret-link-to-share); when shared/unshared; when each upload started/finished -- for a single project, and across all projects