Project

General

Profile

Storing and Organizing Data » History » Revision 11

Revision 10 (Tom Clegg, 04/04/2014 02:33 PM) → Revision 11/33 (Tom Clegg, 04/04/2014 02:39 PM)

h1. Storing and Organizing Data 

 Rough demo outline 

 # Automatic ingest from a POSIX directory to Keep 
 #* Access to existing staging Staging area (e.g., remote NFS share) 
 #* Optional(?) User can manage staging areas hosted inside Arvados is set up ahead of time 
 #* Someone (possibly 3rd party) uploads some files to the staging area via SFTP or whatever 
 #* API call to {something? ingestor app? arvados?} 
 #* API call provides a tag (e.g., a sample ID) 
 #* In Workbench the imported Datasets appear as Collections in the designated project 
 # My data gets into the right project as specified by the uploader (API call)  
 # Subscribe to notifications (by email and/or Workbench dashboard): when files start/finish uploading; when files are shared with customer; when files are downloaded by third party 
 # Move/copy collections between projects (Project RX1234, or Customer X’s files), tag them in destination project with the appropriate string (e.g., sample ID) -- defaulting to existing tag used in source project (e.g., provided at time of upload). 
 #* UI for presenting Groups as Projects/Folders 
 # “Anyone with this secret link can view/download” mode. Enable, disable, change magic link. Use cases: browser + “wget -r”. 
 # See log/overview of who has accessed your shared data (incl. “anonymous user” if using secret-link-to-share); when shared/unshared; when each upload started/finished -- for a single project, and across all projects