Project

General

Profile

Storing and Organizing Data » History » Version 16

Tom Clegg, 04/04/2014 02:48 PM

1 7 Tom Clegg
h1. Storing and Organizing Data
2 3 Tom Clegg
3 7 Tom Clegg
Rough demo outline
4 3 Tom Clegg
5 8 Tom Clegg
# Automatic ingest from a POSIX directory to Keep
6 12 Tom Clegg
#* Access to existing staging area (e.g., remote NFS share) is arranged ahead of time as an admin/setup task
7 11 Tom Clegg
#* Optional(?) User can manage staging areas hosted inside Arvados
8 9 Tom Clegg
#* Someone (possibly 3rd party) uploads some files to the staging area via SFTP or whatever
9 14 Tom Clegg
#* Uploader does an API call to {something - ingestor app? directly to arvados api endpoint?}. In the API call, the uploader provides a tag (e.g., a sample ID) and a list of files, checksums, etc.
10 13 Tom Clegg
#* Ingestor daemon reads the data from the staging area and writes it into Keep; creates one collection per API call made by uploader
11 9 Tom Clegg
#* In Workbench the imported Datasets appear as Collections in the designated project
12 15 Tom Clegg
#* After data has been copied into Keep, ingestor deletes the files from the staging area (this had better be configurable!).
13 6 Tom Clegg
# My data gets into the right project as specified by the uploader (API call) 
14
# Subscribe to notifications (by email and/or Workbench dashboard): when files start/finish uploading; when files are shared with customer; when files are downloaded by third party
15
# Move/copy collections between projects (Project RX1234, or Customer X’s files), tag them in destination project with the appropriate string (e.g., sample ID) -- defaulting to existing tag used in source project (e.g., provided at time of upload).
16 10 Tom Clegg
#* UI for presenting Groups as Projects/Folders
17 6 Tom Clegg
# “Anyone with this secret link can view/download” mode. Enable, disable, change magic link. Use cases: browser + “wget -r”.
18 16 Tom Clegg
#* Perhaps the secret in the secret link is an ApiClientAuthorization token, belonging to the person creating the link, scoped to a single project/collection
19 6 Tom Clegg
# See log/overview of who has accessed your shared data (incl. “anonymous user” if using secret-link-to-share); when shared/unshared; when each upload started/finished -- for a single project, and across all projects