Story #18103

Script to scan S3 bucket for new data & copy to Arvados

Added by Peter Amstutz 17 days ago. Updated 6 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-
Release relationship:
Auto

Description

Script that can run from a cron job or a workflow that scans S3 bucket, finds data, determines batch, sample id, and type. Copies data to Arvados, sets batch id, and updates status of sample to "sequenced".

Path is (bucket) batch / sample id / sample files:

(data-release) MM-002DNA/ MM_0026_DNA_T_04_01/ MM_0026_DNA_T_04_01_L001_R1_001.fastq.gz

Match sample id to existing sample id & upload to Arvados.
Update sample status to SEQUENCED.
Find or create batch and associate sample id with batch.

History

#1 Updated by Peter Amstutz 17 days ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 6 days ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 6 days ago

  • Description updated (diff)

Also available in: Atom PDF