Story #18103

Script to scan S3 bucket for new data & copy to Arvados

Added by Peter Amstutz 9 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-
Release relationship:
Auto

Description

Script that can run from a cron job or a workflow that scans S3 bucket, finds data, determines batch, sample id, and type. Copies data to Arvados, sets batch id, and updates status of sample to "sequenced".

Path is (bucket) batch / sample id / sample files:

(data-release) MM-002DNA/ MM_0026_DNA_T_04_01/ MM_0026_DNA_T_04_01_L001_R1_001.fastq.gz

Match sample id to existing sample id & upload to Arvados.
Update sample status to SEQUENCED.
Find or create batch and associate sample id with batch.

History

#1 Updated by Peter Amstutz 9 months ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 8 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 8 months ago

  • Description updated (diff)

#4 Updated by Peter Amstutz 3 months ago

  • Status changed from New to Resolved

Also available in: Atom PDF