Bug #20826
closed
Long arv-mount Queue time for large input jobs
Added by Alex Coleman 10 months ago.
Updated 10 months ago.
Description
I ran two instances of the same job, and both had a subworkflow step that stayed queued for over an hour, stuck at the Running [arv-mount --foreground --read-write ...] stage.
Container requests UUIDs: pirca-xvhdp-rynp7k752rvbebv, pirca-xvhdp-ne70ju4ksqxjvyu.
Other relevant information may be that it was running on 270 GiB of input data.
Expected behavior: I would expect that it would not stayed queued this long.
- Target version set to Future
in run_agc.cwl
InitialWorkDirRequirement:
listing:
- entry: $(inputs.fileList)
writable: true
- entry: $(inputs.refFile)
writable: true
- entry: $(inputs.outAgc)
writable: true
Why are you doing this? the 1 hour startup time is because it's downloading all the files in order for them to be locally writable.
Ihe inputs shouldn't need to be writable, and shouldn't even need to be in InitialWorkDirRequirement?
- Target version deleted (
Future)
- Status changed from New to Resolved
We established this is working as designed, the data is being staged to the working directory as "writable" which causes all the data to be copied (so that it can be modified by the process). So all that time is being spent doing data copy in preparation to run. This was due to some issues with the upstream agc tool that Alex is addressing.
In theory, crunch-run could do this more efficiently (by preparing the collection via manifest manipulation) but that requires more substantial effort and should be written as a separate ticket.
Also available in: Atom
PDF
Updated by