https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422018-06-18T20:28:37ZArvadosArvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=635632018-06-18T20:28:37ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>Update: actually, the amount of time it takes for crunch-run to prepare 58000 mounts appears to be quite a bit longer - after 3 hours it has gotten to 28000 and is adding additional files at a rate of around 1 per second, so the instantaneous estimate to complete the remaining 30000 is over 8 hours. Not sure why it is slowing down...</p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=635672018-06-18T22:19:06ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>On further investigation, it looks like this many mounts causes the docker daemon to fail - they are just stuck in "Created" state forever: <br /><pre>
# grep bd441dba528a /var/log/syslog
Jun 18 18:29:53 arvados-compute-node-eglyx-006 dockerd[1254]: time="2018-06-18T18:29:53.583014105Z" level=error msg="bd441dba528a93597cadb2461d99bbc7aa67ae14320b45dd2a5da5bdb5e221be cleanup: failed to delete container from containerd: no such container"
Jun 18 18:29:53 arvados-compute-node-eglyx-006 dockerd[1254]: time="2018-06-18T18:29:53.583161488Z" level=error msg="Handler for POST /v1.21/containers/bd441dba528a93597cadb2461d99bbc7aa67ae14320b45dd2a5da5bdb5e221be/start returned error: grpc: received message larger than max (15190409 vs. 4194304): unknown"
# docker ps -a |grep bd441dba528a
bd441dba528a f9185748efcb "python -c 'import j…" 4 hours ago Created <a href="https://arvadosapi.com/eglyx-dz642-pfcmic2v69ymg4m">eglyx-dz642-pfcmic2v69ymg4m</a>
</pre></p>
<p>We have huge numbers of these piling up on our compute nodes:<br /><pre>
# docker ps -a -f status=created | wc -l
281
</pre></p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=637152018-06-21T16:06:32ZTom Morristfmorris@veritasgenetics.com
<ul></ul><p>I suspect the root cause of this is the poor performance of the collection handling code, which the FUSE driver depends on, and the FUSE driver itself.</p>
<p>If you're not actually accessing the file contents, but instead just the names, perhaps you can pass them as strings instead of Files to sidestep the whole issue.</p>
<p>If you need to reference a large number of collections (even if it's just to validate that they contain the necessary files), you are probably in trouble regardless of how you organize the work because all of those collections manifests will need to be fully parsed, turned into ArvFile objects, cached, flushed from the cache, etc.</p>
<p>Without making the underlying collection handling code faster/smarter, there may not be a satisfactory solution to this.</p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=640402018-07-05T18:25:26ZTom Cleggtom@curii.com
<ul></ul><p>If crunch-run is spending a lot of time doing serial calls to Stat(), it might help to stop doing them serially.</p>
<p>The error message suggests the purpose of the Stat() calls is to confirm that the files exist. This could be done without hitting arv-mount at all, by loading the collections with native Go code in crunch-run. It wouldn't have the (presumably desirable) side effect of preloading arv-mount's cache, but it would be much faster.</p>
<p>Either way, I think it's worth adding a way to tell crunch-run that it's OK to mount "/a/b", "/a/c", and "/a/d" by bind-mounting "/a" to a directory that's managed by the fuse driver, rather than bind-mounting b, c, and d individually -- IOW, "/a" can be empty and non-writable regardless of what the docker image has there.</p>
Rather than a "multiple collections" mount or otherwise extending the "mounts" structure to express a tree of mounts, though, I think it would be better to just specify that "/a" is empty and non-writable, and let crunch-run take advantage of this to provide the desired mounts more efficiently.
<ul>
<li>avoids the possibility of ambiguous mount structures where a single path "/a/b" has two different specs, one given under the multi-mount rooted at "/a", and another in a mount rooted at "/a/b" </li>
<li>avoids coupling the shape of the mounts spec to [the client's opinion about] the strategy for implementing a given set of mounts</li>
</ul> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=643392018-07-12T15:33:57ZTom Cleggtom@curii.com
<ul></ul><p>To clarify, even if the Stat() loop is fast, docker can't handle the <em>number</em> of mount points we need. Some form of "one bind mount satisfies multiple container mounts" facility is definitely required.</p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=643422018-07-12T18:18:52ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>As Tom mentioned, in addition to performance issues, we see a more fundamental problem in that docker is unable to launch containers with too many mounts - the problem manifests itself in that the container configuration json is too large to be passed to libcontainer (grpc Max message size exceeded).</p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=658192018-08-16T15:14:13ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>To Be Groomed</i></li></ul> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=662812018-08-31T13:45:36ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>There's arv-mount --mount-by-id but that only lets you rename the logical by_id directory, not specify individual collections.</p>
<p>I agree with Tom Clegg that it would be useful to have a feature to pass a list of PDH or UUIDs that should be accessible through the mount point<br />(probably through a file since passing 58000 command line arguments probably won't work). I don't believe that feature has been implemented. That would require some changes to arv-mount and also updating crunch-run to use it.</p>
<blockquote>
<p>We have one step with over 58000 inputs (which actually does not even need to access any of those file inputs - it is simply a step that processes that transposes a matrix of files without actually accessing any of their contents).</p>
</blockquote>
<p>I don't know exactly what you're doing, but have you considered alternative approaches?</p>
<ol>
<li>Use a CWL ExpressionTool to reorganize the files in javascript into the desired structure</li>
<li>Consolidate inputs into a list of collection PDHs (strings, not File objects) and pass that to a command line tool which uses the Arvados Python SDK to copy the files to an output collection, produce an cwl.output.json referencing the output collection.</li>
</ol> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=665002018-09-05T22:37:05ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>While it's true that our steps that basically just provide CWL JSON rearrangement functionality have the largest number of inputs, we also have "real" steps which have 36k actual input files. For example, the GATK4 workflow we are currently running has 18k gVCFs and 18k gVCF indices which are the outputs from 18k HaplotypeCaller runs that are being fed into a single GenomicsDBImport step to be merged into a single GenomicsDB for subsequent GenotypeGVCFs.</p>
<p>This should be a totally reasonable thing to do, and with the new functionality we have implemented as a fix to this issue, the GenomicsDBImport steps are running on many inputs without issue (GenomicsDBImport opens and reads the inputs in turn rather than trying to open them all at once).</p>
<p>The code we are running that fix 13636 for us are:<br /> - <a class="external" href="https://github.com/wtsi-hgi/bindmapfuse">https://github.com/wtsi-hgi/bindmapfuse</a><br /> - <a class="external" href="https://github.com/wtsi-hgi/arvados/tree/13636-crunch-run-bindmapfuse">https://github.com/wtsi-hgi/arvados/tree/13636-crunch-run-bindmapfuse</a><br /> - <a class="external" href="https://github.com/wtsi-hgi/arvados/tree/13636-a-c-r-empty-keep-mount">https://github.com/wtsi-hgi/arvados/tree/13636-a-c-r-empty-keep-mount</a></p>
<p>Note that our crunch-run changes include disabling the `stat` that it used to perform on all keep bind mounts prior to starting docker, because it is no longer required since the `bindmapfuse` mount point itself is the only thing from the keep tree that docker bind mounts.</p>
<p>As a result, ExpressionTool steps (such as doing a matrix transpose of an array of an array of Files; <a class="external" href="https://github.com/wtsi-hgi/arvados-pipelines/blob/master/cwl/workflows/gatk-4.0.0.0-joint-calling.cwl#L49-L53">https://github.com/wtsi-hgi/arvados-pipelines/blob/master/cwl/workflows/gatk-4.0.0.0-joint-calling.cwl#L49-L53</a>) are now running very quickly because there is little cost to having a lot of keep File inputs that are not actually accessed.</p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=665012018-09-05T22:39:30ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>N.B. that matrix transpose is not a real CWL ExpressionTool but rather a CommandLineTool that we use in place of an expression tool.</p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=666442018-09-11T18:24:03ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>I'm glad you found a solution.</p>
<p>I need to study the problem a bit more, I suspect the permanent solution will be in arv-mount / crunch-run to avoid the need for bindmapfuse.</p> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=942542021-07-06T21:10:20ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> deleted (<del><i>To Be Groomed</i></del>)</li></ul> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=1117972023-02-14T22:22:22ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>60</i></li></ul> Arvados - Bug #13636: crunch-run takes a very long time for CWL steps with large numbers of File inputs - could use a new kind of mounts entry to address thishttps://dev.arvados.org/issues/13636?journal_id=1231792024-03-01T21:11:36ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> set to <i>Future</i></li></ul>