Test singularity support on a cloud cluster by running some real workflows
On a cloud cluster with singularity enabled (requires new compute image + config changes), run the the WGS workflow with small chr19 inputs, eg pirca-xvhdp-m4r8n9qmoh5ggra
#9 Updated by Ward Vandewege 4 days ago
I have built a new compute node image for ce8i5 that has the singularity binary 3.5.2 (that's an old version) in it, cf. commit: 784a3f24d37819186a52ea2c67e15e5bd8639076 on branch 17755-add-singularity-to-compute-image. That commit builds singularity from source while creating the packer image which seems suboptimal (it requires a lot of build-time dependencies, and it takes a while).
I've run the diagnostics workflow with it at ce8i5-xvhdp-el23s4fjsrp1mjb.Observations:
- the conversion of a ~240MiB docker tar file took over 5 minutes (ouch!!), about 4 of which seem to be in the invocation of `mksquashfs`
- this conversion is invisible to the user, it's part of the "loading the image" stage so it's not considered when (e.g.) workbench shows workflow step durations.
- the `hasher3` step ran on the same node as the `hasher1` step, but it repeated the singularity import step. Ouch.