Bug #18260
Updated by Ward Vandewege over 2 years ago
On 9tee4 (configured to use LSF + singularity), test 35 from the Arvados CWL test suite fails consistently (e.g. https://ci.arvados.org/view/Arvados%20build%20pipeline/job/run-cwl-test-9tee4/1251/console):
<pre>
15:02:51 Test 35 failed: arvados-cwl-runner --disable-reuse --compute-checksum --api=containers --outdir=/tmp/tmp36moghbh --quiet 16169-no-listing-hint.cwl listing-job.yml
15:02:51 Test cwltool:LoadListingRequirement propagation
15:02:51 Returned non-zero
15:02:51 ERROR [container 16169-no-listing-hint.cwl] (9tee4-dz642-rw8f8gnf0bbtbf9) error log:
15:02:51
15:02:51 2021-10-08T19:02:31.198703294Z crunch-run Not starting a gateway server (GatewayAuthSecret was not provided by dispatcher)
15:02:51 2021-10-08T19:02:31.198978468Z crunch-run crunch-run 2.3.0~dev20211008165008 (go1.17.1) started
15:02:51 2021-10-08T19:02:31.199000348Z crunch-run Executing container '9tee4-dz642-rw8f8gnf0bbtbf9' using singularity runtime
15:02:51 2021-10-08T19:02:31.199034119Z crunch-run Executing on host 'compute0.9tee4.arvadosapi.com'
15:02:51 2021-10-08T19:02:31.316882478Z crunch-run container token "v2/9tee4-gj3su-nzdweqs16uwdd1c/17n5aocpl1h49w3fdpt6itws6m3jzt2iacgk3hed3kl676czhj/9tee4-dz642-rw8f8gnf0bbtbf9"
15:02:51 2021-10-08T19:02:31.317415885Z crunch-run Running [arv-mount --foreground --allow-other --read-write --storage-classes default --crunchstat-interval=10 --file-cache 268435456 --mount-tmp tmp0 --mount-by-pdh by_id --mount-by-id by_uuid /tmp/crunch-run.9tee4-dz642-rw8f8gnf0bbtbf9.754694554/keep3163623830]
15:02:51 2021-10-08T19:02:31.956248212Z crunch-run Fetching Docker image from collection 'a2498feb8afadffbde268ab00b07fa4b+261'
15:02:51 2021-10-08T19:02:32.048897698Z crunch-run Using Docker image id "sha256:e520e478c5777b695294faa6cb442edeb63112ae3f66b38ea3520c8e755c03c8"
15:02:51 2021-10-08T19:02:32.048970278Z crunch-run Loading Docker image from keep
15:02:51 2021-10-08T19:02:32.978977970Z crunch-run Starting container
15:02:51 2021-10-08T19:02:32.980308098Z crunch-run Waiting for container to finish
15:02:51 2021-10-08T19:02:41.930650770Z stderr INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20211006204117, arvados-python-client 2.3.0.dev20211006204117, cwltool 3.1.20210922203925
15:02:51 2021-10-08T19:02:41.949220287Z stderr INFO Resolved '/var/lib/cwl/workflow.json#main' to 'file:///var/lib/cwl/workflow.json#main'
15:02:51 2021-10-08T19:02:46.251818792Z stderr ERROR Tool definition failed initialization:
15:02:51 2021-10-08T19:02:46.251818792Z stderr Tool definition file:///var/lib/cwl/workflow.json#16169-step.cwl failed validation:
15:02:51 2021-10-08T19:02:46.251818792Z stderr ../../lib/cwl/workflow.json:1:12: checking object `../../lib/cwl/workflow.json#main`
15:02:51 2021-10-08T19:02:46.251818792Z stderr ../../lib/cwl/workflow.json:1:170: checking field `requirements`
15:02:51 2021-10-08T19:02:46.251818792Z stderr ../../lib/cwl/workflow.json:1:186: checking item
15:02:51 2021-10-08T19:02:46.251818792Z stderr ../../lib/cwl/workflow.json:1:187: Field `class` contains undefined reference to
15:02:51 2021-10-08T19:02:46.251818792Z stderr `http://commonwl.org/cwltool#LoadListingRequirement`
15:02:51 2021-10-08T19:02:46.832474311Z crunch-run Complete
15:02:51 ERROR Overall process status is permanentFail
15:02:51 WARNING Final process status is permanentFail
</pre>
On tordo (a-d-c + singularity) this test passes without problems.