Bug #18024
closedcwl ValidationException: not found (Docker installed with snap)
Description
jtd@arvados-shell:~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --always-submit-runner ./bwa-mem.cwl ./bwa-mem-input.yml INFO /usr/bin/arvados-cwl-runner 2.2.1, arvados-python-client 2.2.1, cwltool 3.0.20210319143721 INFO Resolved './bwa-mem.cwl' to 'file:///home/jtd/arvados/doc/user/cwl/bwa-mem/bwa-mem.cwl' INFO Using cluster bwarv (https://workbench2.bwarv.broken.works/) INFO Using collection cache size 256 MiB INFO Uploading Docker image arvados/jobs:2.2.1 bwarv-4zz18-kfqevqgevz58lvr INFO [container bwa-mem.cwl] submitted container_request bwarv-xvhdp-1fsg4ctv3lhjui0 INFO Monitor workflow progress at https://workbench2.bwarv.broken.works/processes/bwarv-xvhdp-1fsg4ctv3lhjui0 INFO [container bwa-mem.cwl] bwarv-xvhdp-1fsg4ctv3lhjui0 is Final ERROR [container bwa-mem.cwl] (bwarv-dz642-hcqzq8cs3elsvax) error log: 2021-08-17T17:22:18.614890952Z crunch-run crunch-run 2.2.1 (go1.16.3) started 2021-08-17T17:22:18.614927351Z crunch-run Executing container 'bwarv-dz642-hcqzq8cs3elsvax' 2021-08-17T17:22:18.614938471Z crunch-run Executing on host 'arvados-crunch' 2021-08-17T17:22:18.661896150Z crunch-run Fetching Docker image from collection '0827d3de27bf4a88d8a76eea05196ba8+261' 2021-08-17T17:22:18.670814144Z crunch-run Using Docker image id 'sha256:8ca8d01e4c53b0c35c16cafe41823cb304176dd4fea364ffa09a1e35b4bf6056' 2021-08-17T17:22:18.671979940Z crunch-run Docker image is available 2021-08-17T17:22:18.672236942Z crunch-run Running [arv-mount --foreground --allow-other --read-write --crunchstat-interval=10 --file-cache 268435456 --mount-tmp tmp0 --mount-by-pdh by_id /tmp/crunch-run.bwarv-dz642-hcqzq8cs3elsvax.950326405/keep531637280] 2021-08-17T17:22:18.873344264Z crunch-run Creating Docker container 2021-08-17T17:22:18.983495638Z crunch-run Attaching container streams 2021-08-17T17:22:19.082622569Z crunch-run Starting Docker container id '10462ef46e800662960a21808d1c9f20275dc6859a31bab40c07314065fb1df5' 2021-08-17T17:22:19.433848916Z crunch-run Waiting for container to finish 2021-08-17T17:22:20.285499738Z stderr INFO /usr/bin/arvados-cwl-runner 2.2.1, arvados-python-client 2.2.1, cwltool 3.0.20210319143721 2021-08-17T17:22:20.527357117Z stderr Traceback (most recent call last): 2021-08-17T17:22:20.527357117Z stderr File "/usr/bin/arvados-cwl-runner", line 10, in <module> 2021-08-17T17:22:20.527357117Z stderr sys.exit(main(sys.argv[1:], sys.stdout, sys.stderr)) 2021-08-17T17:22:20.527357117Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 349, in main 2021-08-17T17:22:20.527357117Z stderr input_required=not (arvargs.create_workflow or arvargs.update_workflow)) 2021-08-17T17:22:20.527357117Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 974, in main 2021-08-17T17:22:20.527357117Z stderr fetcher_constructor=loadingContext.fetcher_constructor, 2021-08-17T17:22:20.527357117Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/load_tool.py", line 98, in resolve_tool_uri 2021-08-17T17:22:20.527357117Z stderr raise ValidationException("Not found: '%s'" % argsworkflow) 2021-08-17T17:22:20.527357117Z stderr schema_salad.exceptions.ValidationException: Not found: '/var/lib/cwl/workflow.json#main' 2021-08-17T17:22:21.008396630Z crunch-run Container exited with code: 1 2021-08-17T17:22:21.107253374Z crunch-run Complete ERROR Overall process status is permanentFail INFO Final output collection 5551f5d7ec57a8aa9758d14a976e87e9+57 INFO Output at https://workbench2.bwarv.broken.works/collections/5551f5d7ec57a8aa9758d14a976e87e9+57 {} WARNING Final process status is permanentFail
Updated by Peter Amstutz about 3 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz about 3 years ago
After a great deal of debugging, we established that, for some reason, Docker had been installed as a snap package, which meant it was slightly containerized, which meant that the /tmp directory visible to the Docker daemon (which was actually /tmp/snap.docker/tmp) and the /tmp directory visible to crunch-run (the real one) were not the same.
As a result, none of the bind mounts worked, resulting in these very strange errors.
Jarret will reinstall Docker using different packages and see if that fixes it.
Updated by Peter Amstutz about 3 years ago
- Assigned To set to Peter Amstutz
- Category changed from CWL to Documentation
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-08-18 sprint to 2021-09-01 sprint
Updated by Peter Amstutz about 3 years ago
- Subject changed from ValidationException: not found to cwl ValidationException: not found
Updated by Peter Amstutz about 3 years ago
- Subject changed from cwl ValidationException: not found to cwl ValidationException: not found (Docker installed with snap)
Updated by Peter Amstutz almost 3 years ago
- Target version changed from 2021-09-01 sprint to 2021-09-15 sprint
Updated by Peter Amstutz almost 3 years ago
18024-no-docker-snap @ commmit:97c90a6cd2650b0799ef99ead3570d513164c805
Add a documentation note about Docker and snap packages.
Updated by Ward Vandewege almost 3 years ago
Peter Amstutz wrote:
18024-no-docker-snap @ commmit:97c90a6cd2650b0799ef99ead3570d513164c805
Add a documentation note about Docker and snap packages.
LGTM, thanks!
Updated by Peter Amstutz almost 3 years ago
- Status changed from In Progress to Resolved