Bug #18024

cwl ValidationException: not found (Docker installed with snap)

Added by Peter Amstutz 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Documentation
Target version:
Start date:
09/08/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

jtd@arvados-shell:~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --always-submit-runner ./bwa-mem.cwl ./bwa-mem-input.yml
INFO /usr/bin/arvados-cwl-runner 2.2.1, arvados-python-client 2.2.1, cwltool 3.0.20210319143721
INFO Resolved './bwa-mem.cwl' to 'file:///home/jtd/arvados/doc/user/cwl/bwa-mem/bwa-mem.cwl'
INFO Using cluster bwarv (https://workbench2.bwarv.broken.works/)
INFO Using collection cache size 256 MiB
INFO Uploading Docker image arvados/jobs:2.2.1
bwarv-4zz18-kfqevqgevz58lvr
INFO [container bwa-mem.cwl] submitted container_request bwarv-xvhdp-1fsg4ctv3lhjui0
INFO Monitor workflow progress at https://workbench2.bwarv.broken.works/processes/bwarv-xvhdp-1fsg4ctv3lhjui0
INFO [container bwa-mem.cwl] bwarv-xvhdp-1fsg4ctv3lhjui0 is Final
ERROR [container bwa-mem.cwl] (bwarv-dz642-hcqzq8cs3elsvax) error log:  2021-08-17T17:22:18.614890952Z crunch-run crunch-run 2.2.1 (go1.16.3) started
  2021-08-17T17:22:18.614927351Z crunch-run Executing container 'bwarv-dz642-hcqzq8cs3elsvax'
  2021-08-17T17:22:18.614938471Z crunch-run Executing on host 'arvados-crunch'
  2021-08-17T17:22:18.661896150Z crunch-run Fetching Docker image from collection '0827d3de27bf4a88d8a76eea05196ba8+261'
  2021-08-17T17:22:18.670814144Z crunch-run Using Docker image id 'sha256:8ca8d01e4c53b0c35c16cafe41823cb304176dd4fea364ffa09a1e35b4bf6056'
  2021-08-17T17:22:18.671979940Z crunch-run Docker image is available
  2021-08-17T17:22:18.672236942Z crunch-run Running [arv-mount --foreground --allow-other --read-write --crunchstat-interval=10 --file-cache 268435456 --mount-tmp tmp0 --mount-by-pdh by_id /tmp/crunch-run.bwarv-dz642-hcqzq8cs3elsvax.950326405/keep531637280]
  2021-08-17T17:22:18.873344264Z crunch-run Creating Docker container
  2021-08-17T17:22:18.983495638Z crunch-run Attaching container streams
  2021-08-17T17:22:19.082622569Z crunch-run Starting Docker container id '10462ef46e800662960a21808d1c9f20275dc6859a31bab40c07314065fb1df5'
  2021-08-17T17:22:19.433848916Z crunch-run Waiting for container to finish
  2021-08-17T17:22:20.285499738Z stderr INFO /usr/bin/arvados-cwl-runner 2.2.1, arvados-python-client 2.2.1, cwltool 3.0.20210319143721
  2021-08-17T17:22:20.527357117Z stderr Traceback (most recent call last):
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/bin/arvados-cwl-runner", line 10, in <module>
  2021-08-17T17:22:20.527357117Z stderr     sys.exit(main(sys.argv[1:], sys.stdout, sys.stderr))
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 349, in main
  2021-08-17T17:22:20.527357117Z stderr     input_required=not (arvargs.create_workflow or arvargs.update_workflow))
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 974, in main
  2021-08-17T17:22:20.527357117Z stderr     fetcher_constructor=loadingContext.fetcher_constructor,
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/load_tool.py", line 98, in resolve_tool_uri
  2021-08-17T17:22:20.527357117Z stderr     raise ValidationException("Not found: '%s'" % argsworkflow)
  2021-08-17T17:22:20.527357117Z stderr schema_salad.exceptions.ValidationException: Not found: '/var/lib/cwl/workflow.json#main'
  2021-08-17T17:22:21.008396630Z crunch-run Container exited with code: 1
  2021-08-17T17:22:21.107253374Z crunch-run Complete
ERROR Overall process status is permanentFail
INFO Final output collection 5551f5d7ec57a8aa9758d14a976e87e9+57
INFO Output at https://workbench2.bwarv.broken.works/collections/5551f5d7ec57a8aa9758d14a976e87e9+57
{}
WARNING Final process status is permanentFail

Subtasks

Task #18030: Documentation says something about not using Snap packagesResolvedPeter Amstutz

Task #18044: Review 18024-no-docker-snapResolvedPeter Amstutz

History

#1 Updated by Peter Amstutz 2 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 2 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 2 months ago

After a great deal of debugging, we established that, for some reason, Docker had been installed as a snap package, which meant it was slightly containerized, which meant that the /tmp directory visible to the Docker daemon (which was actually /tmp/snap.docker/tmp) and the /tmp directory visible to crunch-run (the real one) were not the same.

As a result, none of the bind mounts worked, resulting in these very strange errors.

Jarret will reinstall Docker using different packages and see if that fixes it.

#4 Updated by Peter Amstutz 2 months ago

  • Assigned To set to Peter Amstutz
  • Category changed from CWL to Documentation

#5 Updated by Peter Amstutz 2 months ago

  • Target version changed from 2021-08-18 sprint to 2021-09-01 sprint

#6 Updated by Peter Amstutz 2 months ago

  • Subject changed from ValidationException: not found to cwl ValidationException: not found

#7 Updated by Peter Amstutz 2 months ago

  • Subject changed from cwl ValidationException: not found to cwl ValidationException: not found (Docker installed with snap)

#8 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2021-09-01 sprint to 2021-09-15 sprint

#9 Updated by Peter Amstutz about 2 months ago

18024-no-docker-snap @ commmit:97c90a6cd2650b0799ef99ead3570d513164c805

Add a documentation note about Docker and snap packages.

#10 Updated by Ward Vandewege about 2 months ago

Peter Amstutz wrote:

18024-no-docker-snap @ commmit:97c90a6cd2650b0799ef99ead3570d513164c805

Add a documentation note about Docker and snap packages.

LGTM, thanks!

#11 Updated by Peter Amstutz about 2 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF