Project

General

Profile

Actions

Bug #18024

closed

cwl ValidationException: not found (Docker installed with snap)

Added by Peter Amstutz over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Documentation
Target version:
Story points:
-
Release relationship:
Auto

Description

jtd@arvados-shell:~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --always-submit-runner ./bwa-mem.cwl ./bwa-mem-input.yml
INFO /usr/bin/arvados-cwl-runner 2.2.1, arvados-python-client 2.2.1, cwltool 3.0.20210319143721
INFO Resolved './bwa-mem.cwl' to 'file:///home/jtd/arvados/doc/user/cwl/bwa-mem/bwa-mem.cwl'
INFO Using cluster bwarv (https://workbench2.bwarv.broken.works/)
INFO Using collection cache size 256 MiB
INFO Uploading Docker image arvados/jobs:2.2.1
bwarv-4zz18-kfqevqgevz58lvr
INFO [container bwa-mem.cwl] submitted container_request bwarv-xvhdp-1fsg4ctv3lhjui0
INFO Monitor workflow progress at https://workbench2.bwarv.broken.works/processes/bwarv-xvhdp-1fsg4ctv3lhjui0
INFO [container bwa-mem.cwl] bwarv-xvhdp-1fsg4ctv3lhjui0 is Final
ERROR [container bwa-mem.cwl] (bwarv-dz642-hcqzq8cs3elsvax) error log:  2021-08-17T17:22:18.614890952Z crunch-run crunch-run 2.2.1 (go1.16.3) started
  2021-08-17T17:22:18.614927351Z crunch-run Executing container 'bwarv-dz642-hcqzq8cs3elsvax'
  2021-08-17T17:22:18.614938471Z crunch-run Executing on host 'arvados-crunch'
  2021-08-17T17:22:18.661896150Z crunch-run Fetching Docker image from collection '0827d3de27bf4a88d8a76eea05196ba8+261'
  2021-08-17T17:22:18.670814144Z crunch-run Using Docker image id 'sha256:8ca8d01e4c53b0c35c16cafe41823cb304176dd4fea364ffa09a1e35b4bf6056'
  2021-08-17T17:22:18.671979940Z crunch-run Docker image is available
  2021-08-17T17:22:18.672236942Z crunch-run Running [arv-mount --foreground --allow-other --read-write --crunchstat-interval=10 --file-cache 268435456 --mount-tmp tmp0 --mount-by-pdh by_id /tmp/crunch-run.bwarv-dz642-hcqzq8cs3elsvax.950326405/keep531637280]
  2021-08-17T17:22:18.873344264Z crunch-run Creating Docker container
  2021-08-17T17:22:18.983495638Z crunch-run Attaching container streams
  2021-08-17T17:22:19.082622569Z crunch-run Starting Docker container id '10462ef46e800662960a21808d1c9f20275dc6859a31bab40c07314065fb1df5'
  2021-08-17T17:22:19.433848916Z crunch-run Waiting for container to finish
  2021-08-17T17:22:20.285499738Z stderr INFO /usr/bin/arvados-cwl-runner 2.2.1, arvados-python-client 2.2.1, cwltool 3.0.20210319143721
  2021-08-17T17:22:20.527357117Z stderr Traceback (most recent call last):
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/bin/arvados-cwl-runner", line 10, in <module>
  2021-08-17T17:22:20.527357117Z stderr     sys.exit(main(sys.argv[1:], sys.stdout, sys.stderr))
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 349, in main
  2021-08-17T17:22:20.527357117Z stderr     input_required=not (arvargs.create_workflow or arvargs.update_workflow))
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 974, in main
  2021-08-17T17:22:20.527357117Z stderr     fetcher_constructor=loadingContext.fetcher_constructor,
  2021-08-17T17:22:20.527357117Z stderr   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/load_tool.py", line 98, in resolve_tool_uri
  2021-08-17T17:22:20.527357117Z stderr     raise ValidationException("Not found: '%s'" % argsworkflow)
  2021-08-17T17:22:20.527357117Z stderr schema_salad.exceptions.ValidationException: Not found: '/var/lib/cwl/workflow.json#main'
  2021-08-17T17:22:21.008396630Z crunch-run Container exited with code: 1
  2021-08-17T17:22:21.107253374Z crunch-run Complete
ERROR Overall process status is permanentFail
INFO Final output collection 5551f5d7ec57a8aa9758d14a976e87e9+57
INFO Output at https://workbench2.bwarv.broken.works/collections/5551f5d7ec57a8aa9758d14a976e87e9+57
{}
WARNING Final process status is permanentFail

Subtasks 2 (0 open2 closed)

Task #18030: Documentation says something about not using Snap packagesResolvedPeter Amstutz09/08/2021Actions
Task #18044: Review 18024-no-docker-snapResolvedPeter Amstutz09/08/2021Actions
Actions #1

Updated by Peter Amstutz over 3 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz over 3 years ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz over 3 years ago

After a great deal of debugging, we established that, for some reason, Docker had been installed as a snap package, which meant it was slightly containerized, which meant that the /tmp directory visible to the Docker daemon (which was actually /tmp/snap.docker/tmp) and the /tmp directory visible to crunch-run (the real one) were not the same.

As a result, none of the bind mounts worked, resulting in these very strange errors.

Jarret will reinstall Docker using different packages and see if that fixes it.

Actions #4

Updated by Peter Amstutz over 3 years ago

  • Assigned To set to Peter Amstutz
  • Category changed from CWL to Documentation
Actions #5

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-08-18 sprint to 2021-09-01 sprint
Actions #6

Updated by Peter Amstutz over 3 years ago

  • Subject changed from ValidationException: not found to cwl ValidationException: not found
Actions #7

Updated by Peter Amstutz over 3 years ago

  • Subject changed from cwl ValidationException: not found to cwl ValidationException: not found (Docker installed with snap)
Actions #8

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-09-01 sprint to 2021-09-15 sprint
Actions #9

Updated by Peter Amstutz about 3 years ago

18024-no-docker-snap @ commmit:97c90a6cd2650b0799ef99ead3570d513164c805

Add a documentation note about Docker and snap packages.

Actions #10

Updated by Ward Vandewege about 3 years ago

Peter Amstutz wrote:

18024-no-docker-snap @ commmit:97c90a6cd2650b0799ef99ead3570d513164c805

Add a documentation note about Docker and snap packages.

LGTM, thanks!

Actions #11

Updated by Peter Amstutz about 3 years ago

  • Status changed from In Progress to Resolved
Actions #12

Updated by Peter Amstutz about 3 years ago

  • Release set to 42
Actions

Also available in: Atom PDF