Bug #11257

[CWL] Fails with $schemas referencing remote resources.

Added by Peter Amstutz 7 months ago. Updated about 1 month ago.

Status:NewStart date:
Priority:NormalDue date:
Assignee:-% Done:


Target version:Arvados Future Sprints
Story points-
Velocity based estimate-


Workflows with external http resources listed in $schemas get into trouble with the code to find and upload local dependencies. It should only attempt to upload identifiers with a file:// scheme.

$ arvados-cwl-runner --debug Dockstore.cwl  test.json 
2017-03-15 14:47:25 cwltool INFO: /home/peter/work/scripts/venv/bin/arvados-cwl-runner d363323bfea33aee9182e71fd93387b0c4f944a2 1.0.20170314191734, arvados-python-client 0.1.20170309224426, cwltool 1.0.20170213175853
2017-03-15 14:47:25 cwltool INFO: Resolved 'Dockstore.cwl' to 'file:///home/peter/work/tool-challenge/dockstore-tool-md5sum/Dockstore.cwl'
2017-03-15 14:47:26 cwltool ERROR: Workflow error, try again with --debug for more information:
Input file path 'http://xmlns.com/foaf/spec/20140114.rdf' is invalid
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/cwltool/main.py", line 728, in main
  File "build/bdist.linux-x86_64/egg/arvados_cwl/__init__.py", line 348, in arv_executor
    upload_workflow_deps(self, tool)
  File "build/bdist.linux-x86_64/egg/arvados_cwl/runner.py", line 212, in upload_workflow_deps
  File "build/bdist.linux-x86_64/egg/cwltool/process.py", line 599, in visit
  File "build/bdist.linux-x86_64/egg/arvados_cwl/runner.py", line 209, in upload_tool_deps
  File "build/bdist.linux-x86_64/egg/arvados_cwl/runner.py", line 108, in upload_dependencies
  File "build/bdist.linux-x86_64/egg/arvados_cwl/pathmapper.py", line 29, in __init__
    super(ArvPathMapper, self).__init__(referenced_files, input_basedir, None)
  File "build/bdist.linux-x86_64/egg/cwltool/pathmapper.py", line 150, in __init__
    self.setup(dedup(referenced_files), basedir)
  File "build/bdist.linux-x86_64/egg/arvados_cwl/pathmapper.py", line 96, in setup
    self.visit(srcobj, uploadfiles)
  File "build/bdist.linux-x86_64/egg/arvados_cwl/pathmapper.py", line 56, in visit
    raise WorkflowException("Input file path '%s' is invalid" % st)
  File "build/bdist.linux-x86_64/egg/schema_salad/sourceline.py", line 146, in __exit__
    raise self.makeError(unicode(exc_value))
WorkflowException: Input file path 'http://xmlns.com/foaf/spec/20140114.rdf' is invalid


#1 Updated by Peter Amstutz 7 months ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 7 months ago

Fix pending in #10401 (as part of larger pathmapper refactor)

#3 Updated by Tom Morris about 1 month ago

  • Target version set to Arvados Future Sprints

Also available in: Atom PDF