Bug #16584

Error when submitting cwl using schemas to Arvados

Added by Sarah Zaranek over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CWL
Target version:
Start date:
07/15/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Get errors when submitting cwl code that validates but errors when including the following:

$schemas:
- https://schema.org/version/latest/schema.rdf
- http://edamontology.org/EDAM_1.18.owl

output here from submitting with--debug

sarah@lightning-dev1:/data-sdd/pgp/arvados-tutorial/WGS-processing$ arvados-cwl-runner --debug --no-wait --project-uuid su92l-j7d0g-9wxc78e8jisp7vm ./cwl/helper/fastqc.cwl ./yml/helper/fastqc.yml 
INFO /usr/bin/arvados-cwl-runner 2.0.3, arvados-python-client 2.0.3, cwltool 1.0.20190831161204
INFO Resolved './cwl/helper/fastqc.cwl' to 'file:///data-sdd/pgp/arvados-tutorial/WGS-processing/cwl/helper/fastqc.cwl'
DEBUG Creating collection reader for c96565a96cd428fe92bf70e97488bd80+21857
DEBUG Creating collection reader for 0ea6e3aaa6ead18a2cc4ba67a22939e9+25007
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Downloading https://schema.org/version/latest/schema.rdf (unknown size)
DEBUG {'su92l-bi6l4-giimyo9j82xqwy1': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:07.254531000Z', 'created_at': '2016-03-31T20:04:05.053456000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-giimyo9j82xqwy1', 'href': '/keep_services/su92l-bi6l4-giimyo9j82xqwy1', 'service_ssl_flag': False, 'etag': 'dzbxs5g36c7ielu285ggf1g23', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep34.su92l.arvadosapi.com', '_service_root': 'http://keep34.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}, 'su92l-bi6l4-6we9l09k8jfj2fh': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:21.686136000Z', 'created_at': '2016-03-31T20:03:46.803985000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-6we9l09k8jfj2fh', 'href': '/keep_services/su92l-bi6l4-6we9l09k8jfj2fh', 'service_ssl_flag': False, 'etag': '4de9rczhzpfp91uukzh635flp', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep31.su92l.arvadosapi.com', '_service_root': 'http://keep31.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}, 'su92l-bi6l4-xyajj22w3ipge3o': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:00.623525000Z', 'created_at': '2016-03-31T20:04:21.059892000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-xyajj22w3ipge3o', 'href': '/keep_services/su92l-bi6l4-xyajj22w3ipge3o', 'service_ssl_flag': False, 'etag': 'ae2beeyyf3ubf0v8bbjm1yhwn', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep36.su92l.arvadosapi.com', '_service_root': 'http://keep36.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}, 'su92l-bi6l4-ur0jk39m2n4jxfn': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:11.332379000Z', 'created_at': '2016-03-31T20:03:59.525087000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-ur0jk39m2n4jxfn', 'href': '/keep_services/su92l-bi6l4-ur0jk39m2n4jxfn', 'service_ssl_flag': False, 'etag': '3pe3rg5owqbzmqdgvi0pqf40h', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep33.su92l.arvadosapi.com', '_service_root': 'http://keep33.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}, 'su92l-bi6l4-rurh6qs2m3v74ft': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:31.939504000Z', 'created_at': '2016-03-31T20:03:28.962500000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-rurh6qs2m3v74ft', 'href': '/keep_services/su92l-bi6l4-rurh6qs2m3v74ft', 'service_ssl_flag': False, 'etag': '4fgqf7p2ejepnu0t8dj06d60k', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep29.su92l.arvadosapi.com', '_service_root': 'http://keep29.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}, 'su92l-bi6l4-w98vpbyai653fip': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:03.264124000Z', 'created_at': '2016-03-31T20:04:13.538273000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-w98vpbyai653fip', 'href': '/keep_services/su92l-bi6l4-w98vpbyai653fip', 'service_ssl_flag': False, 'etag': 'hg5cwm6umlzeext7mhdwtk4d', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep35.su92l.arvadosapi.com', '_service_root': 'http://keep35.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}, 'su92l-bi6l4-serzwg2c32ly6cs': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:16.184295000Z', 'created_at': '2016-03-31T20:03:53.158605000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-serzwg2c32ly6cs', 'href': '/keep_services/su92l-bi6l4-serzwg2c32ly6cs', 'service_ssl_flag': False, 'etag': 'asgjwfq8c9j5q9gyz5aerve4', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep32.su92l.arvadosapi.com', '_service_root': 'http://keep32.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}, 'su92l-bi6l4-zye3zr1wny26rwp': {'read_only': False, 'kind': 'arvados#keepService', 'modified_at': '2019-12-14T18:52:26.958244000Z', 'created_at': '2016-03-31T20:03:40.535601000Z', 'service_type': 'disk', 'modified_by_user_uuid': 'su92l-tpzed-nwst99x511vexrg', 'uuid': 'su92l-bi6l4-zye3zr1wny26rwp', 'href': '/keep_services/su92l-bi6l4-zye3zr1wny26rwp', 'service_ssl_flag': False, 'etag': 'e9jnrymd2au263zadxiy59gid', 'service_port': 25107, 'modified_by_client_uuid': 'su92l-ozdt8-c7v2lr2n2nl2enj', 'service_host': 'keep30.su92l.arvadosapi.com', '_service_root': 'http://keep30.su92l.arvadosapi.com:25107/', 'owner_uuid': 'su92l-tpzed-t2kwqqj52w2o56f'}}
DEBUG 70148178e334a98027387a143d7d8149+733488: ['http://keep32.su92l.arvadosapi.com:25107/', 'http://keep33.su92l.arvadosapi.com:25107/', 'http://keep36.su92l.arvadosapi.com:25107/', 'http://keep35.su92l.arvadosapi.com:25107/', 'http://keep31.su92l.arvadosapi.com:25107/', 'http://keep34.su92l.arvadosapi.com:25107/', 'http://keep29.su92l.arvadosapi.com:25107/', 'http://keep30.su92l.arvadosapi.com:25107/']
DEBUG Pool max threads is 3
DEBUG Request: PUT http://keep32.su92l.arvadosapi.com:25107/70148178e334a98027387a143d7d8149
DEBUG Request: PUT http://keep33.su92l.arvadosapi.com:25107/70148178e334a98027387a143d7d8149
DEBUG Request: PUT http://keep36.su92l.arvadosapi.com:25107/70148178e334a98027387a143d7d8149
INFO PUT 200: 733488 bytes in 246.9022274017334 msec (2.833 MiB/sec)
DEBUG KeepWriterThread <KeepWriterThread(Thread-3, started daemon 140613940934400)> succeeded 70148178e334a98027387a143d7d8149+733488 http://keep36.su92l.arvadosapi.com:25107/
INFO PUT 200: 733488 bytes in 252.87485122680664 msec (2.766 MiB/sec)
DEBUG KeepWriterThread <KeepWriterThread(Thread-2, started daemon 140614164645632)> succeeded 70148178e334a98027387a143d7d8149+733488 http://keep33.su92l.arvadosapi.com:25107/
INFO PUT 200: 733488 bytes in 264.2710208892822 msec (2.647 MiB/sec)
DEBUG KeepWriterThread <KeepWriterThread(Thread-1, started daemon 140614173038336)> succeeded 70148178e334a98027387a143d7d8149+733488 http://keep32.su92l.arvadosapi.com:25107/
ERROR Unhandled error:
  <HttpError 422 when requesting https://su92l.arvadosapi.com/arvados/v1/collections?alt=json&ensure_unique_name=true returned "request failed: http://localhost:8000/arvados/v1/collections: 422 Unprocessable Entity: Name cannot contain a '/' character (req-dws0rxfhspqj8jop7j4y)">
Traceback (most recent call last):
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/cwltool/main.py", line 833, in main
    logger=_logger)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/executor.py", line 579, in arv_executor
    merged_map = upload_workflow_deps(self, tool)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/runner.py", line 540, in upload_workflow_deps
    tool.visit(upload_tool_deps)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/cwltool/process.py", line 846, in visit
    op(self.tool)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/runner.py", line 533, in upload_tool_deps
    discovered_secondaryfiles=discovered_secondaryfiles)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/runner.py", line 353, in upload_dependencies
    single_collection=True)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/pathmapper.py", line 62, in __init__
    super(ArvPathMapper, self).__init__(referenced_files, input_basedir, None)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/cwltool/pathmapper.py", line 262, in __init__
    self.setup(dedup(referenced_files), basedir)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/pathmapper.py", line 179, in setup
    self.visit(srcobj, uploadfiles)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/pathmapper.py", line 102, in visit
    keepref = http_to_keep(self.arvrunner.api, self.arvrunner.project_uuid, src)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados_cwl/http.py", line 151, in http_to_keep
    c.save_new(name="Downloaded from %s" % url, owner_uuid=project_uuid, ensure_unique_name=True)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados/arvfile.py", line 469, in must_be_writable_wrapper
    return orig_func(self, *args, **kwargs)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
    return orig_func(self, *args, **kwargs)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados/retry.py", line 177, in num_retries_setter
    return orig_func(self, *args, **kwargs)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/arvados/collection.py", line 1701, in save_new
    self._remember_api_response(self._my_api().collections().create(ensure_unique_name=ensure_unique_name, body=body).execute(num_retries=num_retries))
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.5/site-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
arvados.errors.ApiError: <HttpError 422 when requesting https://su92l.arvadosapi.com/arvados/v1/collections?alt=json&ensure_unique_name=true returned "request failed: http://localhost:8000/arvados/v1/collections: 422 Unprocessable Entity: Name cannot contain a '/' character (req-dws0rxfhspqj8jop7j4y)">

Subtasks

Task #16597: review 16584-with-schemasResolvedPeter Amstutz

Associated revisions

Revision bfcd32be
Added by Peter Amstutz about 1 year ago

Merge branch '16584-with-schemas' refs #16584

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Sarah Zaranek over 1 year ago

Code can be in found in arvados-tutorial repo under WGS-Procressing/cwl/helper/fastqc.cwl and WGS-Procressing/yml/helper/fastqc.yml

Successful run of job with schema commented out can be found here:
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-3cxkohwu8wnzpm1

#2 Updated by Peter Amstutz over 1 year ago

  • Target version set to 2020-07-15
  • Assigned To set to Peter Amstutz

#4 Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress
  • Category set to CWL

#8 Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2020-07-15 to 2020-08-12 Sprint

#9 Updated by Lucas Di Pentima about 1 year ago

  • Description updated (diff)

#10 Updated by Lucas Di Pentima about 1 year ago

This LGTM, thanks!

#11 Updated by Peter Amstutz about 1 year ago

  • Status changed from In Progress to Resolved

#12 Updated by Peter Amstutz about 1 year ago

  • Release set to 25

Also available in: Atom PDF