Task #13495

Story #11162: [GA4GH] Support workflow execution API

Try to get Denis's PCAWG workflow to run

Added by Peter Amstutz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
05/11/2018
Due date:
% Done:

0%

Estimated time:
0.00 h

Description

https://dockstore.org/containers/quay.io/pancancer/pcawg_delly_workflow

This is in reference to this email from Denis Yuen trying out the WES endpoint:

Hi,

Neat, I was able to login and try this out. It works as advertised and I was also able to run a checker workflow that we had handy from the checker workflow tutorial.

I ran into some understandable teething issues with larger workflows (in retrospect, they might not be cwl portable, but the issues I'm running into are unrelated) and since you wanted a heads-up, there's some output below. But overall, its nice to see this in action!

Commands that worked:

1) As expected wes-client https://dockstore.org:8443/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-md5sum/versions/1.0.4/plain-CWL/descriptor/%2FDockstore.cwl https://raw.githubusercontent.com/briandoconnor/dockstore-tool-md5sum/master/test.json

2) Checker workflow worked wes-client https://dockstore.org:8443/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fdockstore-testing%2Fmd5sum-checker%2F_cwl_checker/versions/develop/plain-CWL/descriptor/%2Fchecker-workflow-wrapping-workflow.cwl https://raw.githubusercontent.com/dockstore-testing/md5sum-checker/master/checker-input-cwl.json

Commands that were a bit weird (these are versions of delly, these probably aren't CWL portable but I think the errors are too early for that):
1) wes-client https://dockstore.org:8443/api/ga4gh/v2/toolsquay.io%2Fpancancer%2Fpcawg_delly_workflow/versions/2.0.1-cwl1.0/plain-CWL/descriptor/%2Fdelly_docker%2FDockstore.cwl https://gist.githubusercontent.com/denis-yuen/de531d704e7521eb8bca669237522b99/raw/4600aa24ff24179f6a8c88e28c4b7995d495be13/modified_dockstore.json
Traceback (most recent call last):
File "/usr/local/bin/wes-client", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/wes_client/__init__.py", line 123, in main
"workflow_type_version": "v1.0"}).result()
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 62, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 121, in result
self.response_callbacks)
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 153, in unmarshal_response
raise_on_unexpected(incoming_response)
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 219, in raise_on_unexpected
raise make_http_exception(response=http_response)
bravado.exception.HTTPInternalServerError: 500 Internal Server Error

2) wes-client --no-wait https://dockstore.org:8443/api/ga4gh/v2/tools/quay.io%2Fpancancer%2Fpcawg_delly_workflow/versions/2.0.0-cwl1.0/plain-CWL/descriptor/%2Fdelly_docker%2FDockstore.cwl https://gist.githubusercontent.com/denis-yuen/de531d704e7521eb8bca669237522b99/raw/4600aa24ff24179f6a8c88e28c4b7995d495be13/modified_dockstore.json
Traceback (most recent call last):
File "/usr/local/bin/wes-client", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/wes_client/__init__.py", line 123, in main
"workflow_type_version": "v1.0"}).result()
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 62, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 121, in result
self.response_callbacks)
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 153, in unmarshal_response
raise_on_unexpected(incoming_response)
File "/usr/local/lib/python2.7/dist-packages/bravado/http_future.py", line 219, in raise_on_unexpected
raise make_http_exception(response=http_response)
bravado.exception.HTTPGatewayTimeout: 504 Gateway Time-out

I also ran into a few connection pool issues after delly died with the above commands which cleared up after a but, possible load issue?

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='none', port=443): Max retries exceeded with url: /ga4gh/wes/v1/swagger.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f410479c090>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

After some conversation on gitter:

that makes sense, that specific workflow will probably be replaced outright with a new workflow under icgc-argo. At that time, we'll either write them with best practices for network access, /var/spool/cwl, etc. or communicate them to the team that will be doing the work. (So definitely don't put too much work into it)

History

#1 Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz over 1 year ago

  • Status changed from In Progress to Resolved
  • Remaining (hours) set to 0.0

#3 Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
  • Estimated time set to 0.00 h

Also available in: Atom PDF