Idea #11162
closed
[GA4GH] Support workflow execution API
Description
Deploy workflow-service to demo Arvados WES support and participate in GA4GH interoperability dream challenge.
https://github.com/common-workflow-language/workflow-service
Steps:
- Test Arvados support in workflow-service
- Useful support probably requires a-c-r to support http URLs in input
- Release package
- Add workflow-service to build.list to build deb/rpms
- Talk to ops about deploying to qr1hi:
- Should have its own hostname (eg wes.qr1hi.arvadosapi.com)
- Should be lightweight, go on existing VM or a new cheap VM
- It is a Flask / WSGI app (Python) needs to be set up with an application server (passenger)
- Stateless service, minimal configuration required (just API server)
Related issues
Updated by Tom Morris about 7 years ago
- Target version set to Arvados Future Sprints
Updated by Peter Amstutz over 6 years ago
- Target version changed from Arvados Future Sprints to 2018-05-23 Sprint
Updated by Peter Amstutz over 6 years ago
- Related to Idea #9046: [Crunch2] Can issue container request to pull and import Docker images added
Updated by Peter Amstutz over 6 years ago
- Status changed from New to In Progress
- Target version changed from 2018-05-23 Sprint to 2018-06-06 Sprint
Updated by Peter Amstutz over 6 years ago
11162-wes-support adds two features:
- http downloads. Can have http/https URLs in 'location' fields. They are downloaded to keep are replaced with keep references. Respects http cache headers when determining whether to re-download a file or reuse an existing collection.
- --submit-request-uuid lets the caller provide a container_request uuid that will be used by the runner job. This is so the WES server can create an empty container request and return the uuid to the caller without having to wait for a-c-r to do all the work of getting ready to submit the request (which may include downloading docker images and files).
- I should probably add some new tests.
Running existing tests here https://ci.curoverse.com/job/developer-run-tests/725/
Updated by Lucas Di Pentima over 6 years ago
- Some
sdk/cwltests are failing - On file
sdk/cwl/arvados_cwl/http.py- Line 21: A
returnmay be missing - Line 92: I think
status_code&urlshould be passed to the exception string the other way around
- Line 21: A
- I agree on the test additions
Updated by Lucas Di Pentima over 6 years ago
Reviewing 57e511e900aa1eb175fa7f308b09516ad83492b4
Is it too complicated to write tests for the new http download & cache code? I think it would be convenient but not sure if the http requests are mockable in this case.
Also, on file sdk/cwl/arvados_cwl/arvtool.py line 16, that exception message may need to be more explicit about what's the problem?
Updated by Peter Amstutz over 6 years ago
Lucas Di Pentima wrote:
Reviewing 57e511e900aa1eb175fa7f308b09516ad83492b4
Is it too complicated to write tests for the new http download & cache code? I think it would be convenient but not sure if the http requests are mockable in this case.
Already added 11162-wes-support @ 739b1b9ec3662f988ad09509bcc933ce5c23c4e8
Also, on file
sdk/cwl/arvados_cwl/arvtool.pyline 16, that exception message may need to be more explicit about what's the problem?
Oops, that was a debugging assertion (it should never happen, it was only triggering on unit tests which didn't quite set everything up correctly.) I can take it out.
Updated by Peter Amstutz over 6 years ago
Updated by Lucas Di Pentima over 6 years ago
Updates at c9abf73e0b57739a594aa1b4803726e612cf6313 LGTM, thanks!
Updated by Peter Amstutz over 6 years ago
- Status changed from In Progress to Resolved