Idea #11162
closed
[GA4GH] Support workflow execution API
Added by Peter Amstutz over 7 years ago.
Updated over 6 years ago.
Release relationship:
Auto
Description
Deploy workflow-service to demo Arvados WES support and participate in GA4GH interoperability dream challenge.
https://github.com/common-workflow-language/workflow-service
Steps:
- Test Arvados support in workflow-service
- Useful support probably requires a-c-r to support http URLs in input
- Release package
- Add workflow-service to build.list to build deb/rpms
- Talk to ops about deploying to qr1hi:
- Should have its own hostname (eg wes.qr1hi.arvadosapi.com)
- Should be lightweight, go on existing VM or a new cheap VM
- It is a Flask / WSGI app (Python) needs to be set up with an application server (passenger)
- Stateless service, minimal configuration required (just API server)
- Target version set to Arvados Future Sprints
- Target version changed from Arvados Future Sprints to 2018-05-23 Sprint
- Assigned To set to Peter Amstutz
- Description updated (diff)
- Description updated (diff)
- Related to Idea #9046: [Crunch2] Can issue container request to pull and import Docker images added
- Status changed from New to In Progress
- Target version changed from 2018-05-23 Sprint to 2018-06-06 Sprint
11162-wes-support adds two features:
- http downloads. Can have http/https URLs in 'location' fields. They are downloaded to keep are replaced with keep references. Respects http cache headers when determining whether to re-download a file or reuse an existing collection.
- --submit-request-uuid lets the caller provide a container_request uuid that will be used by the runner job. This is so the WES server can create an empty container request and return the uuid to the caller without having to wait for a-c-r to do all the work of getting ready to submit the request (which may include downloading docker images and files).
- I should probably add some new tests.
Running existing tests here https://ci.curoverse.com/job/developer-run-tests/725/
- Some
sdk/cwl tests are failing
- On file
sdk/cwl/arvados_cwl/http.py
- Line 21: A
return may be missing
- Line 92: I think
status_code & url should be passed to the exception string the other way around
- I agree on the test additions
Reviewing 57e511e900aa1eb175fa7f308b09516ad83492b4
Is it too complicated to write tests for the new http download & cache code? I think it would be convenient but not sure if the http requests are mockable in this case.
Also, on file sdk/cwl/arvados_cwl/arvtool.py line 16, that exception message may need to be more explicit about what's the problem?
Lucas Di Pentima wrote:
Reviewing 57e511e900aa1eb175fa7f308b09516ad83492b4
Is it too complicated to write tests for the new http download & cache code? I think it would be convenient but not sure if the http requests are mockable in this case.
Already added 11162-wes-support @ 739b1b9ec3662f988ad09509bcc933ce5c23c4e8
Also, on file sdk/cwl/arvados_cwl/arvtool.py line 16, that exception message may need to be more explicit about what's the problem?
Oops, that was a debugging assertion (it should never happen, it was only triggering on unit tests which didn't quite set everything up correctly.) I can take it out.
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by Tom Morris 
Updated by 
Updated by