Story #11162

[GA4GH] Support workflow execution API

Added by Peter Amstutz over 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
05/11/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release:
Release relationship:
Auto

Description

Deploy workflow-service to demo Arvados WES support and participate in GA4GH interoperability dream challenge.

https://github.com/common-workflow-language/workflow-service

Steps:

  1. Test Arvados support in workflow-service
    1. Useful support probably requires a-c-r to support http URLs in input
  2. Release package
  3. Add workflow-service to build.list to build deb/rpms
  4. Talk to ops about deploying to qr1hi:
    1. Should have its own hostname (eg wes.qr1hi.arvadosapi.com)
    2. Should be lightweight, go on existing VM or a new cheap VM
    3. It is a Flask / WSGI app (Python) needs to be set up with an application server (passenger)
    4. Stateless service, minimal configuration required (just API server)

Subtasks

Task #13466: Test WES supportResolvedPeter Amstutz

Task #13467: Deploy on qr1hiResolved

Task #13481: Review 11162-wes-supportResolvedPeter Amstutz

Task #13495: Try to get Denis's PCAWG workflow to runResolvedPeter Amstutz


Related issues

Related to Arvados - Story #9046: [Crunch2] Can issue container request to pull and import Docker imagesNew

Associated revisions

Revision 606c29cd
Added by Peter Amstutz about 1 year ago

Merge branch '11162-wes-support' refs #11162

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Tom Morris almost 2 years ago

  • Target version set to Arvados Future Sprints

#2 Updated by Peter Amstutz over 1 year ago

  • Target version changed from Arvados Future Sprints to 2018-05-23 Sprint

#3 Updated by Peter Amstutz over 1 year ago

  • Assigned To set to Peter Amstutz

#4 Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)

#5 Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)

#6 Updated by Peter Amstutz over 1 year ago

  • Related to Story #9046: [Crunch2] Can issue container request to pull and import Docker images added

#7 Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress
  • Target version changed from 2018-05-23 Sprint to 2018-06-06 Sprint

#8 Updated by Peter Amstutz over 1 year ago

11162-wes-support adds two features:

  • http downloads. Can have http/https URLs in 'location' fields. They are downloaded to keep are replaced with keep references. Respects http cache headers when determining whether to re-download a file or reuse an existing collection.
  • --submit-request-uuid lets the caller provide a container_request uuid that will be used by the runner job. This is so the WES server can create an empty container request and return the uuid to the caller without having to wait for a-c-r to do all the work of getting ready to submit the request (which may include downloading docker images and files).
  • I should probably add some new tests.

Running existing tests here https://ci.curoverse.com/job/developer-run-tests/725/

#9 Updated by Lucas Di Pentima over 1 year ago

  • Some sdk/cwl tests are failing
  • On file sdk/cwl/arvados_cwl/http.py
    • Line 21: A return may be missing
    • Line 92: I think status_code & url should be passed to the exception string the other way around
  • I agree on the test additions

#10 Updated by Lucas Di Pentima about 1 year ago

Reviewing 57e511e900aa1eb175fa7f308b09516ad83492b4

Is it too complicated to write tests for the new http download & cache code? I think it would be convenient but not sure if the http requests are mockable in this case.
Also, on file sdk/cwl/arvados_cwl/arvtool.py line 16, that exception message may need to be more explicit about what's the problem?

#11 Updated by Peter Amstutz about 1 year ago

Lucas Di Pentima wrote:

Reviewing 57e511e900aa1eb175fa7f308b09516ad83492b4

Is it too complicated to write tests for the new http download & cache code? I think it would be convenient but not sure if the http requests are mockable in this case.

Already added 11162-wes-support @ 739b1b9ec3662f988ad09509bcc933ce5c23c4e8

Also, on file sdk/cwl/arvados_cwl/arvtool.py line 16, that exception message may need to be more explicit about what's the problem?

Oops, that was a debugging assertion (it should never happen, it was only triggering on unit tests which didn't quite set everything up correctly.) I can take it out.

#14 Updated by Peter Amstutz about 1 year ago

  • Status changed from In Progress to Resolved

#15 Updated by Tom Morris about 1 year ago

  • Release set to 13

Also available in: Atom PDF