Story #14725

[GA4GH] Support WES 1.0

Added by Peter Amstutz 9 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
02/11/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5
Release relationship:
Auto

Description

The GA4GH has approved WES 1.0. It is a minor version tweak from the 0.3.0 development version. Our demo server at wes.qr1hi currently advertises support the version 0.3.0.

Update https://github.com/common-workflow-language/workflow-service and the demo at wes.qr1hi to advertise support for WES 1.0.


Subtasks

Task #14748: Review 14725-wesResolvedPeter Amstutz

Associated revisions

Revision 7a9e57f9
Added by Peter Amstutz 8 months ago

Merge branch '14725-wes' refs #14725

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz 9 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 9 months ago

  • Status changed from In Progress to New

#3 Updated by Peter Amstutz 9 months ago

  • Description updated (diff)

#4 Updated by Peter Amstutz 9 months ago

  • Target version set to Arvados Future Sprints

#5 Updated by Peter Amstutz 9 months ago

  • Subject changed from Support WES 1.0 to [GA4GH] Support WES 1.0

#6 Updated by Peter Amstutz 9 months ago

  • Tracker changed from Feature to Story

#7 Updated by Peter Amstutz 9 months ago

  • Story points set to 1.0

#8 Updated by Peter Amstutz 9 months ago

  • Description updated (diff)

#9 Updated by Tom Morris 9 months ago

  • Target version changed from Arvados Future Sprints to 2019-01-30 Sprint

#10 Updated by Tom Morris 9 months ago

  • Story points changed from 1.0 to 0.5

#11 Updated by Peter Amstutz 9 months ago

  • Assigned To set to Peter Amstutz

#12 Updated by Peter Amstutz 9 months ago

  • Status changed from New to In Progress

#13 Updated by Peter Amstutz 9 months ago

Figure out why this doesn't work:

curl -v -X POST -H "Authorization: Bearer <my token>" -F 'workflow_url=https://raw.githubusercontent.com/Sage-Bionetworks/workflow-interop/develop/tests/testdata/md5sum.cwl' -F 'workflow_params=@md5sum.cwl.fixed.json' -F 'workflow_type=CWL' -F 'workflow_type_version=v1.0' --trace-ascii - https://wes.qr1hi.arvadosapi.com/ga4gh/wes/v1/runs

#14 Updated by Peter Amstutz 9 months ago

  • Target version changed from 2019-01-30 Sprint to 2019-02-13 Sprint

#15 Updated by Peter Amstutz 8 months ago

14725-wes @ 171b9ac5962c21dead1d04f4c53460a81b9b9993

  • Don't give a cluster_id when updating container request record
  • Explicitly open file in binary mode when transferring from http to keep
  • Bump arvados-python-client version pin for python3 fixes

#16 Updated by Lucas Di Pentima 8 months ago

Got 4 failures when running sdk/cwl tests. Other than that, it LGTM.

======================================================================
FAIL: test_http_content_disp (tests.test_http.TestHttpToKeep)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_http.py", line 282, in test_http_content_disp
    cm.open.assert_called_with("file1.txt", "w")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: open('file1.txt', 'w')
Actual call: open('file1.txt', 'wb')

======================================================================
FAIL: test_http_expired (tests.test_http.TestHttpToKeep)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_http.py", line 191, in test_http_expired
    cm.open.assert_called_with("file1.txt", "w")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: open('file1.txt', 'w')
Actual call: open('file1.txt', 'wb')

======================================================================
FAIL: test_http_get (tests.test_http.TestHttpToKeep)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_http.py", line 63, in test_http_get
    cm.open.assert_called_with("file1.txt", "w")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: open('file1.txt', 'w')
Actual call: open('file1.txt', 'wb')

======================================================================
FAIL: test_submit_request_uuid (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_submit.py", line 318, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_submit.py", line 1397, in test_submit_request_uuid
    uuid="zzzzz-xvhdp-yyyyyyyyyyyyyyy", body=JsonDiffMatcher(stubs.expect_container_spec), cluster_id="zzzzz")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: update(body=<tests.matcher.JsonDiffMatcher object at 0x7f4d6ab651d0>, cluster_id='zzzzz', uuid='zzzzz-xvhdp-yyyyyyyyyyyyyyy')
Actual call: update(body={'cwd': '/var/spool/cwl', 'state': 'Committed', 'container_image': '999999999999999999999999999999d3+99', 'mounts': {'/var/lib/cwl/workflow.json': {'kind': 'json', 'content': {'$graph': [ordereddict([('class', 'CommandLineTool'), ('requirements', [ordereddict([('class', 'DockerRequirement'), ('dockerPull', 'debian:8'), ('http://arvados.org/cwl#dockerCollectionPDH', '999999999999999999999999999999d4+99')])]), ('inputs', [ordereddict([('id', '#submit_tool.cwl/x'), ('type', 'File'), ('default', ordereddict([('class', 'File'), ('location', 'keep:5d373e7629203ce39e7c22af98a0f881+52/blub.txt')])), ('inputBinding', ordereddict([('position', 1)]))])]), ('outputs', []), ('baseCommand', 'cat'), ('id', '#submit_tool.cwl')]), ordereddict([('class', 'Workflow'), ('inputs', [ordereddict([('id', '#main/x'), ('type', 'File')]), ordereddict([('id', '#main/y'), ('type', 'Directory')]), ordereddict([('id', '#main/z'), ('type', 'Directory')])]), ('outputs', []), ('steps', [ordereddict([('id', '#main/step1'), ('in', [ordereddict([('id', '#main/step1/x'), ('source', '#main/x')])]), ('out', []), ('run', '#submit_tool.cwl')])]), ('id', '#main')])], 'cwlVersion': 'v1.0'}}, 'stdout': {'kind': 'file', 'path': '/var/spool/cwl/cwl.output.json'}, '/var/spool/cwl': {'kind': 'collection', 'writable': True}, '/var/lib/cwl/cwl.input.json': {'kind': 'json', 'content': ordereddict([('x', ordereddict([('class', 'File'), ('location', 'keep:169f39d466a5438ac4a90e779bf750c7+53/blorp.txt'), ('size', 16), ('basename', 'blorp.txt')])), ('y', ordereddict([('class', 'Directory'), ('location', 'keep:99999999999999999999999999999998+99'), ('basename', '99999999999999999999999999999998+99')])), ('z', ordereddict([('class', 'Directory'), ('basename', 'anonymous'), ('listing', [ordereddict([('basename', 'renamed.txt'), ('class', 'File'), ('location', 'keep:99999999999999999999999999999998+99/file1.txt'), ('size', 0)])])]))])}}, 'properties': {}, 'output_path': '/var/spool/cwl', 'priority': 500, 'use_existing': True, 'secret_mounts': {}, 'name': 'submit_wf.cwl', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--no-log-timestamps', '--disable-validate', '--eval-timeout=20', '--thread-count=1', '--enable-reuse', '--collection-cache-size=256', '--debug', '--on-error=continue', '/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json'], 'runtime_constraints': {'vcpus': 1, 'ram': 1342177280, 'API': True}}, uuid='zzzzz-xvhdp-yyyyyyyyyyyyyyy')

----------------------------------------------------------------------

#17 Updated by Peter Amstutz 8 months ago

  • Status changed from In Progress to Resolved

#18 Updated by Tom Morris 8 months ago

  • Release set to 15

Also available in: Atom PDF