Arvados

Figure out why this doesn't work:

curl -v -X POST -H "Authorization: Bearer <my token>" -F 'workflow_url=https://raw.githubusercontent.com/Sage-Bionetworks/workflow-interop/develop/tests/testdata/md5sum.cwl' -F 'workflow_params=@md5sum.cwl.fixed.json' -F 'workflow_type=CWL' -F 'workflow_type_version=v1.0' --trace-ascii - https://wes.qr1hi.arvadosapi.com/ga4gh/wes/v1/runs

14725-wes @ 171b9ac5962c21dead1d04f4c53460a81b9b9993

• Don't give a cluster_id when updating container request record

• Explicitly open file in binary mode when transferring from http to keep

• Bump arvados-python-client version pin for python3 fixes

Got 4 failures when running sdk/cwl tests. Other than that, it LGTM.

======================================================================
FAIL: test_http_content_disp (tests.test_http.TestHttpToKeep)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_http.py", line 282, in test_http_content_disp
    cm.open.assert_called_with("file1.txt", "w")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: open('file1.txt', 'w')
Actual call: open('file1.txt', 'wb')

======================================================================
FAIL: test_http_expired (tests.test_http.TestHttpToKeep)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_http.py", line 191, in test_http_expired
    cm.open.assert_called_with("file1.txt", "w")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: open('file1.txt', 'w')
Actual call: open('file1.txt', 'wb')

======================================================================
FAIL: test_http_get (tests.test_http.TestHttpToKeep)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_http.py", line 63, in test_http_get
    cm.open.assert_called_with("file1.txt", "w")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: open('file1.txt', 'w')
Actual call: open('file1.txt', 'wb')

======================================================================
FAIL: test_submit_request_uuid (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_submit.py", line 318, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/lucas/arvados_local/sdk/cwl/tests/test_submit.py", line 1397, in test_submit_request_uuid
    uuid="zzzzz-xvhdp-yyyyyyyyyyyyyyy", body=JsonDiffMatcher(stubs.expect_container_spec), cluster_id="zzzzz")
  File "/home/lucas/tmp/VENV3DIR/lib/python3.5/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 3, in raise_from
AssertionError: Expected call: update(body=<tests.matcher.JsonDiffMatcher object at 0x7f4d6ab651d0>, cluster_id='zzzzz', uuid='zzzzz-xvhdp-yyyyyyyyyyyyyyy')
Actual call: update(body={'cwd': '/var/spool/cwl', 'state': 'Committed', 'container_image': '999999999999999999999999999999d3+99', 'mounts': {'/var/lib/cwl/workflow.json': {'kind': 'json', 'content': {'$graph': [ordereddict([('class', 'CommandLineTool'), ('requirements', [ordereddict([('class', 'DockerRequirement'), ('dockerPull', 'debian:8'), ('http://arvados.org/cwl#dockerCollectionPDH', '999999999999999999999999999999d4+99')])]), ('inputs', [ordereddict([('id', '#submit_tool.cwl/x'), ('type', 'File'), ('default', ordereddict([('class', 'File'), ('location', 'keep:5d373e7629203ce39e7c22af98a0f881+52/blub.txt')])), ('inputBinding', ordereddict([('position', 1)]))])]), ('outputs', []), ('baseCommand', 'cat'), ('id', '#submit_tool.cwl')]), ordereddict([('class', 'Workflow'), ('inputs', [ordereddict([('id', '#main/x'), ('type', 'File')]), ordereddict([('id', '#main/y'), ('type', 'Directory')]), ordereddict([('id', '#main/z'), ('type', 'Directory')])]), ('outputs', []), ('steps', [ordereddict([('id', '#main/step1'), ('in', [ordereddict([('id', '#main/step1/x'), ('source', '#main/x')])]), ('out', []), ('run', '#submit_tool.cwl')])]), ('id', '#main')])], 'cwlVersion': 'v1.0'}}, 'stdout': {'kind': 'file', 'path': '/var/spool/cwl/cwl.output.json'}, '/var/spool/cwl': {'kind': 'collection', 'writable': True}, '/var/lib/cwl/cwl.input.json': {'kind': 'json', 'content': ordereddict([('x', ordereddict([('class', 'File'), ('location', 'keep:169f39d466a5438ac4a90e779bf750c7+53/blorp.txt'), ('size', 16), ('basename', 'blorp.txt')])), ('y', ordereddict([('class', 'Directory'), ('location', 'keep:99999999999999999999999999999998+99'), ('basename', '99999999999999999999999999999998+99')])), ('z', ordereddict([('class', 'Directory'), ('basename', 'anonymous'), ('listing', [ordereddict([('basename', 'renamed.txt'), ('class', 'File'), ('location', 'keep:99999999999999999999999999999998+99/file1.txt'), ('size', 0)])])]))])}}, 'properties': {}, 'output_path': '/var/spool/cwl', 'priority': 500, 'use_existing': True, 'secret_mounts': {}, 'name': 'submit_wf.cwl', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--no-log-timestamps', '--disable-validate', '--eval-timeout=20', '--thread-count=1', '--enable-reuse', '--collection-cache-size=256', '--debug', '--on-error=continue', '/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json'], 'runtime_constraints': {'vcpus': 1, 'ram': 1342177280, 'API': True}}, uuid='zzzzz-xvhdp-yyyyyyyyyyyyyyy')

----------------------------------------------------------------------