Bug #10576

[CWL] Link checking must be able to recognize keep references

Added by Peter Amstutz over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
12/05/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Subtasks

Task #10601: Implement custom schema-salad fetcherResolvedPeter Amstutz

Task #10616: Review 10576-cwl-keep-fetcherResolvedPeter Amstutz

Associated revisions

Revision efb3422f
Added by Peter Amstutz over 4 years ago

Merge branch '10576-cwl-keep-fetcher' closes #10576

Revision 386248b6 (diff)
Added by Peter Amstutz over 4 years ago

Bump schema-salad version to fix rdflib-jsonld dependency. refs #10576

Revision a619c2ce (diff)
Added by Peter Amstutz over 4 years ago

Fix CWL tests refs #10576

History

#1 Updated by Peter Amstutz over 4 years ago

  • Subject changed from [CWL] Link checking is able to recognize keep references to [CWL] Link checking must be able to recognize keep references

#2 Updated by Tom Morris over 4 years ago

  • Assigned To set to Peter Amstutz
  • Target version set to 2016-11-23 sprint

#3 Updated by Peter Amstutz over 4 years ago

  • Status changed from New to In Progress
  • Target version changed from 2016-11-23 sprint to 2016-12-14 sprint

#4 Updated by Peter Amstutz over 4 years ago

  • Story points set to 2.0

#5 Updated by Peter Amstutz over 4 years ago

  • Story points changed from 2.0 to 1.0

#6 Updated by Radhika Chippada over 4 years ago

Still reviewing, but an initial observation: 7 tests failing (after a reinstall)

======================================================================
FAIL: test_submit_arvworkflow (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 213, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 636, in test_submit_arvworkflow
    body=expect_container)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: create(body={'priority': 1, 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'arvwf:962eh-7fd4e-gkbzl62qqtfig37#main', 'mounts': {'/var/lib/cwl/workflow.json': {'kind': 'json', 'json': {'cwlVersion': 'v1.0', '$graph': [{'inputs': [{'inputBinding': {'position': 1}, 'type': 'string', 'id': '#submit_tool.cwl/x'}], 'requirements': [{'dockerPull': 'debian:8', 'class': 'DockerRequirement'}], 'outputs': [], 'baseCommand': 'cat', 'id': '#submit_tool.cwl', 'class': 'CommandLineTool'}, {'inputs': [{'type': 'string', 'id': '#main/x'}], 'steps': [{'out': [], 'run': '#submit_tool.cwl', 'id': '#main/step1', 'in': [{'source': '#main/x', 'id': '#main/step1/x'}]}], 'id': '#main', 'outputs': [], 'class': 'Workflow'}]}}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/job/cwl.input.json': {'portable_data_hash': 'e5454f8ca7d5b181e21ecd45841e3373+58/cwl.input.json', 'kind': 'collection'}, '/var/spool/cwl': {'writable': True, 'kind': 'collection'}}, 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow.json#main', '/var/lib/cwl/job/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'vcpus': 1, 'API': True, 'ram': 1073741824}})
Actual call: create(body={'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/workflow.json': {'kind': 'json', 'json': {'cwlVersion': 'v1.0', '$graph': [{'inputs': [{'inputBinding': {'position': 1}, 'type': 'string', 'id': '#submit_tool.cwl/x'}], 'requirements': [{'dockerPull': 'debian:8', 'class': 'DockerRequirement'}], 'id': '#submit_tool.cwl', 'outputs': [], 'baseCommand': 'cat', 'class': 'CommandLineTool'}, {'id': '#main', 'inputs': [{'type': 'string', 'id': '#main/x'}], 'steps': [{'in': [{'source': '#main/x', 'id': '#main/step1/x'}], 'run': '#submit_tool.cwl', 'id': '#main/step1', 'out': []}], 'class': 'Workflow', 'outputs': []}]}}, '/var/lib/cwl/cwl.input.json': {'content': {u'x': u'XxX', 'job_order': None}, 'kind': 'json'}}, 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'arvwf:962eh-7fd4e-gkbzl62qqtfig37#main', 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'API': True, 'vcpus': 1, 'ram': 1073741824}})

======================================================================
FAIL: test_submit_container (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 213, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 404, in test_submit_container
    body=expect_container)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: create(body={'name': 'submit_wf.cwl', 'runtime_constraints': {'vcpus': 1, 'API': True, 'ram': 1073741824}, 'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/job/cwl.input.json': {'portable_data_hash': 'd20d7cddd1984f105dd3702c7f125afb+60/cwl.input.json', 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}}, 'state': 'Committed', 'owner_uuid': None, 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/job/cwl.input.json'], 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'output_path': '/var/spool/cwl', 'cwd': '/var/spool/cwl'})
Actual call: create(body={'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/cwl.input.json': {'content': {'y': {'basename': '99999999999999999999999999999998+99', 'location': 'keep:99999999999999999999999999999998+99', 'class': 'Directory'}, 'x': {'basename': u'blorp.txt', 'class': 'File', 'location': u'keep:99999999999999999999999999999994+99/blorp.txt'}, 'z': {'basename': 'anonymous', 'class': 'Directory', 'listing': [{'basename': 'renamed.txt', 'class': 'File', 'location': 'keep:99999999999999999999999999999998+99/file1.txt'}]}}, 'kind': 'json'}}, 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'submit_wf.cwl', 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'API': True, 'vcpus': 1, 'ram': 1073741824}})

======================================================================
FAIL: test_submit_container_no_reuse (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 213, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 424, in test_submit_container_no_reuse
    body=expect_container)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: create(body={'name': 'submit_wf.cwl', 'runtime_constraints': {'vcpus': 1, 'API': True, 'ram': 1073741824}, 'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/job/cwl.input.json': {'portable_data_hash': 'd20d7cddd1984f105dd3702c7f125afb+60/cwl.input.json', 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}}, 'state': 'Committed', 'owner_uuid': None, 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--disable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/job/cwl.input.json'], 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'output_path': '/var/spool/cwl', 'cwd': '/var/spool/cwl'})
Actual call: create(body={'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/cwl.input.json': {'content': {'y': {'basename': '99999999999999999999999999999998+99', 'location': 'keep:99999999999999999999999999999998+99', 'class': 'Directory'}, 'x': {'basename': u'blorp.txt', 'class': 'File', 'location': u'keep:99999999999999999999999999999994+99/blorp.txt'}, 'z': {'basename': 'anonymous', 'class': 'Directory', 'listing': [{'basename': 'renamed.txt', 'class': 'File', 'location': 'keep:99999999999999999999999999999998+99/file1.txt'}]}}, 'kind': 'json'}}, 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'submit_wf.cwl', 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--disable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'API': True, 'vcpus': 1, 'ram': 1073741824}})

======================================================================
FAIL: test_submit_container_output_name (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 213, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 446, in test_submit_container_output_name
    body=expect_container)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: create(body={'name': 'submit_wf.cwl', 'runtime_constraints': {'vcpus': 1, 'API': True, 'ram': 1073741824}, 'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/job/cwl.input.json': {'portable_data_hash': 'd20d7cddd1984f105dd3702c7f125afb+60/cwl.input.json', 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}}, 'state': 'Committed', 'owner_uuid': None, 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--output-name=test_output_name', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/job/cwl.input.json'], 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'output_path': '/var/spool/cwl', 'cwd': '/var/spool/cwl'})
Actual call: create(body={'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/cwl.input.json': {'content': {'y': {'basename': '99999999999999999999999999999998+99', 'location': 'keep:99999999999999999999999999999998+99', 'class': 'Directory'}, 'x': {'basename': u'blorp.txt', 'class': 'File', 'location': u'keep:99999999999999999999999999999994+99/blorp.txt'}, 'z': {'basename': 'anonymous', 'class': 'Directory', 'listing': [{'basename': 'renamed.txt', 'class': 'File', 'location': 'keep:99999999999999999999999999999998+99/file1.txt'}]}}, 'kind': 'json'}}, 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'submit_wf.cwl', 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--output-name=test_output_name', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'API': True, 'vcpus': 1, 'ram': 1073741824}})

======================================================================
FAIL: test_submit_container_output_tags (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 213, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 468, in test_submit_container_output_tags
    body=expect_container)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: create(body={'name': 'submit_wf.cwl', 'runtime_constraints': {'vcpus': 1, 'API': True, 'ram': 1073741824}, 'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/job/cwl.input.json': {'portable_data_hash': 'd20d7cddd1984f105dd3702c7f125afb+60/cwl.input.json', 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}}, 'state': 'Committed', 'owner_uuid': None, 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--output-tags=tag0,tag1,tag2', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/job/cwl.input.json'], 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'output_path': '/var/spool/cwl', 'cwd': '/var/spool/cwl'})
Actual call: create(body={'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/cwl.input.json': {'content': {'y': {'basename': '99999999999999999999999999999998+99', 'location': 'keep:99999999999999999999999999999998+99', 'class': 'Directory'}, 'x': {'basename': u'blorp.txt', 'class': 'File', 'location': u'keep:99999999999999999999999999999994+99/blorp.txt'}, 'z': {'basename': 'anonymous', 'class': 'Directory', 'listing': [{'basename': 'renamed.txt', 'class': 'File', 'location': 'keep:99999999999999999999999999999998+99/file1.txt'}]}}, 'kind': 'json'}}, 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'submit_wf.cwl', 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--output-tags=tag0,tag1,tag2', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'API': True, 'vcpus': 1, 'ram': 1073741824}})

======================================================================
FAIL: test_submit_container_runner_ram (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 213, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 488, in test_submit_container_runner_ram
    body=expect_container)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: create(body={'name': 'submit_wf.cwl', 'runtime_constraints': {'vcpus': 1, 'API': True, 'ram': 2147483648}, 'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/job/cwl.input.json': {'portable_data_hash': 'd20d7cddd1984f105dd3702c7f125afb+60/cwl.input.json', 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}}, 'state': 'Committed', 'owner_uuid': None, 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/job/cwl.input.json'], 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'output_path': '/var/spool/cwl', 'cwd': '/var/spool/cwl'})
Actual call: create(body={'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999991+99', 'kind': 'collection'}, '/var/lib/cwl/cwl.input.json': {'content': {'y': {'basename': '99999999999999999999999999999998+99', 'location': 'keep:99999999999999999999999999999998+99', 'class': 'Directory'}, 'x': {'basename': u'blorp.txt', 'class': 'File', 'location': u'keep:99999999999999999999999999999994+99/blorp.txt'}, 'z': {'basename': 'anonymous', 'class': 'Directory', 'listing': [{'basename': 'renamed.txt', 'class': 'File', 'location': 'keep:99999999999999999999999999999998+99/file1.txt'}]}}, 'kind': 'json'}}, 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'submit_wf.cwl', 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow/submit_wf.cwl', '/var/lib/cwl/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'API': True, 'vcpus': 1, 'ram': 2147483648}})

======================================================================
FAIL: test_submit_keepref (tests.test_submit.TestSubmit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 213, in wrapped
    return func(self, stubs, *args, **kwargs)
  File "/home/radhika/arvados/sdk/cwl/tests/test_submit.py", line 552, in test_submit_keepref
    body=expect_container)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/tmp/tmp.328JRcQcJn/VENVDIR/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: create(body={'priority': 1, 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'expect_arvworkflow.cwl#main', 'mounts': {'/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999994+99', 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/job/cwl.input.json': {'portable_data_hash': 'e5454f8ca7d5b181e21ecd45841e3373+58/cwl.input.json', 'kind': 'collection'}, '/var/spool/cwl': {'writable': True, 'kind': 'collection'}}, 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow/expect_arvworkflow.cwl#main', '/var/lib/cwl/job/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'vcpus': 1, 'API': True, 'ram': 1073741824}})
Actual call: create(body={'priority': 1, 'mounts': {'/var/spool/cwl': {'writable': True, 'kind': 'collection'}, 'stdout': {'path': '/var/spool/cwl/cwl.output.json', 'kind': 'file'}, '/var/lib/cwl/workflow': {'portable_data_hash': '99999999999999999999999999999994+99', 'kind': 'collection'}, '/var/lib/cwl/cwl.input.json': {'content': {u'x': u'XxX', 'job_order': None}, 'kind': 'json'}}, 'state': 'Committed', 'owner_uuid': None, 'output_path': '/var/spool/cwl', 'name': 'expect_arvworkflow.cwl#main', 'container_image': 'arvados/jobs:ed9be4cc10c12051bdb1a65210726ab1d688108c', 'command': ['arvados-cwl-runner', '--local', '--api=containers', '--enable-reuse', '/var/lib/cwl/workflow/expect_arvworkflow.cwl#main', '/var/lib/cwl/cwl.input.json'], 'cwd': '/var/spool/cwl', 'runtime_constraints': {'API': True, 'vcpus': 1, 'ram': 1073741824}})

----------------------------------------------------------------------
Ran 38 tests in 6.730s

FAILED (failures=7)

            ********** !!!!!! sdk/cwl tests FAILED !!!!!! **********

#7 Updated by Radhika Chippada over 4 years ago

Review comments:

  • sdk/cwl tests are passing now
  • (If possible) Seems like a name such as data_fetcher might be better for kwargs["fetcher_constructor"] …
  • I can now run a workflow at command prompt using arvados-cwl-runner 9tee4-7fd4e-xox74u3j2yrmlk6 by providing the workflow uuid to a-c-r. I am wondering if it would be more user-friendly if it took the uuid as a --workflow arg
    arvados-cwl-runner  --workflow 9tee4-7fd4e-xox74u3j2yrmlk6
    
  • Needs user guide documentation update at "Writing a CWL workflow" with the command to run using a workflow uuid

#8 Updated by Radhika Chippada over 4 years ago

@ 50a3cf5df

  • arvcontainer.py
    • Could there be errors during: self.arvrunner.api.workflows().get(uuid=wfuuid).execute() ?
    • Could there be errors during: yaml.safe_load(wfrecord["definition”]) ?
  • doc: Submitting a workflow to an Arvados cluster
    • It would be nice to say somewhere that ‘--name’ can be used to set a name for the resulting CR and the default name if not provided?
  • doc: Registering a workflow to use in Workbench
    • This is not longer a true statement, correct? The registered workflow can be used to run it at command prompt as well
    • Can we add ‘--name’ to one of the examples or mention it?
  • doc: Running registered workflows at the command line
    • It appears that “Control a workflow locally” belongs in this page with the “Submit a workflow and wait for results” etc than in the “Writing a CWL workflow”? At least, it should be added in this page also to be complete.
    • Can we also have one example with no input and mention --name somewhere?
       ~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner 962eh-7fd4e-660fw86nz0w77mk 
  • doc: Arvados CWL extensions
    • It would be nice to add a bit more explanation on the top explaining what they are used for
    • string or array of string => string or array of stringS
    • It is out of scope for this context but arv:RuntimeConstraints => outputDirType with camel case and keep_cache with underscores seems weird; would be nice to stick with one or the other
  • doc: Is user/topics/running-workflow-command-line.html.textile.liquid needed? (It is zero bytes length)

#9 Updated by Radhika Chippada over 4 years ago

Mostly looks good @ 3da0fb0eef

The options in the table in user/cwl/cwl-run-options.html.textile.liquid are getting formatted. The following will resolve the issue:

-|  --basedir BASEDIR|     Base directory used to resolve relative references in the input, default to directory of input object file or current directory (if inputs piped/provided on command line).|
-|  --version|             Print version and exit|
-|  --verbose|             Default logging|
+|  ==--basedir== BASEDIR|     Base directory used to resolve relative references in the input, default to directory of input object file or current directory (if inputs piped/provided on command line).|
+|  ==--version==|             Print version and exit|
+|  ==--verbose==|             Default logging|
. . .

#11 Updated by Radhika Chippada over 4 years ago

LGTM

#12 Updated by Peter Amstutz over 4 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:efb3422f119be36e133ffbd14519c5d6c74addc8.

Also available in: Atom PDF