Project

General

Profile

Actions
Copy link

Bug #12347

closed

[CWL] reuse hints appear to be broken

Added by Ward Vandewege over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Clegg
Category:
-
Target version:
2017-10-11 Sprint
Story points:
-

Description

Using

sdk/cwl/tests/noreuse.cwl

from the arvados tree, I've tested on 9tee4 (head, crunchv2), tb05z (0.1.20170925193954.68bdf4c, crunchv1), and qr1hi (0.1.20170817171636.0b48006, crunchv1).

wardtest@shell.9tee4:~$ cwl-runner --debug noreuse.cwl 
2017-09-27 14:51:11 cwltool INFO: /usr/bin/cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 14:51:11 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 14:51:13 cwltool DEBUG: Parsed job order from command line: {
    "id": "noreuse.cwl" 
}
2017-09-27 14:51:13 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523873.98
2017-09-27 14:51:13 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523873.98 0.000186204910278
2017-09-27 14:51:13 arvados.cwl-runner.metrics DEBUG: ENTER run 1506523873.98
2017-09-27 14:51:14 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-ye3hso3hnsw5g10
2017-09-27 14:51:14 arvados.keep DEBUG: {u'9tee4-bi6l4-uwgvqzzw4z97qca': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-uwgvqzzw4z97qca'), (u'kind', u'arvados#keepService'), (u'etag', u'c65xiys117q01qz9s7zvjt66i'), (u'uuid', u'9tee4-bi6l4-uwgvqzzw4z97qca'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:19:29.655913000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:21.044273000Z'), (u'service_host', u'keep0.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep0.9tee4.arvadosapi.com:25107/')]), u'9tee4-bi6l4-1fum8ggyzc88g87': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-1fum8ggyzc88g87'), (u'kind', u'arvados#keepService'), (u'etag', u'dn6g7rkwx5gpwq0h053wyquax'), (u'uuid', u'9tee4-bi6l4-1fum8ggyzc88g87'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:20:27.578073000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:43.978890000Z'), (u'service_host', u'keep1.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep1.9tee4.arvadosapi.com:25107/')])}
2017-09-27 14:51:14 arvados.keep DEBUG: 99914b932bd37a50b983c5e7c90ae93b+2+A61f031b2c8357576abfe330f1dfb9632e618f99d@59de2fe2: ['http://keep1.9tee4.arvadosapi.com:25107/', 'http://keep0.9tee4.arvadosapi.com:25107/']
2017-09-27 14:51:14 arvados.keep DEBUG: Request: GET http://keep1.9tee4.arvadosapi.com:25107/99914b932bd37a50b983c5e7c90ae93b+2+A61f031b2c8357576abfe330f1dfb9632e618f99d@59de2fe2
2017-09-27 14:51:14 arvados.keep INFO: GET 200: 2 bytes in 88.7131690979 msec (0.000 MiB/sec)
2017-09-27 14:51:14 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:51:14 arvados.cwl-runner.metrics DEBUG: EXIT run 1506523874.84 0.865196943283
2017-09-27 14:51:14 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523874.84
2017-09-27 14:51:14 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523874.84 6.19888305664e-05
2017-09-27 14:51:14 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:51:14 cwltool INFO: Final process status is success

wardtest@shell.9tee4:~$ cwl-runner --debug noreuse.cwl 
2017-09-27 14:52:38 cwltool INFO: /usr/bin/cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 14:52:38 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 14:52:39 cwltool DEBUG: Parsed job order from command line: {
    "id": "noreuse.cwl" 
}
2017-09-27 14:52:40 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523960.51
2017-09-27 14:52:40 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523960.51 0.000166893005371
2017-09-27 14:52:40 arvados.cwl-runner.metrics DEBUG: ENTER run 1506523960.51
2017-09-27 14:52:40 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-y7f9m3nzjdldoso
2017-09-27 14:52:41 arvados.keep DEBUG: {u'9tee4-bi6l4-uwgvqzzw4z97qca': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-uwgvqzzw4z97qca'), (u'kind', u'arvados#keepService'), (u'etag', u'c65xiys117q01qz9s7zvjt66i'), (u'uuid', u'9tee4-bi6l4-uwgvqzzw4z97qca'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:19:29.655913000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:21.044273000Z'), (u'service_host', u'keep0.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep0.9tee4.arvadosapi.com:25107/')]), u'9tee4-bi6l4-1fum8ggyzc88g87': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-1fum8ggyzc88g87'), (u'kind', u'arvados#keepService'), (u'etag', u'dn6g7rkwx5gpwq0h053wyquax'), (u'uuid', u'9tee4-bi6l4-1fum8ggyzc88g87'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:20:27.578073000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:43.978890000Z'), (u'service_host', u'keep1.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep1.9tee4.arvadosapi.com:25107/')])}
2017-09-27 14:52:41 arvados.keep DEBUG: 99914b932bd37a50b983c5e7c90ae93b+2+A5804900581ce783388a56366a5d2908f238b03c4@59de3039: ['http://keep1.9tee4.arvadosapi.com:25107/', 'http://keep0.9tee4.arvadosapi.com:25107/']
2017-09-27 14:52:41 arvados.keep DEBUG: Request: GET http://keep1.9tee4.arvadosapi.com:25107/99914b932bd37a50b983c5e7c90ae93b+2+A5804900581ce783388a56366a5d2908f238b03c4@59de3039
2017-09-27 14:52:41 arvados.keep INFO: GET 200: 2 bytes in 25.9730815887 msec (0.000 MiB/sec)
2017-09-27 14:52:41 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:52:41 arvados.cwl-runner.metrics DEBUG: EXIT run 1506523961.53 1.02316498756
2017-09-27 14:52:41 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523961.53
2017-09-27 14:52:41 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523961.53 7.60555267334e-05
2017-09-27 14:52:41 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:52:41 cwltool INFO: Final process status is success

Even adding --disable-reuse doesn't help:

wardtest@shell.9tee4:~$ cwl-runner --disable-reuse --debug noreuse.cwl 
2017-09-27 15:16:13 cwltool INFO: /usr/bin/cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 15:16:13 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 15:16:15 cwltool DEBUG: Parsed job order from command line: {
    "id": "noreuse.cwl" 
}
2017-09-27 15:16:15 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506525375.98
2017-09-27 15:16:15 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506525375.98 0.000164031982422
2017-09-27 15:16:15 arvados.cwl-runner.metrics DEBUG: ENTER run 1506525375.98
2017-09-27 15:16:16 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-ktk14izbiujbnkc
2017-09-27 15:16:16 arvados.keep DEBUG: {u'9tee4-bi6l4-uwgvqzzw4z97qca': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-uwgvqzzw4z97qca'), (u'kind', u'arvados#keepService'), (u'etag', u'c65xiys117q01qz9s7zvjt66i'), (u'uuid', u'9tee4-bi6l4-uwgvqzzw4z97qca'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:19:29.655913000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:21.044273000Z'), (u'service_host', u'keep0.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep0.9tee4.arvadosapi.com:25107/')]), u'9tee4-bi6l4-1fum8ggyzc88g87': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-1fum8ggyzc88g87'), (u'kind', u'arvados#keepService'), (u'etag', u'dn6g7rkwx5gpwq0h053wyquax'), (u'uuid', u'9tee4-bi6l4-1fum8ggyzc88g87'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:20:27.578073000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:43.978890000Z'), (u'service_host', u'keep1.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep1.9tee4.arvadosapi.com:25107/')])}
2017-09-27 15:16:16 arvados.keep DEBUG: 99914b932bd37a50b983c5e7c90ae93b+2+Ab6bdb8008dc52ee665f3e33681a4a24d15c632df@59de35c0: ['http://keep1.9tee4.arvadosapi.com:25107/', 'http://keep0.9tee4.arvadosapi.com:25107/']
2017-09-27 15:16:16 arvados.keep DEBUG: Request: GET http://keep1.9tee4.arvadosapi.com:25107/99914b932bd37a50b983c5e7c90ae93b+2+Ab6bdb8008dc52ee665f3e33681a4a24d15c632df@59de35c0
2017-09-27 15:16:16 arvados.keep INFO: GET 200: 2 bytes in 25.8030891418 msec (0.000 MiB/sec)
2017-09-27 15:16:16 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 15:16:16 arvados.cwl-runner.metrics DEBUG: EXIT run 1506525376.86 0.883622169495
2017-09-27 15:16:16 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506525376.86
2017-09-27 15:16:16 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506525376.86 6.29425048828e-05
2017-09-27 15:16:16 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 15:16:16 cwltool INFO: Final process status is success

On qr1hi:

wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 14:50:44 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 14:50:44 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 14:50:47 arvados.cwl-runner INFO: Uploaded to qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 14:51:01 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-ailajv1w1bv54s5
2017-09-27 14:53:17 arvados.cwl-runner INFO: [job noreuse.cwl] qr1hi-8i9sb-yfp2iy4c5lztlvc is Running
2017-09-27 14:59:03 arvados.cwl-runner INFO: [job noreuse.cwl] qr1hi-8i9sb-yfp2iy4c5lztlvc is Complete
2017-09-27 14:59:04 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:59:04 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:59:04 cwltool INFO: Final process status is success
wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 14:59:11 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 14:59:11 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 14:59:14 arvados.cwl-runner INFO: Using collection qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 14:59:15 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-xqiqoz7z58xbc0q
2017-09-27 14:59:15 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:59:15 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:59:15 cwltool INFO: Final process status is success
wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 15:00:06 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 15:00:06 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 15:00:10 arvados.cwl-runner INFO: Using collection qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 15:00:10 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-yei99yw56p1atru
2017-09-27 15:00:11 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 15:00:11 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 15:00:11 cwltool INFO: Final process status is success
wardv@shell.qr1hi:~$ vi noreuse.cwl 
wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 15:01:47 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 15:01:47 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 15:01:50 arvados.cwl-runner INFO: Using collection qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 15:01:51 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-ajm8a2tndcojpj9
2017-09-27 15:01:51 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 15:01:51 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 15:01:51 cwltool INFO: Final process status is success

Subtasks 2 (0 open — 2 closed)

Task #12365: Review 12347-disable-reuseResolvedTom Clegg09/28/2017Actions
Task #12373: Review 12347-obey-wf-reuse-hintResolvedPeter Amstutz09/28/2017Actions
Actions
Copy link
 #1

Updated by Ward Vandewege over 6 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Ward Vandewege over 6 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Tom Clegg over 6 years ago

Reading source:sdk/cwl/arvados_cwl/arvcontainer.py it looks like RunnerContainer.arvados_job_spec() fails to set "use_existing":False in the container request that it uses to invoke the "arvados-cwl-runner --disable-reuse" container. So, in effect, we get "run this with --disable-reuse, unless we've already run this with --disable-reuse, in which case we can use the existing result." Whoops. If this analysis is correct, the fix is something like

         if self.enable_reuse:
             command.append("--enable-reuse")
+            container_req["use_existing"] = True
         else:
             command.append("--disable-reuse")
+            container_req["use_existing"] = False

This only affects api=containers (crunch2), of course.

Actions
Copy link
 #4

Updated by Ward Vandewege over 6 years ago

Tom Clegg wrote:

Reading source:sdk/cwl/arvados_cwl/arvcontainer.py it looks like RunnerContainer.arvados_job_spec() fails to set "use_existing":False in the container request that it uses to invoke the "arvados-cwl-runner --disable-reuse" container. So, in effect, we get "run this with --disable-reuse, unless we've already run this with --disable-reuse, in which case we can use the existing result." Whoops. If this analysis is correct, the fix is something like

[...]

This only affects api=containers (crunch2), of course.

I tested on shell.9tee4 by patching the version of arvcontainer.py. This has the effect that if you specify --disable-reuse on the command line, it no longer reuses. That is good.

wardtest@shell.9tee4:~$ arvados-cwl-runner --disable-reuse noreuse.cwl 
2017-09-27 16:34:16 cwltool INFO: /usr/bin/arvados-cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 16:34:16 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 16:34:18 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-x8a5djr07yo8xbq
2017-09-27 16:35:18 arvados.cwl-runner INFO: [container noreuse.cwl] 9tee4-xvhdp-x8a5djr07yo8xbq is Final
2017-09-27 16:35:18 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 16:35:18 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 16:35:18 cwltool INFO: Final process status is success
wardtest@shell.9tee4:~$ arvados-cwl-runner --disable-reuse noreuse.cwl 
2017-09-27 16:35:33 cwltool INFO: /usr/bin/arvados-cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 16:35:33 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 16:35:35 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-jq8owx258lqbpoj
2017-09-27 16:36:35 arvados.cwl-runner INFO: [container noreuse.cwl] 9tee4-xvhdp-jq8owx258lqbpoj is Final
2017-09-27 16:36:36 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 16:36:36 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}

But if I don't set --disable-reuse on the command line, and just rely on the hint, it is still being ignored:

2017-09-27 16:36:36 cwltool INFO: Final process status is success
wardtest@shell.9tee4:~$ arvados-cwl-runner noreuse.cwl 
2017-09-27 16:36:46 cwltool INFO: /usr/bin/arvados-cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 16:36:46 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 16:36:49 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-bmacb1j8adanhvn
2017-09-27 16:36:49 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 16:36:49 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 16:36:49 cwltool INFO: Final process status is success

Tom suggested we need an updated arvados/jobs image to fix that.

Actions
Copy link
 #5

Updated by Tom Morris over 6 years ago

  • Target version set to 2017-10-11 Sprint
Actions
Copy link
 #6

Updated by Peter Amstutz over 6 years ago

Oh, it also needs to look at the toplevel requirement on the workflow. Good point.

Actions
Copy link
 #7

Updated by Tom Clegg over 6 years ago

  • Assigned To set to Tom Clegg
Actions
Copy link
 #8

Updated by Tom Clegg over 6 years ago

12347-disable-reuse @ 4febac9dd03bc4be3cf59827bfc4c8f5bcbe1a57
  • When --disable-reuse is given, don't reuse an existing outer cwl-runner job/container, even if there's one that ran with --disable-reuse.

(I also have a branch 12347-obey-wf-reuse-hint with a test for "engage --disable-reuse behaviors when the workflow says enableReuse:false", but haven't figured out how to implement that.)

Actions
Copy link
 #9

Updated by Tom Clegg over 6 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #10

Updated by Peter Amstutz over 6 years ago

12347-disable-reuse @ 4febac9dd03bc4be3cf59827bfc4c8f5bcbe1a57 LGTM

Actions
Copy link
 #11

Updated by Peter Amstutz over 6 years ago

Try this:

class Runner(object):
    """Base class for runner processes, which submit an instance of
    arvados-cwl-runner and wait for the final result.""" 

    def __init__(self, runner, tool, job_order, enable_reuse,
                 output_name, output_tags, submit_runner_ram=0,
                 name=None, on_error=None, submit_runner_image=None,
                 intermediate_output_ttl=0):
        self.arvrunner = runner
        self.tool = tool
        self.job_order = job_order
        self.running = False
        if enable_reuse:
            reuse_req, _ = self.tool.get_feature(self, "http://arvados.org/cwl#ReuseRequirement")
            if reuse_req:
                enable_reuse = reuse_req["enableReuse"]
        self.enable_reuse = enable_reuse
        .
        .
        .
Actions
Copy link
 #12

Updated by Tom Clegg over 6 years ago

Thanks!

12347-obey-wf-reuse-hint @ 6fa8ea28132c59c75f3356ecc62a7d4fdef0d5e0

Actions
Copy link
 #13

Updated by Peter Amstutz over 6 years ago

12347-obey-wf-reuse-hint @ 6fa8ea28132c59c75f3356ecc62a7d4fdef0d5e0 LGTM

Actions
Copy link
 #14

Updated by Anonymous over 6 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:f7a197bc9f1d416c37467feb01ff3b87c323e2b2.

Actions
Copy link
 #15

Updated by Ward Vandewege over 6 years ago

Hrm, this appears to still be broken:

wardtest@shell.9tee4:~$ cat noreuse.cwl 
# Copyright (C) The Arvados Authors. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

cwlVersion: v1.0
class: Workflow
$namespaces:
  arv: "http://arvados.org/cwl#" 
inputs: []
outputs: []
steps:
  step1:
    in:
      message:
        default: "hello world" 
    out: [output]
    hints:
      arv:ReuseRequirement:
        enableReuse: false
    run: stdout.cwl
wardtest@shell.9tee4:~$ dpkg -l |grep arv
ii  arvados-src                      0.1.20170929182415.9e58fc0-1   all          The Arvados source code
ii  libpam-arvados                   0.1.20170629115132-2           all          PAM module for authenticating shell logins using Arvados API tokens
ii  python-arvados-cwl-runner        1.0.20170929125635-3           all          The Arvados CWL runner
ii  python-arvados-fuse              0.1.20170923003803-2           all          The Keep FUSE driver
ii  python-arvados-python-client     0.1.20170818194607-2           all          The Arvados Python SDK

wardtest@shell.9tee4:~$ arvados-cwl-runner noreuse.cwl 
2017-09-29 20:21:55 cwltool INFO: /usr/bin/arvados-cwl-runner f7a197bc9f1d416c37467feb01ff3b87c323e2b2 1.0.20170929125635, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-29 20:21:55 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-29 20:21:58 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-q3kj3u44jxjs2u9
2017-09-29 20:21:59 arvados.cwl-runner INFO: Overall process status is success
2017-09-29 20:21:59 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-29 20:21:59 cwltool INFO: Final process status is success

wardtest@shell.9tee4:~$ arvados-cwl-runner noreuse.cwl 
2017-09-29 20:22:03 cwltool INFO: /usr/bin/arvados-cwl-runner f7a197bc9f1d416c37467feb01ff3b87c323e2b2 1.0.20170929125635, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-29 20:22:03 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-29 20:22:06 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-hra5p96c6zrxoqj
2017-09-29 20:22:07 arvados.cwl-runner INFO: Overall process status is success
2017-09-29 20:22:07 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-29 20:22:07 cwltool INFO: Final process status is success

Am I missing something?

Actions
Copy link
 #16

Updated by Ward Vandewege over 6 years ago

Update - Tom suggested I move the noreuse hint to the top level, which does indeed work now:

wardtest@shell.9tee4:~$ cat noreuse.cwl 
# Copyright (C) The Arvados Authors. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

cwlVersion: v1.0
class: Workflow
$namespaces:
  arv: "http://arvados.org/cwl#" 
inputs: []
outputs: []

hints:
  arv:ReuseRequirement:
    enableReuse: false

steps:
  step1:
    in:
      message:
        default: "hello world" 
    out: [output]
    run: stdout.cwl
wardtest@shell.9tee4:~$ arvados-cwl-runner noreuse.cwl 
2017-09-29 21:33:16 cwltool INFO: /usr/bin/arvados-cwl-runner f7a197bc9f1d416c37467feb01ff3b87c323e2b2 1.0.20170929125635, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-29 21:33:16 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-29 21:33:19 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-oaosw6g1eb1561r

2017-09-29 21:34:48 arvados.cwl-runner INFO: [container noreuse.cwl] 9tee4-xvhdp-oaosw6g1eb1561r is Final
2017-09-29 21:34:49 arvados.cwl-runner INFO: Overall process status is success
2017-09-29 21:34:49 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-29 21:34:49 cwltool INFO: Final process status is success
wardtest@shell.9tee4:~$ 
wardtest@shell.9tee4:~$ 
wardtest@shell.9tee4:~$ 
wardtest@shell.9tee4:~$ arvados-cwl-runner noreuse.cwl 
2017-09-29 21:34:59 cwltool INFO: /usr/bin/arvados-cwl-runner f7a197bc9f1d416c37467feb01ff3b87c323e2b2 1.0.20170929125635, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-29 21:34:59 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-29 21:35:02 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-1ojhfqj1estq3yc
2017-09-29 21:36:17 arvados.cwl-runner INFO: [container noreuse.cwl] 9tee4-xvhdp-1ojhfqj1estq3yc is Final
2017-09-29 21:36:17 arvados.cwl-runner INFO: Overall process status is success
2017-09-29 21:36:17 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-29 21:36:17 cwltool INFO: Final process status is success
Actions
Copy link

Also available in: Atom PDF