Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422023-01-20T15:22:27ZArvados
Redmine Arvados - Bug #19966 (New): collection sharing link missing wget/curl authentication instructionshttps://dev.arvados.org/issues/199662023-01-20T15:22:27ZMichael Crusoe
<p>Example:</p>
<p><a class="external" href="https://jutro-4zz18-it9ovofkgkj0a5s.collections.jutro.arvadosapi.com/">https://jutro-4zz18-it9ovofkgkj0a5s.collections.jutro.arvadosapi.com/</a></p>
<p><img src="https://dev.arvados.org/attachments/download/3201/clipboard-202301201615-g46ej.png" alt="" /></p>
<pre>
$ wget --mirror --no-parent --no-host --cut-dirs=0 https://jutro-4zz18-it9ovofkgkj0a5s.collections.jutro.arvadosapi.com/
--2023-01-20 15:12:31-- https://jutro-4zz18-it9ovofkgkj0a5s.collections.jutro.arvadosapi.com/
Resolving <a href="https://arvadosapi.com/jutro-4zz18-it9ovofkgkj0a5s">jutro-4zz18-it9ovofkgkj0a5s</a>.collections.jutro.arvadosapi.com (<a href="https://arvadosapi.com/jutro-4zz18-it9ovofkgkj0a5s">jutro-4zz18-it9ovofkgkj0a5s</a>.collections.jutro.arvadosapi.com)... 107.21.164.177
Connecting to <a href="https://arvadosapi.com/jutro-4zz18-it9ovofkgkj0a5s">jutro-4zz18-it9ovofkgkj0a5s</a>.collections.jutro.arvadosapi.com (<a href="https://arvadosapi.com/jutro-4zz18-it9ovofkgkj0a5s">jutro-4zz18-it9ovofkgkj0a5s</a>.collections.jutro.arvadosapi.com)|107.21.164.177|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Username/Password Authentication Failed.
</pre>
<p>There is no link to the rest of the Arvados interface. I had to guess that I could go to jutro.arvadosapi.com and from there search for <a href="https://arvadosapi.com/jutro-4zz18-it9ovofkgkj0a5s">jutro-4zz18-it9ovofkgkj0a5s</a> ; and then<br />1. for workbench 2: guess that the 3 dot menu contained a "Open with 3rd party client option" to find the working wget/curl instructions (--http-user=michaelcrusoe --http-passwd=XXX)<br />2. for workbench 1: use "create sharing link", click it, and then use the working wget/curl instructions</p> Arvados - Bug #19169 (Resolved): error with Single host Arvados setuphttps://dev.arvados.org/issues/191692022-05-30T14:01:15ZMichael Crusoe
<p>Summary for local <br />------------- <br />Succeeded: 79 (changed=68) <br />Failed: 2 <br />-------------</p>
<p>nginx failed? Adding "server_names_hash_bucket_size: 128" to the nginx config file allowed me to start the service, but the arvados interface still doesn't come up</p> Arvados - Bug #17245 (Resolved): install/crunch2-slurm/install-slurm.html missing from ToChttps://dev.arvados.org/issues/172452021-01-07T09:29:47ZMichael Crusoe
<p>This document seems important, but does not appear to be linked to from any other doc or the sidebar table of contents.</p>
<p>Perhaps there are other orphaned docs as well..</p> Arvados - Feature #17149 (New): Checkpoint / snapshot to support migrating tasks when more memory...https://dev.arvados.org/issues/171492020-11-19T10:42:04ZMichael Crusoe
<p>Related to <a class="external" href="https://dev.arvados.org/issues/7568">https://dev.arvados.org/issues/7568</a> (why was that closed?)</p>
<p>Could be hourly snapshots, or based upon a memory threshold. Restoring on a node with more resources could be automatic or require user intervention.</p>
<p>Would not be available for `networkAccess: true` CWL tasks, for simplicity.</p> Arvados - Bug #17008 (Resolved): AttributeError: module 'httplib2' has no attribute 'SSLHandshake...https://dev.arvados.org/issues/170082020-10-15T14:54:42ZMichael Crusoe
<p>From running CWL v1.0 conformance tests against <a class="external" href="https://workbench.covid19workflows-vu.surf-hosted.nl">https://workbench.covid19workflows-vu.surf-hosted.nl</a></p>
<pre>
Test 47 failed: /home/michael/common-workflow-language/arvenv3.6/bin/arvados-cwl-runner --compute-checksum --project-uuid=testc-j7d0g-n7la9srj0ycdvix --outdir=/tmp/tmpc8__hhgl --quiet v1.0/env-wf2.cwl v1.0/env-job.json
Test requirements override hints
Returned non-zero
ERROR [container env-wf2.cwl] (<a href="https://arvadosapi.com/testc-dz642-uych2j0mwa6vm49">testc-dz642-uych2j0mwa6vm49</a>) error log:
2020-10-15T14:52:21.285202240Z stderr for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2020-10-15T14:52:21.285202240Z stderr socket.gaierror: [Errno -3] Temporary failure in name resolution
2020-10-15T14:52:21.285202240Z stderr
2020-10-15T14:52:21.285202240Z stderr During handling of the above exception, another exception occurred:
2020-10-15T14:52:21.285202240Z stderr
2020-10-15T14:52:21.285202240Z stderr Traceback (most recent call last):
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados/api.py", line 85, in _intercept_http_request
2020-10-15T14:52:21.285202240Z stderr return self.orig_http_request(uri, method, headers=headers, **kwargs)
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/httplib2/__init__.py", line 1994, in request
2020-10-15T14:52:21.285202240Z stderr cachekey,
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
2020-10-15T14:52:21.285202240Z stderr conn, request_uri, method, body, headers
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/httplib2/__init__.py", line 1564, in _conn_request
2020-10-15T14:52:21.285202240Z stderr raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
2020-10-15T14:52:21.285202240Z stderr httplib2.ServerNotFoundError: Unable to find the server at hackathon.covid19workflows-vu.surf-hosted.nl
2020-10-15T14:52:21.285202240Z stderr
2020-10-15T14:52:21.285202240Z stderr During handling of the above exception, another exception occurred:
2020-10-15T14:52:21.285202240Z stderr
2020-10-15T14:52:21.285202240Z stderr Traceback (most recent call last):
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 297, in main
2020-10-15T14:52:21.285202240Z stderr keep_params={"num_retries": 4})
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados/safeapi.py", line 33, in __init__
2020-10-15T14:52:21.285202240Z stderr self.localapi()
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados/safeapi.py", line 40, in localapi
2020-10-15T14:52:21.285202240Z stderr **self.api_params)
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados/api.py", line 278, in api_from_config
2020-10-15T14:52:21.285202240Z stderr return api(version=version, host=host, token=token, insecure=insecure, **kwargs)
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados/api.py", line 236, in api
2020-10-15T14:52:21.285202240Z stderr svc = apiclient_discovery.build('arvados', version, cache_discovery=False, **kwargs)
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
2020-10-15T14:52:21.285202240Z stderr return wrapped(*args, **kwargs)
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/googleapiclient/discovery.py", line 222, in build
2020-10-15T14:52:21.285202240Z stderr requested_url, discovery_http, cache_discovery, cache)
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/googleapiclient/discovery.py", line 269, in _retrieve_discovery_doc
2020-10-15T14:52:21.285202240Z stderr resp, content = http.request(actual_url)
2020-10-15T14:52:21.285202240Z stderr File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados/api.py", line 99, in _intercept_http_request
2020-10-15T14:52:21.285202240Z stderr except httplib2.SSLHandshakeError as e:
2020-10-15T14:52:21.285202240Z stderr AttributeError: module 'httplib2' has no attribute 'SSLHandshakeError'
2020-10-15T14:52:21.610802843Z crunch-run Container exited with code: 1
2020-10-15T14:52:21.753959895Z crunch-run Complete
ERROR Overall process status is permanentFail
WARNING Final process status is permanentFail
</pre> Arvados - Bug #17005 (New): CWL v1.2 Conformance test 83 failed twice on the new Arvados Playgroundhttps://dev.arvados.org/issues/170052020-10-14T16:19:59ZMichael Crusoe
<p>Test [83/337] wf_scatter_oneparam_valueFrom: Test valueFrom eval on scattered input parameter<br />Test 83 failed: /home/michael/common-workflow-language/arvenv3.6/bin/arvados-cwl-runner --compute-checksum --project-uuid=pirca-j7d0g-fkh3kn4kq1vvqtm --outdir=/tmp/tmp6xk5ho35 --quiet tests/scatter-valuefrom-wf6.cwl tests/scatter-valuefrom-job3.json<br />Test valueFrom eval on scattered input parameter<br />Returned non-zero<br />ERROR [container scatter-valuefrom-wf6.cwl] (<a href="https://arvadosapi.com/pirca-dz642-uwrxwr19y6xcrqw">pirca-dz642-uwrxwr19y6xcrqw</a>) error log:</p>
<pre><code>2020-10-14T16:11:35.897539571Z crunch-run crunch-run 2.1.0 (go1.13.4) started<br /> 2020-10-14T16:11:35.897567852Z crunch-run Executing container '<a href="https://arvadosapi.com/pirca-dz642-uwrxwr19y6xcrqw">pirca-dz642-uwrxwr19y6xcrqw</a>'<br /> 2020-10-14T16:11:35.897578659Z crunch-run Executing on host 'ip-10-254-254-121'<br /> 2020-10-14T16:11:36.005817744Z crunch-run Fetching Docker image from collection 'af41a455939e61edb3a11aafce28be9a+261'<br /> 2020-10-14T16:11:36.021792711Z crunch-run Using Docker image id 'sha256:39b6c29c7a40c32a9a4ff0b4f77d2cd3dea27b8d27f044c1561bd1763d91c5e8'<br /> 2020-10-14T16:11:36.022646887Z crunch-run Loading Docker image from keep<br /> 2020-10-14T16:11:48.783548379Z crunch-run Docker response: {"stream":"Loaded image ID: sha256:39b6c29c7a40c32a9a4ff0b4f77d2cd3dea27b8d27f044c1561bd1763d91c5e8\n"}<br /> 2020-10-14T16:11:48.783943080Z crunch-run Running [arv-mount --foreground --allow-other --read-write --crunchstat-interval=10 --file-cache 268435456 --mount-tmp tmp0 --mount-by-pdh by_id /tmp/crunch-run.<a href="https://arvadosapi.com/pirca-dz642-uwrxwr19y6xcrqw">pirca-dz642-uwrxwr19y6xcrqw</a>.778018510/keep224304085]<br /> 2020-10-14T16:11:50.406993608Z crunch-run Creating Docker container<br /> 2020-10-14T16:11:51.108306690Z crunch-run Attaching container streams<br /> 2020-10-14T16:11:51.233805082Z crunch-run Starting Docker container id '0830f4de61a1c1e4184f0ff78b5ad3a07dca85f9d79636914fc95295c1ddd020'<br /> 2020-10-14T16:11:52.011094758Z crunch-run Waiting for container to finish<br /> 2020-10-14T16:11:53.854639266Z stderr INFO /usr/bin/arvados-cwl-runner 2.1.0, arvados-python-client 2.1.0, cwltool 3.0.20200807132242<br /> 2020-10-14T16:11:53.866567328Z stderr INFO Resolved '/var/lib/cwl/workflow.json#main' to 'file:///var/lib/cwl/workflow.json#main'<br /> 2020-10-14T16:11:56.118985095Z stderr INFO Using cluster pirca (<a class="external" href="https://pirca.arvadosapi.com/">https://pirca.arvadosapi.com/</a>)<br /> 2020-10-14T16:11:56.306442111Z stderr INFO Using collection cache size 256 MiB<br /> 2020-10-14T16:11:56.330548611Z stderr INFO Running inside container <a href="https://arvadosapi.com/pirca-dz642-uwrxwr19y6xcrqw">pirca-dz642-uwrxwr19y6xcrqw</a><br /> 2020-10-14T16:11:56.346885341Z stderr INFO [workflow workflow.json#main] start<br /> 2020-10-14T16:11:56.347331871Z stderr INFO [workflow workflow.json#main] starting step step1<br /> 2020-10-14T16:11:56.348069440Z stderr INFO [step step1] start<br /> 2020-10-14T16:11:56.444169560Z stderr INFO [container step1] <a href="https://arvadosapi.com/pirca-xvhdp-pj4omuag6507i69">pirca-xvhdp-pj4omuag6507i69</a> state is Committed<br /> 2020-10-14T16:11:56.444169560Z stderr INFO [step step1] start<br /> 2020-10-14T16:11:56.541120952Z stderr INFO [container step1_2] <a href="https://arvadosapi.com/pirca-xvhdp-yu3w69u03ijtnm9">pirca-xvhdp-yu3w69u03ijtnm9</a> state is Committed<br /> 2020-10-14T16:12:04.141247545Z stderr ERROR Caught signal 15, exiting.<br /> 2020-10-14T16:12:04.198345475Z stderr ERROR Interrupted, workflow will be cancelled<br /> 2020-10-14T16:12:04.262219931Z stderr ERROR Workflow error, try again with --debug for more information:<br /> 2020-10-14T16:12:04.262219931Z stderr Workflow did not return a result.<br /> 2020-10-14T16:12:04.685214802Z crunch-run Container exited with code: 1<br /> 2020-10-14T16:12:04.741748338Z crunch-run Complete<br />ERROR Overall process status is permanentFail<br />WARNING Final process status is permanentFail</code></pre>
<p>The third time it succeeded.</p> Arvados - Bug #16883 (Resolved): mention Debian packages in installation instructionshttps://dev.arvados.org/issues/168832020-09-24T14:28:14ZMichael Crusoe
<p>Example: <a class="external" href="https://doc.arvados.org/v2.0/install/crunch2-slurm/install-dispatch.html">https://doc.arvados.org/v2.0/install/crunch2-slurm/install-dispatch.html</a> could be updated to mention the Debian packages first, instead of the "install SLURM from scratch" link:</p>
<p>The Introduction says:</p>
<code>
This assumes you already have a SLURM cluster, and have set up all of your compute nodes . For information on installing SLURM, see this install guide.
</code>
<p>The "this install guide" string links to <a class="external" href="https://slurm.schedmd.com/quickstart_admin.html">https://slurm.schedmd.com/quickstart_admin.html</a> which is a recipe for installation from source.</p> Arvados - Bug #16633 (New): "Arvados VM" is not defined in the docshttps://dev.arvados.org/issues/166332020-07-26T10:53:56ZMichael Crusoe
<p>How does that relate to <a class="external" href="https://doc.arvados.org/v2.0/install/install-shell-server.html">https://doc.arvados.org/v2.0/install/install-shell-server.html</a> ?</p> Arvados - Bug #16477 (Resolved): arvados-cwl-runner is missing --skip-schemashttps://dev.arvados.org/issues/164772020-06-02T08:44:47ZMichael Crusoe
<p>For example, due to <a class="external" href="https://github.com/schemaorg/schemaorg/issues/2593">https://github.com/schemaorg/schemaorg/issues/2593</a> , CWL descriptions that contain a `$schemas: [ <a class="external" href="https://schema.org/docs/schema_org_rdfa.html">https://schema.org/docs/schema_org_rdfa.html</a> ]` or similar will fail to run with `arvados-cwl-runner` as that link now 404s</p> Arvados - Bug #16409 (New): User tool runs out of memory and is killed ; job still reported as be...https://dev.arvados.org/issues/164092020-05-06T13:52:08ZMichael Crusoe
<p><a class="external" href="https://workbench.lugli.arvadosapi.com/container_requests/lugli-xvhdp-ygaeadxu1omtmmp">https://workbench.lugli.arvadosapi.com/container_requests/lugli-xvhdp-ygaeadxu1omtmmp</a></p>
<p>The log shows that execution was halted <a class="external" href="https://collections.lugli.arvadosapi.com/c=617e16305447ab851e6367dea137b11d-10070/_/results/log.20200505-11-20-17.log?disposition=inline">https://collections.lugli.arvadosapi.com/c=617e16305447ab851e6367dea137b11d-10070/_/results/log.20200505-11-20-17.log?disposition=inline</a></p>
<p>It is known that this tool needs more memory than was asked for / assigned</p> Arvados - Bug #16401 (New): Provenance graph not correctly capturing scatter/gatherhttps://dev.arvados.org/issues/164012020-05-04T12:10:28ZMichael Crusoe
<p><a class="external" href="https://workbench.lugli.arvadosapi.com/container_requests/lugli-xvhdp-8vjms0p5il7lypf#Provenance">https://workbench.lugli.arvadosapi.com/container_requests/lugli-xvhdp-8vjms0p5il7lypf#Provenance</a></p>
<p>The relationship between bin_paths_[1-6] and the aggregate node is not indicated.</p> Arvados - Bug #16400 (New): Project Description editing lacks preview mode and lacks formatting i...https://dev.arvados.org/issues/164002020-05-03T08:50:48ZMichael CrusoeArvados - Feature #16372 (New): Vertical autoscalinghttps://dev.arvados.org/issues/163722020-04-26T09:14:52ZMichael Crusoe
<p><a class="external" href="https://dl.acm.org/doi/pdf/10.1145/3342195.3387524?download=true">https://dl.acm.org/doi/pdf/10.1145/3342195.3387524?download=true</a></p>
<p>Would eliminate the need for users to guess the ResourceRequirement parameters</p> Arvados - Bug #16371 (New): <ArvadosModel::UnresolvableContainerError: docker image "arvados/jobs...https://dev.arvados.org/issues/163712020-04-25T16:01:29ZMichael Crusoe
<p>go to <a class="external" href="https://workbench.lugli.arvadosapi.com/container_requests/lugli-xvhdp-2nr90r0wkh27pmt">https://workbench.lugli.arvadosapi.com/container_requests/lugli-xvhdp-2nr90r0wkh27pmt</a> and click "Run"</p> Arvados - Bug #16370 (Closed): data collections from sub projects not listed in the workflow conf...https://dev.arvados.org/issues/163702020-04-25T12:12:18ZMichael Crusoe
<p>Example, I tried to run</p>
<p><a class="external" href="https://workbench.lugli.arvadosapi.com/workflows/lugli-7fd4e-4rfpciv0vp7717p">https://workbench.lugli.arvadosapi.com/workflows/lugli-7fd4e-4rfpciv0vp7717p</a><br />in <br /><a class="external" href="https://workbench.lugli.arvadosapi.com/projects/lugli-j7d0g-n8ogmx16xiqfj2z">https://workbench.lugli.arvadosapi.com/projects/lugli-j7d0g-n8ogmx16xiqfj2z</a></p>
<p>When configuring the sole input I was unable to see any of the datasets in <a class="external" href="https://workbench.lugli.arvadosapi.com/projects/lugli-j7d0g-825x3r5vcs41dus">https://workbench.lugli.arvadosapi.com/projects/lugli-j7d0g-825x3r5vcs41dus</a> (a subproject)</p>
<p>I had to move the workflow execution request to the subproject to see those data sets.</p>