Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422024-03-22T13:25:49ZArvados
Redmine Arvados Workbench 2 - Idea #21615 (New): Details Panel should show details for every type of reso...https://dev.arvados.org/issues/216152024-03-22T13:25:49ZLisa Knox
<p>The details panel currently only shows details when viewing a project, workflow, process, or collection and shows empty everywhere else. This includes on the Shell Access view, Instance Types, and a few other places where there is really nothing to display, but the details panel will remain open if it was open when navigating to that view.</p>
<p>The details panel should display something useful if the current view allows for it, or should close automatically when navigating to a view where it is irrelevant. Having it display an empty element with the word "Projects" at the top in all non-specified cases is not helpful.</p> Arvados - Idea #21610 (New): Evaluate the feasibility of exporting a prometheus-compatible API fo...https://dev.arvados.org/issues/216102024-03-20T15:55:49ZPeter Amstutzpeter.amstutz@curii.com
<p>Useful links</p>
<p><a class="external" href="https://prometheus.io/docs/prometheus/latest/querying/api/">https://prometheus.io/docs/prometheus/latest/querying/api/</a></p>
<p><a class="external" href="https://www.npmjs.com/package/chartjs-plugin-datasource-prometheus">https://www.npmjs.com/package/chartjs-plugin-datasource-prometheus</a></p> Arvados - Idea #21595 (New): 'shared' should use usernames, not full nameshttps://dev.arvados.org/issues/215952024-03-14T14:52:12ZPeter Amstutzpeter.amstutz@curii.com
<p>The 'shared' directory uses full names. These have a couple of problems:</p>
<ul>
<li>Always contain spaces and may have other characters that make it awkward with Unix tooling</li>
<li>Not unique. For example pirca has multiple accounts with full_name "Peter Amstutz". FUSE ends up picking one account and the other accounts just can't be accessed through FUSE.</li>
</ul>
<p>It should use 'username' instead, which is unique on a given Arvados instance.</p>
<p>I think the only question is whether it is worth the effort to maintain backwards compatibility (by making the 'username' behavior a new option) or we just change the existing behavior in place.</p>
<p>I suppose one way to do it would be to change to using usernames by default but add an option that restores the previous behavior of using full names.</p> Arvados - Idea #21581 (New): Crunch saves compute node journals to collections readable only by a...https://dev.arvados.org/issues/215812024-03-12T17:57:35ZBrett Smithbrett.smith@curii.com
<p>Problem:</p>
<ul>
<li>Compute nodes and tasks can fail for any number of reasons. You basically need a full system log to diagnose some problems.</li>
<li>We can't just give users the system log, there's too much sensitive information in there and it's practically impossible to reliably know what needs to be redacted.</li>
<li>And even if it wasn't, regular users mostly can't act on this information, and it may nede to be subject to different retention policies than regular container logs, etc.</li>
</ul>
<p>Big idea: Crunch occasionally saves the system journal (and other logs?) to a collection that should only be readable by Arvados administrators. Administrators can go back and review these logs to diagnose problems.</p>
<p>Implementation idea:</p>
<ul>
<li>crunch-run gains a subcommand to upload the journal to a collection. When you run it, it:
<ul>
<li>Runs <code>journalctl --sync</code> to make sure all entries so far are written to disk
<ul>
<li>TBD: Does this need sudo?</li>
<li>The rest of the work should probably continue even if this command fails. Even if it means we can't get all the logs, we might as well capture what we can.</li>
</ul>
</li>
<li>Creates a collection from the recursive contents of <code>/var/log/journal</code>
<ul>
<li>TBD: Any other log files we should throw in?</li>
<li>The collection should have a property that indicates which container(s) these system logs correspond to. This should be a system property with the <code>arv:</code> prefix that's documented.</li>
<li>The collection should have a <code>trashed_at</code> time in the future. TBD: Should this time be configurable? If it's set to zero, should this functionality be disabled?</li>
</ul>
</li>
</ul>
</li>
<li>crunch-dispatch calls this crunch-run command when specific events occur
<ul>
<li>When a container finishes</li>
<li>When the cloud dispatcher decides to terminate a node</li>
</ul></li>
</ul>
<p>Setup that needs to happen:</p>
<ul>
<li>There needs to be a dedicated Unix account on the compute nodes to run this
<ul>
<li>It should be a member of the <code>systemd-journal</code> group to read the journal</li>
<li>It may need sudo permission to run <code>journalctl --sync</code> passwordless</li>
</ul>
</li>
<li>Permissions can be limited on the Arvados side
<ul>
<li>This can't use the same API token as the container because the permissions are completely different</li>
<li>The token could be scoped pretty narrowly: just permission to <code>PUT</code> a collection and <code>GET</code> the owning project and similar related resources</li>
<li>It seems like we either need (a) a dedicated user account that just has all these journal collections in its home project, or (b) a configurable UUID of a project where all these journal collections are saved</li>
</ul></li>
</ul>
<p>Background:</p>
<ul>
<li>Read a saved journal with <code>journalctl --root=PATH</code></li>
<li>We considered setting something up that automatically does this when the node goes down (a service that's <code>WantedBy=shutdown.target</code>?) It has the advantage that it could work even if crunch-dispatch has trouble coordinating with the compute node, but:
<ul>
<li>The upload might take a while and we're not sure if systemd and/or the cloud provider would be patient enough to let it run</li>
<li>It would require us to permanently store credentials somewhere, which isn't insurmountable but something we generally avoid doing</li>
</ul></li>
</ul> Arvados - Idea #21573 (New): Publish fewer packageshttps://dev.arvados.org/issues/215732024-03-04T20:48:22ZBrett Smithbrett.smith@curii.com
<p>We build a lot of individual packages that are basically just remnants of older versions and build processes of Arvados, and serve no functional purpose. We could migrate to this much simpler set. This would reduce build times, simplify install instructions, and take less space on installed machines.</p>
<ul>
<li><code>arvados-server</code>: The <code>arvados-server</code> binary, configuration files, documentation, and symlinks in <code>/usr/bin</code> for all the different names you can call it by</li>
<li><code>arvados-client</code>: The same thing for the <code>arvados-client</code> binary</li>
<li><code>arvados-python-client</code>: A single venv with the PySDK, a-c-r, arv-mount, crunchstat-summary, and user-activity all bundled.</li>
<li><code>arvados-python-compute-node</code>: A single venv with arv-mount and docker-cleaner all bundled. We would need to think about how to handle the arv-mount conflict. If Crunch could learn to call it directly from this venv, that would help. Worst case, we could just declare the venv packages conflict with each other, I'm not aware of anyone who wants to install both on the same system.</li>
<li><code>arvados-api-server</code>, <code>arvados-workbench2</code>, <code>arvados-src</code>, and <code>libpam-arvados-go</code> remain as they are now, as well as any other little utility tools I overlooked.</li>
</ul> Arvados Epics - Idea #21569 (New): User notificationshttps://dev.arvados.org/issues/215692024-03-01T21:37:50ZPeter Amstutzpeter.amstutz@curii.comArvados - Idea #21551 (New): Python SDK methods to generate keep: URIshttps://dev.arvados.org/issues/215512024-02-28T14:57:49ZBrett Smithbrett.smith@curii.com
<p>These would be utility methods to make it easier to write Python that generates CWL inputs. Something like:</p>
<pre><code class="python syntaxhl"><span class="k">class</span> <span class="nc">ArvadosFile</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">keep_uri</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">parent</span><span class="p">.</span><span class="n">keep_uri</span><span class="p">()</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">'</span>
<span class="k">class</span> <span class="nc">Collection</span><span class="p">(</span><span class="n">RichCollectionBase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">keep_uri</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
<span class="k">if</span> <span class="n">path</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">parts</span> <span class="o">=</span> <span class="p">[</span><span class="sa">f</span><span class="s">'keep:</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">portable_data_hash</span><span class="p">()</span><span class="si">}</span><span class="s">'</span><span class="p">]</span>
<span class="n">parts</span><span class="p">.</span><span class="n">extend</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">stream_name</span><span class="p">().</span><span class="n">split</span><span class="p">(</span><span class="s">'/'</span><span class="p">))</span>
<span class="k">if</span> <span class="n">parts</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s">'.'</span><span class="p">:</span>
<span class="n">parts</span><span class="p">.</span><span class="n">pop</span><span class="p">()</span>
<span class="n">parts</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">''</span><span class="p">)</span>
<span class="k">return</span> <span class="s">'/'</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">find</span><span class="p">(</span><span class="n">path</span><span class="p">).</span><span class="n">keep_uri</span><span class="p">()</span>
</code></pre>
<p>Basically given any file or collection object, it should be possible to get the corresponding keep: URI. This might need additional handling for corner cases but this is the gist.</p> Arvados - Idea #21549 (New): aws-cli-cwl supports uploading entire directorieshttps://dev.arvados.org/issues/215492024-02-28T14:28:48ZBrett Smithbrett.smith@curii.com
<p>Today <code>aws-s3-bulk-upload.cwl</code> only supports File inputs. Extend it to support Directory inputs as well. When the input is a directory, <code>aws s3 cp</code> should be called with the <code>--recursive</code> flag.</p> Arvados - Bug #17434 (New): project not shown in hierarchy on the left of wb2https://dev.arvados.org/issues/174342021-03-01T21:07:39ZDaniel Kutyła
<p>Project that is being shared is not visible in the left tree menu</p> Arvados - Bug #17396 (New): Favorites copy dialog further issueshttps://dev.arvados.org/issues/173962021-02-16T22:31:18ZDaniel Kutyła
<p><a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: Favorites in copy dialog is different to favorite list (Resolved)" href="https://dev.arvados.org/issues/17306">#17306</a> solved main issues reported by our users but there are some stuff that came out while fixing those:</p>
<p>- We can not copy element which is located in a subdir without copying other elements as parent folder will not be selected and thus it will be removed <br />- For a large collections files copy operation might be very slow as we remove non selected elements in a post copy operation</p> Arvados - Bug #17300 (New): array of array of File causes 'NoneType' object has no attribute 'res...https://dev.arvados.org/issues/173002021-01-29T14:19:14ZStanislaw Adaszewski
<p>When running an ExpressionTool which has an output of the following type:</p>
<pre>
outputs:
subject_images:
type:
type: array # subjects
items:
type: array # timepoints
items: File
</pre>
<p>I receive the following error:</p>
<pre>
ERROR Unhandled error:
'NoneType' object has no attribute 'resolved'
Traceback (most recent call last):
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/main.py", line 833, in main
logger=_logger)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 767, in arv_executor
self.final_output, self.final_output_collection = self.make_output_collection(self.output_name, storage_classes, self.output_tags, self.final_output)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 467, in make_output_collection
adjustFileObjs(outputObj, rewrite)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/pathmapper.py", line 48, in adjustFileObjs
visit_class(rec, ("File",), op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 221, in visit_class
visit_class(rec[d], cls, op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 224, in visit_class
visit_class(d, cls, op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 224, in visit_class
visit_class(d, cls, op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 219, in visit_class
op(rec)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 461, in rewrite
fileobj["location"] = generatemapper.mapper(fileobj["location"]).target
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/pathmapper.py", line 332, in mapper
return MapperEnt(v.resolved, v.target + src[i:], v.type, v.staged)
AttributeError: 'NoneType' object has no attribute 'resolved'
</pre>
<p>If I replace type: File with type: string and store locations instead of File objects the request finishes successfully and returns the expected list of locations.</p> Arvados - Bug #16228 (New): NullPointerException while uploading file using Java SDKhttps://dev.arvados.org/issues/162282020-03-10T07:28:41ZKrzysztof Majewski
<p>When trying to upload a files using Java SDK I get the following error:</p>
<pre>
2020-03-10 08:07:23 ERROR - Cannot upload file to Keep server.
org.arvados.client.exception.ArvadosApiException: java.net.ConnectException: Failed to connect to keep1.ardev.roche.com/10.115.164.103:25107
at org.arvados.client.api.client.BaseApiClient.getResponseBody(BaseApiClient.java:76)
at org.arvados.client.api.client.BaseApiClient.newCall(BaseApiClient.java:49)
at org.arvados.client.api.client.KeepServerApiClient.upload(KeepServerApiClient.java:42)
at org.arvados.client.logic.keep.FileTransferHandler.put(FileTransferHandler.java:35)
at org.arvados.client.logic.keep.KeepClient.lambda$put$1(KeepClient.java:110)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.net.ConnectException: Failed to connect to keep1.ardev.roche.com/10.115.164.103:25107
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:270)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:176)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:236)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:109)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:77)
at okhttp3.internal.connection.Transmitter.newExchange$okhttp(Transmitter.kt:162)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:35)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:82)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:71)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:184)
at okhttp3.RealCall.execute(RealCall.kt:66)
at org.arvados.client.api.client.BaseApiClient.getResponseBody(BaseApiClient.java:63)
... 10 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:126)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:268)
... 29 more
</pre>
<p>The keep1 server is taken from: <a class="external" href="https://api.ardev.roche.com/arvados/v1/keep_services/accessible">https://api.ardev.roche.com/arvados/v1/keep_services/accessible</a>. If I manipulate the response and remove it, leaving just keep2 everything works fine.</p>
<p>This is my code used for this example:</p>
<pre>
String serverUrl = "ardev.roche.com";
ExternalConfigProvider config = ExternalConfigProvider.builder()
.apiToken("api_token")
.apiHost(format("api.%s", serverUrl))
.apiPort(443)
.keepWebHost(format("collections.%s", serverUrl))
.keepWebPort(443)
.apiProtocol("https")
.apiHostInsecure(true)
.fileSplitSize(64)
.fileSplitDirectory(new File("C:\\file-split"))
.numberOfCopies(2)
.numberOfRetries(0)
.build();
List<File> files = asList(new File("C:\\samples\\sample.fcs"));
Collection collection = new CollectionsApiClient(config).get("collection_uuid");
new ArvadosFacade(config).uploadToExistingCollection(files, collection.getUuid());
</pre> Arvados - Bug #15183 (New): [Workbench2] No errors reported when file is missing a blockhttps://dev.arvados.org/issues/151832019-05-01T21:39:26ZTom Morristfmorris@veritasgenetics.com
<p>When looking at collection <a href="https://arvadosapi.com/su92l-4zz18-44qy35pg0lxshkf">su92l-4zz18-44qy35pg0lxshkf</a>/cwl.input.yml the file is cwl.output.json is listed as containing 361 bytes, but when downloaded from Workbench, it produces a zero byte file, but without giving any error or warning.</p>
<pre>
$ md5sum keep/by_id/su92l-4zz18-44qy35pg0lxshkf/*
a890894f5a748d2a93440fd046c73069 keep/by_id/su92l-4zz18-44qy35pg0lxshkf/cwl.input.yml
md5sum: keep/by_id/su92l-4zz18-44qy35pg0lxshkf/cwl.output.json: Input/output error
d41d8cd98f00b204e9800998ecf8427e keep/by_id/su92l-4zz18-44qy35pg0lxshkf/hu27FD1F_var-GS000034212-ASM.bed
b02868b9ac1264bd1f04b61d04518f57 keep/by_id/su92l-4zz18-44qy35pg0lxshkf/workflow.cwl
</pre> GET-Evidence - Bug #835 (New): Error when saving edits or submittinghttps://dev.arvados.org/issues/8352012-02-06T16:44:19ZMadeleine Ballmpball@gmail.com
<p>When saving or submitting we're seeing this error message now: "Save failed -- received bogus response from server."</p>
<p>Seems to date to changes made in commit <a class="changeset" title="if session is expired when "save draft" is clicked, pop-up a login window instead of falsely proc..." href="https://dev.arvados.org/projects/get-evidence/repository/8/revisions/5dc9adee7bd5cde0d1c9b16652cf0b5e8d6229d3">5dc9adee</a> ?</p>
<p>Sasha pointed out that the changes still get saved, but it's still a serious bug -- especially since we just sent the paper out for review and were planning to encourage people to edit...</p> GET-Evidence - Bug #468 (New): latest-flat incorrectly displays non single-base substitution vari...https://dev.arvados.org/issues/4682011-02-15T11:15:42ZMadeleine Ballmpball@gmail.com
<p>I haven't checked, but this problem likely extends to deletion & other new length changing or multiple-aa variants.</p>
<p>This bug blocks creating a flatfile version of gff_getevidence_map.py</p>
<p>For example, FIG4-K278Shift is showing up like this (2nd and 3rd columns are incorrect):<br />FIG4 278 278 pathogenic Moderate clinical importance, Uncertain pathogenic recessive 0 0 0 0 0 3 Y -- - Y 0 Y 3 - 2 - Y - - - - Y 4 Y 2N 1 0 This variant is predicted to cause a frameshift and may cause Charcot-Marie-Tooth Disease Type 4J in an autosomal recessive manner. Other variants in this gene which cause frameshift and premature termination have been implicated in causing this disease when compound heterozygous with another FIG4 variant.</p>
<p>While MYL2-A13T shows up like this:<br />MYL2 Ala13Thr A13T pathogenic Low clinical importance, Uncertain pathogenic dominant 1 1 1 6 6 3 Y 4Y 0 Y ! Y 4 Y 3 Y Y Y - - Y Y 1 - Familial Hypertrophic Cardiomyopathy 4596 1 455 3.054 4 N 0 0 This rare variant is implicated in causing late-onset familial hypertrophic cardiomyopathy. The variant has been found in five affected Caucasian individuals (in four families), but affected non-carriers and unaffected carriers have also been observed. No statistically significant enrichment of this variant in cases vs. controls has been shown.</p>