Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422022-04-11T20:24:42ZArvados
Redmine Arvados - Bug #18990 (New): should reflect the value of TLS/Insecure in the "Get API Token" dialoghttps://dev.arvados.org/issues/189902022-04-11T20:24:42ZWard Vandewegeward@curii.com
<p>When <code>TLS/Insecure</code> is set to <code>true</code>, the "Get API Token" dialog should say</p>
<pre><code>export ARVADOS_API_HOST_INSECURE=true</code></pre>
<p>and otherwise, it should say</p>
<pre><code>unset ARVADOS_API_HOST_INSECURE</code></pre>
<p>Currently, workbench2 always does the latter.</p> Arvados - Feature #18988 (New): [CWL] support singularity/docker hint to make debugging workflows...https://dev.arvados.org/issues/189882022-04-11T18:28:05ZWard Vandewegeward@curii.com
<p>The <code>arvados-client shell</code> feature only works when a workflow is executed with Docker, because Singularity has a different operating model.</p>
<p>It would be nice if a workflow could suggest which executor was to be used, so that interactive debugging becomes possible by switching to Docker.</p> Arvados - Feature #18970 (New): Add support for browsing as anonymous userhttps://dev.arvados.org/issues/189702022-04-04T15:13:10ZWard Vandewegeward@curii.com
<p>When the config says:</p>
<pre>
AllowAnonymousUserAccess: true
</pre>
Workbench2 should:
<ul>
<li>have a prominent "Browse public projects" link, even when ther user is not logged in</li>
<li>add the anonymous user to the "share with" options when objects are shared</li>
<li>add the anonymous group to the "share with" options when objects are shared</li>
</ul>
<p>When <code>AllowAnonymousUserAccess</code> is false, it should do none of these things.</p> Arvados - Feature #18944 (New): [controller] should log the user uuid used for the requesthttps://dev.arvados.org/issues/189442022-03-29T19:47:11ZWard Vandewegeward@curii.comArvados - Bug #18936 (New): [api] [controller] remove reader_token supporthttps://dev.arvados.org/issues/189362022-03-25T13:28:26ZWard Vandewegeward@curii.com
<p>Workbench 1 appends the anonymous user token in a "reader token" to each GET request to make sure that content shared with the anonymous user is available to authenticated users, even if not shared with explicitly with them.</p>
<p>Controller just appends any reader tokens received to the token list for the request.</p>
<p>API uses reader_tokens for GET requests in (services/api/app/controllers/application_controller.rb).</p>
<p>But it also does something else; in services/api/app/middlewares/arvados_api_token.rb it seems that if the primary session token is not valid, the first working reader token is used instead.</p>
<p>Workbench 2 does not use reader_tokens (which means authenticated users can not access data only shared with the anonymous user!).</p>
<p>Nothing else in our codebase appears to use reader_tokens.</p>
<p>Our documentation does not mention reader_tokens.</p>
<p><a class="issue tracker-2 status-1 priority-3 priority-lowest" title="Feature: [config] simplify AnonymousUserToken configuration (New)" href="https://dev.arvados.org/issues/18937">#18937</a> is about simplifying the anonymous token configuration - basically, doing away with the need for an anonymous token at all. Once that is done, we can remove the controller and API code that handles reader_tokens. Maybe log a warning if a reader token is used (though, as long as WB1 is around, that's going to generate a lot of noise in the logs)?</p> Arvados - Feature #18897 (New): [go services] should log the uuid of the token used for each requ...https://dev.arvados.org/issues/188972022-03-21T20:05:37ZWard Vandewegeward@curii.comArvados - Bug #18762 (New): rails background tasks scaling issueshttps://dev.arvados.org/issues/187622022-02-14T21:08:24ZWard Vandewegeward@curii.com
<p>The rails api has a few background threads that should only run once, even when multiple rails api instances are active.</p>
<p>- ward: fill in which tasks</p>
<p>Just like we did in <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: SweepTrashedObjects scaling issues (Resolved)" href="https://dev.arvados.org/issues/18339">#18339</a>, the existing background tasks in the rails api code should be put inside a mutex. Either move the code inside controller, or if that is hard, apply the same solution as in <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: SweepTrashedObjects scaling issues (Resolved)" href="https://dev.arvados.org/issues/18339">#18339</a>.</p> Arvados - Feature #18672 (New): [go sdk] describe + implement desired Go SDKhttps://dev.arvados.org/issues/186722022-01-24T21:25:54ZWard Vandewegeward@curii.com
<p>Something like</p>
<pre>
import (
"git.arvados.org/arvados.git/lib/arvados"
)
</pre>
<pre>
// Get an Arvados API object (cf. https://pkg.go.dev/git.arvados.org/arvados.git@v0.0.0-20220124190027-e372194dc9b4/sdk/go/arvados#API)
arv := arvados.NewClient()
</pre>
<p>We can then add examples for all the functions available on the API object.</p>
<p>We can rename all the other types in the arvados module? Or move them elsewhere? Or at least make the godoc index more indented/easier to read?</p> Arvados - Bug #18671 (New): [go sdk] update documentationhttps://dev.arvados.org/issues/186712022-01-24T21:09:44ZWard Vandewegeward@curii.com
<p>The documentation at </p>
<pre><code><a class="external" href="https://doc.arvados.org/sdk/go/index.html">https://doc.arvados.org/sdk/go/index.html</a><br /><a class="external" href="https://doc.arvados.org/sdk/go/example.html">https://doc.arvados.org/sdk/go/example.html</a></code></pre>
<p>refers to the old Go SDK. The godoc link for the go/sdk/arvados directory describes the current SDK. We want to move over to an RPC interface as per <a class="issue tracker-2 status-1 priority-4 priority-default" title="Feature: [go sdk] describe + implement desired Go SDK (New)" href="https://dev.arvados.org/issues/18672">#18672</a>.</p>
<ul>
<li>The godoc should be updated/improved to incorporate all the examples from the examples page at our documentation site.</li>
<li>Examples should be added for important features (e.g. the CollectionFileSystem)</li>
<li>The old pages should be removed from the Arvados documentation, with only the godoc link remaining. Any Go programmer should be able to use the Arvados Go sdk with just the godoc page.</li>
</ul> Arvados - Task #18669 (New): review 18668-add-project-support-to-deduplication-reporthttps://dev.arvados.org/issues/186692022-01-24T16:50:40ZWard Vandewegeward@curii.comArvados - Bug #18618 (New): Reusing workflows/steps is too slowhttps://dev.arvados.org/issues/186182022-01-07T15:26:37ZWard Vandewegeward@curii.com
<p>Arvados takes too long to figure out if a workflow or step has already been run and can be reused.</p>
<p>A user reported that it can take ~1 minute for that determination to be made.</p> Arvados - Feature #18564 (New): [art] run jenkins release build steps with a set of parametershttps://dev.arvados.org/issues/185642021-12-08T15:43:29ZWard Vandewegeward@curii.com
<p>Maybe something like this:</p>
<p>1. Start a release with version number and git commits<br />2. Make it possible to run the appropriate jenkins jobs with those settings, individually<br />3. As part of 2, automatically update the redmine release ticket with the jenkins run IDs and version number / git commit</p> Arvados - Feature #18563 (New): Simplify/streamline InternalURLs/ExternalURL situationhttps://dev.arvados.org/issues/185632021-12-07T21:58:52ZWard Vandewegeward@curii.com
<p>As we struggled with documenting the precise meaning of these configuration keys in <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: Install docs explains InternalURL / ExternalURL, private networks & split DNS (Resolved)" href="https://dev.arvados.org/issues/17667">#17667</a>, it became clear that there is room for improvement here.</p>
<p>Starting from the documentation of the current state (<a class="external" href="https://doc.arvados.org/v2.3/admin/config-urls.html">https://doc.arvados.org/v2.3/admin/config-urls.html</a>) and the ideas/code in <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: Install docs explains InternalURL / ExternalURL, private networks & split DNS (Resolved)" href="https://dev.arvados.org/issues/17667">#17667</a> (e.g. <a class="external" href="https://dev.arvados.org/issues/17667#note-11">https://dev.arvados.org/issues/17667#note-11</a>), come up with a plan for simplification and implement it.</p>
<p>The documentation will need to be updated as well.</p> Arvados - Bug #18393 (New): [workbench2] forces relogin on every new window/tabhttps://dev.arvados.org/issues/183932021-11-19T14:37:25ZWard Vandewegeward@curii.com
<p>How to reproduce:</p>
<p>1. open a new browser window or tab for <a class="external" href="https://workbench2.ce8i5.arvadosapi.com">https://workbench2.ce8i5.arvadosapi.com</a>. Log in.<br />2. open another browser window or tab for <a class="external" href="https://workbench2.ce8i5.arvadosapi.com">https://workbench2.ce8i5.arvadosapi.com</a>. The login page is shown again.</p>
<p>Observed on ce8i5 which is configured with direct Google authentication, and is the login cluster for a login federation. Relevant config:</p>
<pre>
...
RemoteClusters:
ce8i5:
Host: ce8i5.arvadosapi.com
Proxy: true
ActivateUsers: true
tordo:
Host: tordo.arvadosapi.com
Proxy: true
ActivateUsers: true
9tee4:
Host: 9tee4.arvadosapi.com
Proxy: true
ActivateUsers: true
API:
MaxTokenLifetime: 24h
Login:
LoginCluster: ce8i5
# TokenLifetime: 8h
Google:
Enable: true
AlternateEmailAddresses: true
...
</pre>
<p>Not seeing this on tordo or 9tee4.</p> Arvados - Bug #18385 (New): arvados-server config-dump | arvados-server config-check -config=- sp...https://dev.arvados.org/issues/183852021-11-16T20:58:23ZWard Vandewegeward@curii.com
<p>This was observed on ce8i5 as part of keepstore-on-the-compute-node testing in <a class="issue tracker-3 status-3 priority-4 priority-default closed" title="Support: Run chr19 WGS test on ce8i5 to test compute-local keepstore (Resolved)" href="https://dev.arvados.org/issues/18320">#18320</a> (the `keepstore.txt` file listed the about below as warnings). We reproduced it with `arvados-server config-dump | arvados-server config-check`:</p>
<pre>
arvados-server config-dump | arvados-server config-check -config=-
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.d2asv4.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.d2asv4.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.e4asv4.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.e4asv4.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.e8asv4.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.e8asv4.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.e2asv4.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.e2asv4.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.a1v2.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.a1v2.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.a2v2.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.a2v2.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.a4v2.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.a4v2.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.d4asv4.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.d4asv4.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.d8asv4.Scratch"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.InstanceTypes.d8asv4.Name"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.Users.NewInactiveUserNotificationRecipients.REDACTED@curii.com"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.Users.NewUserNotificationRecipients.REDACTED@curii.com"
time="2021-11-16T20:57:41Z" level=warning msg="deprecated or unknown config entry: Clusters.ce8i5.Collections.ManagedProperties.responsible_person_uuid.Value"
</pre>
<p>All of these warnings are wrong. The keys are not invalid, and none of the InstanceType definitions have a `Name` or `Scratch` field defined.</p>
<p>Note that running `arvados-server config-check` by itself produces no output, as expected. The problem must be in the output generated by config-dump, e.g. this is the `InstanceTypes` section:</p>
<pre>
InstanceTypes:
a1v2:
AddedScratch: 0
IncludedScratch: 10000000000
Name: a1v2
Preemptible: false
Price: 0.043
ProviderType: Standard_A1_v2
RAM: 2147483648
Scratch: 10000000000
VCPUs: 1
a2v2:
AddedScratch: 0
IncludedScratch: 20000000000
Name: a2v2
Preemptible: false
Price: 0.091
ProviderType: Standard_A2_v2
RAM: 4294967296
Scratch: 20000000000
VCPUs: 2
...
</pre>
<p>The actual config file has:</p>
<pre>
InstanceTypes:
a1v2:
ProviderType: Standard_A1_v2
VCPUs: 1
RAM: 2GiB
IncludedScratch: 10GB
Price: 0.043
a2v2:
ProviderType: Standard_A2_v2
VCPUs: 2
RAM: 4GiB
IncludedScratch: 20GB
Price: 0.091
...
</pre>
<p>Here's the `config-dump` output for the other keys that generate warnings:</p>
<pre>
ManagedProperties:
responsible_person_uuid:
Function: original_owner
Protected: true
Value: null
...
NewInactiveUserNotificationRecipients:
REDACTED@curii.com: {}
NewUserNotificationRecipients:
REDACTED@curii.com: {}
</pre>