Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422024-03-19T03:59:41ZArvados
Redmine Arvados - Feature #21606 (In Progress): configurable keep-web output buffer to reduce delay betwe...https://dev.arvados.org/issues/216062024-03-19T03:59:41ZTom Cleggtom@curii.com
<p>According to <a class="issue tracker-2 status-2 priority-4 priority-default parent" title="Feature: Go FileSystem / FUSE mount supports block prefetch (In Progress)" href="https://dev.arvados.org/issues/18961">#18961</a>, now that <a class="issue tracker-2 status-2 priority-4 priority-default parent" title="Feature: Keepstore can stream GET and PUT requests using keep-gateway API (In Progress)" href="https://dev.arvados.org/issues/2960">#2960</a> has reduced the TTFB for fetching a block, predicting and pre-fetching the next block appears to be more complex than it's worth.</p>
<p>Instead, in a typical scenario where the backend (keepstore→keep-web) bandwidth is faster than the frontend (keep-web→client), keep-web can reduce or eliminate the between-block delay by writing to an asynchronous output buffer. While keep-web is waiting a few milliseconds for the next block to start arriving from the backend, the client continues to receive the data that has accumulated in the output buffer.</p>
<p>The size of the output buffer should be configurable.</p> Arvados - Bug #21603 (In Progress): Not recognizing subnet error returned as InvalidParameterValuehttps://dev.arvados.org/issues/216032024-03-18T14:27:16ZPeter Amstutzpeter.amstutz@curii.com
<p><code>Mar 18 03:53:48 ip-172-25-144-184 arvados-dispatch-cloud[283002]: {"ClusterID":"xxxxx","InstanceType":"r52xlarge.preemptible","PID":283002,"error":"InvalidParameterValue: Not enough free addresses in subnet subnet-0f83ca79\n\tstatus code: 400, request id: 6cbcffe1-5b77-4dee-8fbf-c20f67892c95","level":"error","msg":"create failed","time":"2024-03-18T03:53:48.927972989Z"}</code></p>
<p>This is a subnet-specific error (it should switch to the other subnet) but the current function won't recognize it as such:</p>
<pre>
func isErrorSubnetSpecific(err error) bool {
aerr, ok := err.(awserr.Error)
if !ok {
return false
}
code := aerr.Code()
return strings.Contains(code, "Subnet") ||
code == "InsufficientInstanceCapacity" ||
code == "InsufficientVolumeCapacity" ||
code == "Unsupported"
}
</pre>
<p>Because the error was unrecognized, it seems the fallback behavior seems to be to rate limit itself by setting maximum concurrent containers.</p> Arvados - Bug #21601 (In Progress): fpm virtualenv packages not using branch versions for depende...https://dev.arvados.org/issues/216012024-03-15T20:38:09ZPeter Amstutzpeter.amstutz@curii.com
<p><a class="external" href="https://dev.arvados.org/issues/19744#note-30">https://dev.arvados.org/issues/19744#note-30</a></p>
<p>The python3-arvados-cwl-runner_2.8.0~dev20240314145937-1_amd64.deb package has arvados-python-client 2.7.1 and crunchstat-summary 2.7.1, when it should have the dev versions from the same commit.</p>
<p>I went back and looked at earlier packages: python3-arvados-cwl-runner_2.7.1~rc3-1_amd64.deb has arvados-python-client 2.7.1rc3 (as expected) and python3-arvados-cwl-runner_2.7.0~dev20230908133938-1_amd64.deb has arvados-python-client 2.7.0.dev20230908133938 (also as expected).</p>
<p>My current theory is that this behavior got lost in the changes made in 20846-package-build-fixes, but I need to find out how it worked before.</p> Arvados - Bug #21412 (In Progress): User profile bugs on refreshhttps://dev.arvados.org/issues/214122024-01-24T18:02:15ZLisa Knox
<p>When viewing a User Profile page, if you refresh, the breadcrumbs reset to "Users > [uuid]" when previously they said either "Users > [full name]" or "Groups > [Group Name] > [full name]". When viewing another user's home project page, if you refresh, it shows the "project not found" screen.</p> Tapestry - Feature #8707 (In Progress): Arvados job: download data from remote site into Keephttps://dev.arvados.org/issues/87072016-03-15T21:35:13ZTom Cleggtom@curii.com
<p>...to satisfy an API request like <a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: Accept authenticated API calls from data providers to add datasets to a public profile (Resolved)" href="https://dev.arvados.org/issues/8688">#8688</a></p>
<a name="Implementation"></a>
<h2 >Implementation<a href="#Implementation" class="wiki-anchor">¶</a></h2>
<p>One task per requested file -- this avoids retrying <em>everything</em> whenever one file fails</p>
<p>Use writable FUSE (task output dir)</p>
<p>Run wget or curl, probably with some sort of batch-progress flag</p> Tapestry - Feature #8697 (In Progress): Check for finished download/processing jobshttps://dev.arvados.org/issues/86972016-03-15T17:50:45ZTom Cleggtom@curii.com
<p>Implement as a long-running process that polls and then waits {configurable} seconds</p>
When a download job succeeds, create a UserFile record for each file in the output collection that
<ul>
<li>has location == output of download job</li>
<li>belongs to the appropriate participant</li>
<li>appears on the participant's public profile page</li>
<li>appears on the participant's "data files" list</li>
</ul>
When a processing job succeeds,
<ul>
<li>update report_content_locator = job output</li>
<li>send email to user</li>
<li>add entry to user log</li>
</ul>
When any job fails,
<ul>
<li>send email to site admins</li>
</ul>
<a name="Implementation"></a>
<h2 >Implementation<a href="#Implementation" class="wiki-anchor">¶</a></h2>
New UserFile attributes
<ul>
<li><pre>
t.integer "study_id"
</pre></li>
</ul> Tapestry - Feature #8689 (In Progress): Trigger an arvados job to retrieve the data and import it...https://dev.arvados.org/issues/86892016-03-15T00:28:10ZWard Vandewegeward@curii.com
When the API call described in <a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: Accept authenticated API calls from data providers to add datasets to a public profile (Resolved)" href="https://dev.arvados.org/issues/8688">#8688</a> is received, that should trigger
<ul>
<li>an email to the participant saying the transfer has been requested/started</li>
<li>a user log entry to that effect</li>
<li>an Arvados job that will ingest the data (when that is done, Tapestry needs to get notified or notice; see <a class="issue tracker-2 status-2 priority-4 priority-default parent" title="Feature: Check for finished download/processing jobs (In Progress)" href="https://dev.arvados.org/issues/8697">#8697</a>)</li>
</ul> GET-Evidence - Bug #5883 (New): migrate to updated oauthhttps://dev.arvados.org/issues/58832015-05-01T18:31:33ZAbram Connellyabram.connelly@gmail.com
<p>There is a warning on that says OAuth2 is going away. We need to migrate the authentication on GET-Evidence to make sure login is still possible.</p> GET-Evidence - Feature #5873 (New): GET-Evidence pipeline code should be publichttps://dev.arvados.org/issues/58732015-04-30T22:55:20ZAbram Connellyabram.connelly@gmail.com
<p>The GET-Evidence pipeline should be put under a public git repository.</p>
<p>Currently the GET-Evidence repository is under my (Abram Connelly) private Arvados git repository. This should probably remain but some hooks should be added somewhere so that the repository is pushed to a public Curoverse repository.</p> GET-Evidence - Bug #5872 (In Progress): 23andMe report pipeline is brokenhttps://dev.arvados.org/issues/58722015-04-30T22:52:53ZAbram Connellyabram.connelly@gmail.com
<p>When participants upload their 23andMe data, this kicks off a pipeline run through Arvados (now) through the same code path that the dataset submission goes through from Tapestry. The pipeline is failing to correctly generate reports for 23andMe data.</p>
<p>Porting the pipeline over to Arvados, the format for the input was changed and that's why the pipelines are failing on 23andMe data. I think the best thing to do going forward is to be more liberal about what input collection is specified so that 23andMe reports along with older reports can be run (or re-run) without having to worry about updating to the new format.</p>
<p>The old format just specifies a portable data hash. The new format expects <PDH>/<filename>.</p> Arvados - Idea #3304 (New): [Workbench] Add checkboxes to multi-type chooser modal to filter by o...https://dev.arvados.org/issues/33042014-07-18T21:04:07ZTom Cleggtom@curii.comArvados - Bug #3257 (In Progress): [Workbench] Hard to find user agreement after it has been signedhttps://dev.arvados.org/issues/32572014-07-16T14:33:47ZPjotr Prins
<p>After logging in the user agreement should be available in a straightforward way. Add an item to one of the dropdown menus at the top right.</p>
<p>There's a plan to consolidate these menus based on feedback from our UX/UI person. Tom can tell you more about that.</p>
<p>The user agreement is just a collection stored in Keep. Users can actually see it if they go to their 'Links' page and look for the entry with link class 'Signature'. But of course that's not user friendly.</p> Tapestry - Idea #2850 (New): Comprehensive "Sectioning" of Tapestryhttps://dev.arvados.org/issues/28502014-05-22T13:24:56ZPhil Hodgsonphil@curoverse.com
<p>Continuing the work done as proof of concept in Story <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: Make it possible to disable/enable "sections" (Resolved)" href="https://dev.arvados.org/issues/2503">#2503</a>, a comprensive sectioning off of Tapestry will force us to assess what parts of Tapestry are out of use and how the pieces fit together conceptually. The ideal will be a small set of "sections", where each one is optional.</p>
<p>The methodology is:</p>
<ol>
<li>Identify "modules" and assign config.yml keys</li>
<li>Reference configuration when rendering partials and cells</li>
<li>Reference configuration when authorizing in controllers</li>
</ol>
<p>At each point we risk encountering questions of architecture because it will be evident that some reorganization of the code being done at the same time as adding these "sections" would make sense. Therefore this is a non-trivial but important story overall.</p> Tapestry - Idea #2552 (New): Configurable mini-consent form and validationshttps://dev.arvados.org/issues/25522014-04-08T12:13:43ZPhil Hodgsonphil@curoverse.comTapestry - Idea #2544 (New): The "site override" feature is testedhttps://dev.arvados.org/issues/25442014-04-07T13:02:53ZPhil Hodgsonphil@curoverse.com