https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422017-04-11T19:37:39ZArvadosArvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=505832017-04-11T19:37:39ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/50583/diff?detail_id=48791">diff</a>)</li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=544392017-08-28T21:52:17ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>Arvados Future Sprints</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=560932017-10-17T18:49:33ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/56093/diff?detail_id=53766">diff</a>)</li><li><strong>Story points</strong> set to <i>2.0</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=581632017-12-06T20:20:58ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2017-12-20 Sprint</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=581672017-12-06T20:29:09ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=588952018-01-02T16:12:35ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2017-12-20 Sprint</i> to <i>2018-01-17 Sprint</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=594122018-01-15T21:40:34ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=594132018-01-15T22:06:20ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Should the configuration parameters be published from the home API server or at the workbench level?</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=594472018-01-17T19:16:23ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2018-01-17 Sprint</i> to <i>2018-01-31 Sprint</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=595992018-01-22T21:11:26ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="11454: Conditional login to remote API servers. When adding new sites to the multi site search, ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/95c228e25cd1a483ec391a9b2c5d6fb048f620f1">95c228e25</a></p>
<ul>
<li>Added a javascript SHA1 library to be able to create salted tokens from the multi site search.</li>
<li>Modified the <code>SessionDB.login()</code> function to use a salted token when necessary.</li>
<li>Fixed <code>SessionDB.fillMissingUUIDs()</code> function to use <code>user.owner_uuid</code> as a source to get the remote cluster's prefix uuid instead of <code>user.uuid</code> that can be from another cluster.</li>
<li>Added a way to pass a list of remote hosts to the workbench. <strong>Pending</strong>: auto-login to those remote hosts using a salted token if reviewer confirms that's a correct behavior.</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=596052018-01-22T21:48:00ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Also pending: configuration to avoid adding sites not listed on <code>remote_hosts</code> via DNS fallback, should these 2 configurations (remote_hosts & remote_hosts_via_dns) be added to workbench as they are meant to be on the "home cluster" of a federation?</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=596602018-01-24T17:17:41ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Testing procedure:</p>
Currently <code>9tee4</code> & <code>c97qk</code> are configured with federation enabled (see #12956), both clusters accept accounts each other.
<ul>
<li>Log into any of these clusters (let's say, <code>c97qk</code>) using the branch workbench.</li>
<li>Go to <code>/search</code> url (multi site search) & check if <code>9tee4</code> is already added to the multi site search.
<ul>
<li>If that's the case, you should clear the local storage cache as it'll have conflicts with the new version.</li>
</ul>
</li>
<li>Go to the "add site" section, adding <code>9tee4.arvadosapi.com</code>.
<ul>
<li>What's expected is that you'll log into <code>9tee4</code> using your <code>c97qk</code>'s account.
<ul>
<li>If the account wasn't already created on <code>9tee4</code>, it'll get created but not activated: You'll land on the "account inactive" page on <code>9tee4</code> and will need to be activated manually before retrying.</li>
<li>If the account was already created & active, you'll get logged in automatically without being asked for your login credentials, as the log in procedure is made using a salted token from <code>c97qk</code></li>
</ul></li>
</ul></li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=596612018-01-24T17:36:04ZTom Morristfmorris@veritasgenetics.com
<ul></ul><p>Can we offer the user the option of clearing local storage from the UI instead of forcing them to use the browser interface?</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=596972018-01-25T18:21:01ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><ul>
<li>Putting in just "9tee4" doesn't work.</li>
<li>When you put in a bad hostname, it takes a long time to tell you that it doesn't work.</li>
<li>It successfully creates a federated user on the remote cluster, but the remote account is inactive. There is no indication that the account is inactive.</li>
<li>If you log out of the remote workbench, there is no way to log in again.</li>
<li>Clicking through to the remote cluster takes you to the "not found, try logging in" page.</li>
<li>The "sessions" page should have a button that goes to the remote workbench dashboard with your federated identity.</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=596982018-01-25T18:26:40ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>To be clear, the actual federated search seems to work, the main problem is integration with remote workbench.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=597112018-01-25T22:02:15ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Some notes related to discussions on chat and more:</p>
<ul>
<li>Prototype implementation aimed to take advantage of the salted token to login automatically using the same method that’s being used currently
<ul>
<li>This has the side effect of saving the user’s “normal” token on <code>SessionDB</code> instead of just using the salted token for searches. (as discussed on chat, this is not desirable)</li>
</ul>
</li>
<li>The app should ask (and cache) the API server for the local account token’s uuid on startup, to have it available for generating salted tokens.</li>
<li>We need a way to distinguish between federated and non-federated clusters, as federated clusters just accept the salted token versus the other ones need a login procedure.
<ul>
<li>Maybe if we define that <code>remote_hosts</code> clusters are the ones that are federated and the ones added through the “add/remove site” UI are the non-federated clusters, it would be easy to implement because we don’t need to check if a federated search is successful before retrying a classic login on a manually added site.</li>
<li>Other way is getting the remote cluster discovery doc and see if the local one is listed on the <code>remote_hosts</code> setting, or if <code>remote_hosts_via_dns</code> is true. This implies a request delay.</li>
</ul>
</li>
<li>As discussed on the chat, discovery doc’s <code>remote_hosts</code> & <code>remote_hosts_via_dns</code> settings should be used to set up home cluster’s multi-site search app, taking into consideration that the home cluster would be also allowing logins from the listed clusters’ accounts.</li>
<li>I suppose that the current behavior should be maintained when <code>remote_hosts_via_dns</code> is set to true, otherwise it shouldn’t allow adding remote sites that aren’t listed on <code>remote_hosts</code>, please confirm this.</li>
<li>As Peter pointed out, links to results on federated workbenches should include ‘<code>?api_token=v2/…</code>” salted token so that the user can auto-login to the destination workbench when clicking.</li>
<li>What happens if an old multi site search session has non-federated sessions already saved? This produces strange behavior on the current session management UI.
<ul>
<li>Also, this brings up the question: Should the user be able to have a federated & non-federated session to the same remote cluster?</li>
</ul></li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=597602018-01-30T15:02:07ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Some notes related to discussions on chat and more:</p>
<ul>
<li>Prototype implementation aimed to take advantage of the salted token to login automatically using the same method that’s being used currently
<ul>
<li>This has the side effect of saving the user’s “normal” token on <code>SessionDB</code> instead of just using the salted token for searches. (as discussed on chat, this is not desirable)</li>
</ul></li>
</ul>
</blockquote>
<p>Yes, we should use salted tokens only, which means we need the token uuid.</p>
<blockquote>
<ul>
<li>We need a way to distinguish between federated and non-federated clusters, as federated clusters just accept the salted token versus the other ones need a login procedure.</li>
</ul>
</blockquote>
<p>I think we attempt the salted token login, and if that fails, try normal login. If salted token login takes a long time to fail, that's something we need to address directly.</p>
<blockquote>
<ul>
<li>As discussed on the chat, discovery doc’s <code>remote_hosts</code> & <code>remote_hosts_via_dns</code> settings should be used to set up home cluster’s multi-site search app, taking into consideration that the home cluster would be also allowing logins from the listed clusters’ accounts.</li>
</ul>
</blockquote>
<p>Yes, I think we agreed that was a simplifying assumption.</p>
<blockquote>
<ul>
<li>I suppose that the current behavior should be maintained when <code>remote_hosts_via_dns</code> is set to true, otherwise it shouldn’t allow adding remote sites that aren’t listed on <code>remote_hosts</code>, please confirm this.</li>
</ul>
</blockquote>
<p>I agree.</p>
<blockquote>
<ul>
<li>As Peter pointed out, links to results on federated workbenches should include ‘<code>?api_token=v2/…</code>” salted token so that the user can auto-login to the destination workbench when clicking.</li>
</ul>
</blockquote>
<p>Yes. I would also like to request we add a remote workbench link on the "sessions" page.</p>
<blockquote>
<ul>
<li>What happens if an old multi site search session has non-federated sessions already saved? This produces strange behavior on the current session management UI.</li>
</ul>
</blockquote>
<p>Can we detect this and reset it? The user will just have to set it up again.</p>
<blockquote>
<ul>
<li>Also, this brings up the question: Should the user be able to have a federated & non-federated session to the same remote cluster?</li>
</ul>
</blockquote>
<p>Probably not. Even if a user wants to have multiple accounts, this interface probably won't be the right way to manage it.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=597642018-01-30T16:37:15ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Further question from Peter's answers & confirmations:</p>
Clarified requirements:
<ul>
<li><code>remote_hosts</code> & <code>remote_hosts_via_dns</code> on the “home cluster” will be used to set up the multi site search app behavior. Current app behavior will be maintained when <code>remote_hosts_via_dns=true</code>.</li>
<li>Links to remote workbenches will use api_token=v2/… on their urls and the app’s session page will also have a link to every workbench with it.</li>
</ul>
Questions:
<ul>
<li>We should use salted tokens only, so we need the token uuid: We could get it by querying the API server from the Mithril app (like it’s being done on the branch currently) or pass it through the mount tag when loading the app. Thoughts/preference?</li>
<li>About using the salted token and having a fallback to the usual login method: The salted token method does not involve login in, so we would need to make some no-op operation with the salted token on app initialization to check if it works or mark the session as inactive instead of redirecting the browser to a login page when some search fails with the salted token, right?</li>
<li>On app initialization, check if active non-federated sessions on SessionDB exists on clusters listed on <code>remote_hosts</code>. Reset any of these kind of sessions if the fact that some cluster is listed there implicitly tells the system that federated mode should be used.</li>
<li>If federated & non-federated sessions shouldn’t happen on the same app, <code>remote_hosts_via_dns</code> setting will behave differently when <code>remote_hosts</code> is empty (non-federated case) versus when it has some value? Or do we need an additional boolean setting that selects federated / non-federated modes?</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=597692018-01-30T18:50:37ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Further question from Peter's answers & confirmations:</p>
Clarified requirements:
<ul>
<li><code>remote_hosts</code> & <code>remote_hosts_via_dns</code> on the “home cluster” will be used to set up the multi site search app behavior. Current app behavior will be maintained when <code>remote_hosts_via_dns=true</code>.</li>
<li>Links to remote workbenches will use api_token=v2/… on their urls and the app’s session page will also have a link to every workbench with it.</li>
</ul>
Questions:
<ul>
<li>We should use salted tokens only, so we need the token uuid: We could get it by querying the API server from the Mithril app (like it’s being done on the branch currently) or pass it through the mount tag when loading the app. Thoughts/preference?</li>
</ul>
</blockquote>
<p>Querying it from Mithril is probably better, the less it relies on Rails the better.</p>
<blockquote>
<ul>
<li>About using the salted token and having a fallback to the usual login method: The salted token method does not involve login in, so we would need to make some no-op operation with the salted token on app initialization to check if it works or mark the session as inactive instead of redirecting the browser to a login page when some search fails with the salted token, right?</li>
</ul>
</blockquote>
<p>It should request "/arvados/v1/users/current" to test if a salted token works.</p>
<blockquote>
<ul>
<li>On app initialization, check if active non-federated sessions on SessionDB exists on clusters listed on <code>remote_hosts</code>. Reset any of these kind of sessions if the fact that some cluster is listed there implicitly tells the system that federated mode should be used.</li>
</ul>
</blockquote>
<p>That makes sense.</p>
<blockquote>
<ul>
<li>If federated & non-federated sessions shouldn’t happen on the same app, <code>remote_hosts_via_dns</code> setting will behave differently when <code>remote_hosts</code> is empty (non-federated case) versus when it has some value? Or do we need an additional boolean setting that selects federated / non-federated modes?</li>
</ul>
</blockquote>
<p>I think having federated & non-federated sessions are okay, but there should only be one session per remote cluster. Also, for clusters listed in <code>remote_hosts</code>, only federated sessions should be allowed. We can always determine a given cluster's prefix by querying the discovery document.</p>
<p>Additional thought: what happens if a user visiting a foreign workbench with a remote token goes to multi-site search or /sessions? Maybe that page should only be available to local users?</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=597712018-01-30T20:03:01ZTom Cleggtom@curii.com
<ul></ul><p>Re token UUID: yes, any new API call we need to do should be done from the client, not the Rails app.</p>
<p>Re testing a salted token: yes, get /arvados/v1/users/current</p>
<p>Re auto-replacing old non-federated sessions: given the migration stuff, yes, it might be really useful to test a salted token on each remote, and if that works, replace the non-federated session with the salted token. One less opportunity for users to wonder why they're not getting any search results after the admin migrates their accounts.</p>
<p>Mixing federated and non-federated seems fine. I think we should support this.</p>
<p>Current behavior</p>
<table>
<tr>
<td>current token </td>
<td>search target </td>
<td>session always listed? </td>
<td>can add session? </td>
<td>can disable session? </td>
<td>use token </td>
<td>notes </td>
</tr>
<tr>
<td>local </td>
<td>this cluster </td>
<td>yes </td>
<td>- </td>
<td>- </td>
<td>current </td>
<td> </td>
</tr>
<tr>
<td>local </td>
<td>remote cluster</td>
<td>no </td>
<td>yes </td>
<td>yes </td>
<td>obtain via login </td>
<td> </td>
</tr>
<tr>
<td>salted </td>
<td>this cluster </td>
<td>yes </td>
<td>- </td>
<td>- </td>
<td>current </td>
<td>bug <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [Federation] Wrong cluster ID shown on multisite search page (Resolved)" href="https://dev.arvados.org/issues/12959">#12959</a> </td>
</tr>
</table>
<p>Desired behavior</p>
<table>
<tr>
<td>current token </td>
<td>search target </td>
<td>session always listed? </td>
<td>can add session? </td>
<td>can disable session? </td>
<td>use token </td>
<td>notes </td>
</tr>
<tr>
<td>local </td>
<td>this cluster </td>
<td>yes </td>
<td>- </td>
<td>- </td>
<td>current </td>
<td> </td>
</tr>
<tr>
<td>local </td>
<td>configured remote </td>
<td>yes? </td>
<td>- </td>
<td>- </td>
<td>salted </td>
<td> </td>
</tr>
<tr>
<td>local </td>
<td>configured remote via dns </td>
<td>no </td>
<td>yes </td>
<td>yes </td>
<td>salted or via login </td>
<td> </td>
</tr>
<tr>
<td>local </td>
<td>user-entered remote </td>
<td>no </td>
<td>yes </td>
<td>yes </td>
<td>salted or via login </td>
<td> </td>
</tr>
<tr>
<td>salted </td>
<td>this cluster </td>
<td>yes </td>
<td>- </td>
<td>- </td>
<td>current </td>
<td>bug <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [Federation] Wrong cluster ID shown on multisite search page (Resolved)" href="https://dev.arvados.org/issues/12959">#12959</a> </td>
</tr>
</table>
As for whether to fall back to the login process for sites that don't accept a salted token: I think we might need to leave this up to the user, because there are various reasons for a salted token to fail, and not all of them mean the user wants to use a different account on that site. Perhaps:
<ul>
<li>If site is specifically listed in discovery.remoteHosts, add it to the sessions list</li>
<li>If user types hostname into text box (or xyzzy as a shortcut for xyzzy.arvadosapi.com), add it to the sessions list</li>
<li>For each site in the list, try a salted token</li>
<li>If site accepts salted token, enable it, using the salted token</li>
<li>If site listed in discovery.remoteHosts does not accept salted token, disable it (maybe an "enable" button just retries with the salted token?)</li>
<li>If user-entered site does not accept salted token, and we have a stored remote session token, use the stored remote session token (just like old behavior)</li>
<li>If user-entered site does not accept salted token, and we don't have a stored remote session token, offer a "Log in to remote" button</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=597942018-02-01T14:39:03ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Working on this story I think I might have stumbled into a (couple?) bug:</p>
<p>Having a salted token on an inactive account allows me to re-active it by using the token to login to workbench. Let's say I have clusterH (home) & clusterR (remote), and my clusterH account on clusterR is inactive, if I go to the link <code>https://workbench.clusterR.arvadosapi.com/?api_token=v2/clusterHsaltedtoken</code>, it logs me in without any issue, and the account is suddenly active.</p>
<p>Also, I was able to do a search using the salted token on an inactive account.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=597972018-02-01T15:54:28ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-7 priority-4 priority-default closed parent" href="/issues/13020">Bug #13020</a>: Login into workbench with a salted token activates a previously inactive federated account</i> added</li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598162018-02-01T20:36:47ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2018-01-31 Sprint</i> to <i>2018-02-14 Sprint</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598692018-02-02T14:54:37ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> deleted (<del><i>Lucas Di Pentima</i></del>)</li></ul><p>Another thought, when you are a federated user on remote cluster workbench, the multi-site search button should always take you to your home cluster (you can't do federated search from anywhere else).</p>
<p>What happens if you visit the /sessions page as a federated user on a remote cluster? It should probably say something like "can only set sessions on home cluster" and give you a link to your home workbench.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598702018-02-02T14:58:27ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>On a process note, I know we all hate adding requirements in the review process, but linking between home and remote workbench instances is an absolutely critical aspect of the story that was overlooked in the original story description. Without it, federated identity / remote search is unusable.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598712018-02-02T14:58:55ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598722018-02-02T15:01:03ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="11454: Removed unnecessary call on migrateNonFederatedSessions() Added auto loading of remote hos..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0e3c1f9c71f537d000459cbd3159ca6c198461c8">0e3c1f9c7</a></p>
Changes:
<ul>
<li>Do federated logins by requesting the current user object instead of the classic login flow.</li>
<li>When the user is inactive on the remote cluster, label the session properly and exclude it from SessionDB.loadActive()'s results.</li>
<li>Show a link to the remote workbench on the sessions page.</li>
<li>On page load, detect those non-federated active sessions and try to migrate them to federated logins just in case they're stale sessions. If the federated login doesn't succeed, keep the session's previous state.</li>
<li>Removed workbench remote_hosts config.</li>
<li>Added auto loading of remote hosts listed on home cluster's api discovery doc. (WIP - there are some UI issues as this operation is asynchronous and the search and sessions pages don't always update correctly)</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598732018-02-02T15:48:34ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>What happens if you visit the /sessions page as a federated user on a remote cluster? It should probably say something like "can only set sessions on home cluster" and give you a link to your home workbench.</p>
</blockquote>
<p>I tried this, and the results are incredibly confusing.</p>
<ol>
<li>I logged in to c97qk</li>
<li>I added 9tee4 as a federated identity</li>
<li>I went to the 9tee4 workbench, and clicked on multi-site search. Despite being on the 9tee4 workbench, only "c97qk" showed up as a search option.</li>
<li>I tried adding 9tee4 on the remote session. (I realize now, since the remote workbench isn't running this branch, it isn't federation aware)</li>
<li>The sessions page reloaded showing that it would search of 9tee4 but not c97qk. On subsequent investigation I realized that adding 9tee4 had the result of <em>logging me in as a local user</em> to 9tee4 (because I already have a user account.)</li>
</ol>
<p>I think even if both clusters were federation aware, we would still end up with some strange behavior because the federated user on the remote workbench can't do federated logins to other clusters (it doesn't have access to the original token). Then it falls back to regular login which results in being logged in as a different user, and not the federated user you intended.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598752018-02-02T16:19:55ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>The search links are missing "?api_token=" so if you log out from a remote workbench and then try to follow the search link it won't work.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598762018-02-02T16:33:08ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>When you are a federated user on a remote workbench, the /search and /sessions pages should either say "click here to go to your home cluster" or just automatically redirect there.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=598842018-02-02T22:38:06ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="11454: On /search & /sessions page, check if the current user is from a remote cluster, and if th..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0312a5e5aa076fa3458dd83a33e72663d23bb1ba">0312a5e5a</a></p>
<ul>
<li>Remote search result links now have <code>?api_token=v2/...</code> on their URL.</li>
<li>Automatic redirection when a remote user tries to get to <code>/search</code> or <code>/sessions</code> pages.</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=599282018-02-05T21:49:02ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="11454: Remotes listed on remoteHosts show "enable/disable" action button instead of "Log in/Log o..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/1b350dc780c38740819ff0dbb495557742aabd1a">1b350dc78</a></p>
<ul>
<li>When using just a uuid prefix on the "Log in" box, append <code>.arvadosapi.com</code> before trying to get the API server url.</li>
<li>Remotes listed on remoteHosts show "enable/disable" action button instead of "Log in/Log out".</li>
<li>When a listed federated remote rejects the salted token, the session shows as disabled.</li>
<li>When a user-entered (ie: not listed) federated remote rejects the salted token, classic login is attempted.</li>
</ul>
<p>To test automatic redirection described on note-31, you can have the branch workbench pointing to <code>c97qk</code>, add <code>9tee4</code> as a remote, and get one of the search result links pointing to <code>9tee4</code>, then restart workbench pointing to <code>9tee4</code>, modify the link accordingly to use it with localhost and log in using the salted token. When going to <code>/search</code> or <code>/sessions</code> you'll get redirected to <code>workbench.c97qk.arvadosapi.com/...</code>.<br />I tried to set up 2 wb instances on localhost IP addresses and make the <code>/etc/hosts</code> file to point the real domains to the local IPs but couldn't make it work properly.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=599702018-02-07T14:14:11ZTom Cleggtom@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed parent" href="/issues/12959">Bug #12959</a>: [Federation] Wrong cluster ID shown on multisite search page</i> added</li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=599832018-02-07T16:35:20ZTom Cleggtom@curii.com
<ul></ul><p>Did a quick test, everything works.</p>
<p>It looks like migrateNonFederatedSessions happens on every page load. Would it be easy enough to run it only when the "sessions" page is loaded -- e.g., in SessionsTable.oninit? (Unless I'm forgetting something, it nearly always ends up achieving nothing, and even in cases where a session is migrate-able, it's not really necessary to migrate right away, because the old session will still work. The only real reason for migrating is to make the /sessions page stay consistent across all users as clusters get upgraded/reconfigured. Is that right?)</p>
Adding <code>?api_token=...</code> to "show collection" links seems like a significant security problem (worse than the other places we do this): if I right-click a "show" button and hit "copy link address" and paste it in chat -- which seems like a reasonable/likely thing to do -- anyone who clicks it will be using my account on that cluster. They won't even realize that they've switched to my account unless they check the "logged in" dropdown or notice all the content they couldn't see before. I think we should make a real effort to avoid this. Some ideas:
<ul>
<li>use ajax reqs to check for an existing session and, if needed, create one (this might be hard/impossible due to browser security model, and might involve some CORS support on the remote workbench)</li>
<li>drop the query string from the href attr (so "copy link" doesn't have it), but intercept the onclick handler to pass it when clicking the link (this might be hard to do without breaking shift/ctrl-click)</li>
<li>pass the cluster ID or API host, but not a token. The remote workbench can handle the "no session" case by initiating login through the indicated cluster. Pasted links would behave strangely for users who don't have an account on the same "home" cluster, but at least they wouldn't be a gaping security hole.</li>
<li>link to a local page (without a token), and have the local page add the <code>?api_token=</code> part and redirect.</li>
</ul>
<p>A session for a remote site that's listed in remote_hosts should not offer a "Remove" button. (Currently, I can hit "remove", and it disappears from the list, but then it gets re-added automatically when I reload.)</p>
<p>"Disable" doesn't survive reload. Maybe sessions need a "disabled" flag to remember this state, or we need to avoid automatically adding a salted token to a session whose token has been cleared? Alternatively, we could just hide the button entirely -- but this would mean a user can't disable an unresponsive/irrelevant cluster that's interfering with searches.</p>
<p>Could we call db.autoRedirectToHomeCluster() from SessionsTable.oninit and Search.oninit, which already have a db, rather than adding new Rails html templates and creating additional SessionDB objects?</p>
<p>Currently SearchTable.view instantiates a SessionDB. This means it runs in every render cycle! But Search.oninit already creates a session db and keeps it around for the life of the page. Instead of adding all the session stuff in SearchTable, just have Search pass along the appropriate tokenParam the same way it passes along objectType and workbenchBaseURL.</p>
<p>(TBC. I haven't had a proper look at session_db.js yet.)</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=599902018-02-07T17:55:22ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Regarding ?api_token, what about using a form post button instead of a link? It would supply api_token and _method=GET as hidden fields.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600152018-02-07T20:38:07ZTom Cleggtom@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>Regarding ?api_token, what about using a form post button instead of a link? It would supply api_token and _method=GET as hidden fields.</p>
</blockquote>
<p>I think this is the easiest fix. Drawback: it would also prevent the "copy/paste link" operation entirely, as well as breaking shift/ctrl-click.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600372018-02-08T16:57:36ZTom Cleggtom@curii.com
<ul></ul><p><code>apiHostname.indexOf('arvadosapi.com') >= 0</code> ...should be... <code>apiHostname.endsWith('.arvadosapi.com')</code></p>
<p><code>session.token.indexOf('v2/') < 0</code> is being used as a test for "not a salted token" (therefore try migrating), but it really only tests token format, so it'll stop working when the remote starts handing out v2 tokens. Maybe the condition we're really interested in here is <code>session.token != saltedToken(uuidPrefix)</code> ...?</p>
<p>Although <code>session.user.owner_uuid.slice(0,5)</code> should be equal to it anyway, would it be clearer to use localDD.uuidPrefix?</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600482018-02-08T22:13:57ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="11454: Code cleanup. Removed local wb link on sessions page. Arvados-DCO-1.1-Signed-off-by: Luca..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/71e1465f6c7aa20a70bf46e56ac20f8b74423d4a">71e1465f6</a><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/570/">https://ci.curoverse.com/job/developer-run-tests/570/</a></p>
<ul>
<li>Changed search results links to form buttons.</li>
<li>Addressed comments on note-37.</li>
<li>Removed useless local workbench link on the <code>/sessions</code> page.</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600492018-02-09T02:28:20ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="11454: Replaced javascript events with proper form values on the search results form buttons. Ar..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/80eedd862b4956ed840af233dce0c3608fe627b2">80eedd862</a></p>
<p>Replaced javascript events with proper form values on the search results form buttons.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600722018-02-09T20:05:43ZTom Cleggtom@curii.com
<ul></ul><p>Even though we are fairly certain tokenParam will never have square brackets etc., we should encourage good habits:</p>
<pre><code class="javascript syntaxhl"><span class="o">-</span> <span class="nx">m</span><span class="p">(</span><span class="dl">'</span><span class="s1">input[type=hidden][name=api_token][value=</span><span class="dl">'</span><span class="o">+</span><span class="nx">tokenParam</span><span class="o">+</span><span class="dl">'</span><span class="s1">]</span><span class="dl">'</span><span class="p">),</span>
<span class="o">+</span> <span class="nx">m</span><span class="p">(</span><span class="dl">'</span><span class="s1">input[type=hidden][name=api_token]</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span><span class="na">value</span><span class="p">:</span> <span class="nx">tokenParam</span><span class="p">}),</span>
</code></pre>
The "link to remote workbench" on the session page...
<ul>
<li>has the same copy-paste-secret problem again (let's solve it the same way)</li>
<li>has a tooltip that basically lies about the link target, which seems weird</li>
<li>seems misplaced/unnecessary. If we need a link, we can add an explicit "visit remote Workbench" button somewhere, but I don't think overloading the "cluster ID" label is the way to do it. The point of this page is to use multiple clusters from a single Workbench. What's the motivation for having such a link?</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600892018-02-12T15:09:18ZTom Cleggtom@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>What's the motivation for having such a link?</p>
</blockquote>
Seems to have come from Peter's comment "linking between home and remote workbench instances is an absolutely critical aspect of the story" in note-25 above. Linking from the sessions page to remote Workbench dashboards doesn't seem "absolutely critical". I think the comment is referring to one or both of:
<ol>
<li>The "click Show button/link, despite note having a session at the remote workbench" scenario should continue to work (in the non-federated case, we rely on the remote workbench to initiate a login in this case, but that doesn't work yet with federated sites, especially without <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: [Federation] Workbench login chooser (Closed)" href="https://dev.arvados.org/issues/12958">#12958</a>)</li>
<li>The "multi-site search" button should do something sensible on a remote workbench (i.e., when the Rails session token is itself a salted token). We already had a separate bug <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [Federation] Wrong cluster ID shown on multisite search page (Resolved)" href="https://dev.arvados.org/issues/12959">#12959</a> for this, but once we started setting up federated-token sessions <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [Federation] Wrong cluster ID shown on multisite search page (Resolved)" href="https://dev.arvados.org/issues/12959">#12959</a> became a blocker, so fixing it right here seemed even better.</li>
</ol> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600952018-02-12T17:23:03ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="11454: Avoid instantiating new SessionDB before autoRedirectToHomeCluster() Arvados-DCO-1.1-Sign..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/3c31a5e643b930d1cdf227d2f5b151df48f20601">3c31a5e64</a><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/575/">https://ci.curoverse.com/job/developer-run-tests/575/</a></p>
<ul>
<li>Removed workbench links on <code>/sessions</code> page.</li>
<li>Only try to migrate non federated sessions on <code>/sessions</code> page instead of everywhere.</li>
<li><code>autoLoadRemoteHosts()</code> only loads missing remotes, without reactivating disabled sessions.</li>
<li>Avoid instantiating new <code>SessionDB</code> objects before calling <code>autoRedirectToHomeCluster()</code>.</li>
</ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=600962018-02-12T18:16:47ZTom Cleggtom@curii.com
<ul></ul><p>LGTM</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=601002018-02-12T19:06:28ZAnonymous
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '11454-wb-federated-search' Closes #11454 Refs #12959 Arvados-DCO-1.1-Signed-off-by..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c0a5fa033d0a9bfbeab001f287969756e9e1d16d">arvados|c0a5fa033d0a9bfbeab001f287969756e9e1d16d</a>.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=602442018-02-15T15:11:31ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Branch <code>11454-remove-button-removal</code> - <a class="changeset" title="11454: Don't show 'Remove' session button when listed on remoteHosts. Arvados-DCO-1.1-Signed-off..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9333d60538c8dcd9312e7b2dddb56dcf25e52d91">9333d6053</a><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/588/">https://ci.curoverse.com/job/developer-run-tests/588/</a></p>
<p>This is a small update to not show the "Remove" session button on <code>/sessions</code> when the remote is listed on remoteHosts.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=602452018-02-15T15:26:41ZTom Cleggtom@curii.com
<ul></ul><p>Not a huge fan of the nested ternary. How about something like:</p>
<pre><code class="diff syntaxhl"><span class="gd">- m('td', session.isFromRails ? null :
- session.listedHost ? null :
- m('button.btn.btn-xs.btn-default', {
</span><span class="gi">+ m('td', (session.isFromRails || session.listedHost) ? null :
+ m('button.btn.btn-xs.btn-default', {
</span></code></pre>
<p>or</p>
<pre><code class="diff syntaxhl"><span class="gi">+ m('td', !session.isFromRails && !session.listedHost &&
+ m('button.btn.btn-xs.btn-default', {
</span></code></pre> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=602472018-02-15T16:02:46ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Update at <a class="changeset" title="11454: Code readability enhancement. Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9c699bfa811f94b2cd7c3a82ad1197dc07029b17">9c699bfa8</a></p>
<p>The first option is better, IMO.</p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=602502018-02-15T16:32:28ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Test run successful: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/589/">https://ci.curoverse.com/job/developer-run-tests/589/</a></p> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=611262018-03-19T19:39:01ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/12958">Feature #12958</a>: [Federation] Workbench login chooser</i> added</li></ul> Arvados - Idea #11454: Support federated search across a set of Arvados clustershttps://dev.arvados.org/issues/11454?journal_id=647952018-07-23T19:22:31ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Release</strong> set to <i>17</i></li></ul>