https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422020-11-12T16:46:04ZArvadosArvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=886362020-11-12T16:46:04ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=886372020-11-12T16:46:09ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>New</i></li><li><strong>Subject</strong> changed from <i>Cannot use federated tokens with keep-web S3 API </i> to <i>Cannot use federated tokens with keep-web S3 API</i></li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=886402020-11-12T16:46:15ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=886412020-11-12T16:47:38ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/88641/diff?detail_id=85412">diff</a>)</li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=886422020-11-12T16:48:24ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/88642/diff?detail_id=85413">diff</a>)</li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=887062020-11-16T18:23:18ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-12-02 Sprint</i> to <i>2020-11-18</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=887122020-11-16T18:28:50ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>36</i></li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=887492020-11-16T22:16:49ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> changed from <i>36</i> to <i>37</i></li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=887602020-11-17T02:28:46ZTom Cleggtom@curii.com
<ul></ul><p>Our preferred approach for s3 credentials (access key = token uuid, secret key = token secret part) can't work on a cluster that uses a remote LoginCluster, because keep-web can't use the uuid to look up the secret part (that only works if the token is in its own local database).</p>
<p>Using the entire v2 token as the access key and secret key is unlikely to work, because "/" is used as a delimiter in the authorization header, and real AWS access keys don't use that character. Some clients might handle this in various different ways, but some will surely reject it.</p>
<p>We can accept the entire v2 token as the access key and secret key if "/" is replaced with "_". 17106-s3-fed-token @ <a class="changeset" title="17106: Test S3 with modified v2 token issued by LoginCluster. Arvados-DCO-1.1-Signed-off-by: Tom..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/70be08860db9e45d78a037d86b9a0420f1e392a1">70be08860db9e45d78a037d86b9a0420f1e392a1</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2177/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2177/">developer-run-tests: #2177 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2177" alt="" /></a></a></p>
<p>Using just the secret part should be possible, but doesn't currently work. I suspect we need to change RailsAPI's verification process so it calls "get current token" to get the original UUID (it currently only calls "get current user" to verify that the token is valid, then invents a UUID for the local database).</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=888332020-11-18T14:40:56ZTom Cleggtom@curii.com
<ul></ul><p>17106-s3-fed-token @ <a class="changeset" title="17106: Improve handling of bare tokens issued by remote clusters. When caching, use the remote c..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f194bb8b667815f3f3fbd01a3d7ba824b05416ed">f194bb8b667815f3f3fbd01a3d7ba824b05416ed</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2182/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2182/">developer-run-tests: #2182 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2182" alt="" /></a></a></p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=888382020-11-18T15:30:28ZTom Cleggtom@curii.com
<ul></ul>17106-s3-fed-token @ <a class="changeset" title="17106: Update docs. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/bee9aff3bd6b69f81a0dd53fa7b4118d0eeeb0a9">bee9aff3bd6b69f81a0dd53fa7b4118d0eeeb0a9</a>
<ul>
<li>Accept secret part of token, even if token was issued by LoginCluster</li>
<li>Accept full v2 token, with "/" replaced by "_" (unsure whether we want to keep/document this; it's awkward but it can make more federation cases work)</li>
<li>Comments/docs updated since test run above in note-10</li>
</ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=888452020-11-18T16:29:04ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-11-18</i> to <i>2020-12-02 Sprint</i></li></ul> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=889362020-11-19T22:57:39ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
17106-s3-fed-token @ <a class="changeset" title="17106: Update docs. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/bee9aff3bd6b69f81a0dd53fa7b4118d0eeeb0a9">bee9aff3bd6b69f81a0dd53fa7b4118d0eeeb0a9</a>
<ul>
<li>Accept secret part of token, even if token was issued by LoginCluster</li>
</ul>
</blockquote>
<p>So if you can present a bare secret and if the LoginCluster accepts it, it can be used. It behaves the same way whether it is an OIDC access token or just a bare arvados token? Meaning what gets stored is the hash?</p>
<p>So in both the OIDC access token and bare token cases, you have to present "secret/secret" (or secret/none). Does the value of the SecretKey even matter in this case, except to be able to check if the signature is correct or not, even though we are not basing out authentication decision on the signature?</p>
<blockquote>
<ul>
<li>Accept full v2 token, with "/" replaced by "_" (unsure whether we want to keep/document this; it's awkward but it can make more federation cases work)</li>
</ul>
</blockquote>
<p>I think the federation case you are referring to is this: you can present the munged v2 token in the "access key" part and then it gets decoded to a regular v2 token by keep-web, now we have a token uuid telling us what cluster issued the token, for the case where you have a federation but not a LoginCluster.</p>
<p>I don't suppose we could work around this by supporting AWS signatures in the API server? Or does that not work because the signature is an hmac that includes the actual request, so we'd have to somehow encapsulate/forward the entire request (more work, might be awkward)?</p>
<p>So I think I understand the thought process that gets us here. The underscore workaround seems fine and should stay. Providing access instructions in Workbench (<a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: Collection/project pages include instructions to connect via WebDAV and S3 (Resolved)" href="https://dev.arvados.org/issues/16622">#16622</a>) we can provide the modified token for copy and paste.</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=889392020-11-20T15:26:25ZTom Cleggtom@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>So if you can present a bare secret and if the LoginCluster accepts it, it can be used. It behaves the same way whether it is an OIDC access token or just a bare arvados token? Meaning what gets stored is the hash?</p>
</blockquote>
<p>Yes, exactly.</p>
<blockquote>
<p>So in both the OIDC access token and bare token cases, you have to present "secret/secret" (or secret/none). Does the value of the SecretKey even matter in this case, except to be able to check if the signature is correct or not, even though we are not basing out authentication decision on the signature?</p>
</blockquote>
<p>For v4 requests we do check the signature, so "none" wouldn't work.</p>
<p>For v2 requests we don't check the signature, we just check that the access key is a real token.</p>
<p>Supporting v2 signatures might not be important enough to bother implementing a proper signature check. Perhaps we should just stop accepting them, like AWS.</p>
<blockquote><blockquote>
<ul>
<li>Accept full v2 token, with "/" replaced by "_" (unsure whether we want to keep/document this; it's awkward but it can make more federation cases work)</li>
</ul>
</blockquote>
<p>I think the federation case you are referring to is this: you can present the munged v2 token in the "access key" part and then it gets decoded to a regular v2 token by keep-web, now we have a token uuid telling us what cluster issued the token, for the case where you have a federation but not a LoginCluster.</p>
</blockquote>
<p>Yes, exactly.</p>
<blockquote>
<p>I don't suppose we could work around this by supporting AWS signatures in the API server? Or does that not work because the signature is an hmac that includes the actual request, so we'd have to somehow encapsulate/forward the entire request (more work, might be awkward)?</p>
</blockquote>
<p>We could give controller a "validate v4 signature" endpoint, so an intermediate cluster could accept a request that uses only the token UUID as its access key. I think that part would be easy enough. However, the intermediate cluster still wouldn't know the entire token, so it wouldn't be able to retrieve/update collections needed to <em>serve</em> the request. (Even in the cases where the "validate v4 signature" endpoint could technically look up and return the secret, that doesn't seem like a good road to travel.)</p>
<blockquote>
<p>So I think I understand the thought process that gets us here. The underscore workaround seems fine and should stay. Providing access instructions in Workbench (<a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: Collection/project pages include instructions to connect via WebDAV and S3 (Resolved)" href="https://dev.arvados.org/issues/16622">#16622</a>) we can provide the modified token for copy and paste.</p>
</blockquote>
<p>Good point, that goes a long way to addressing the UX weirdness.</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=889402020-11-20T15:48:18ZTom Cleggtom@curii.com
<ul></ul>17106-s3-fed-token @ <a class="changeset" title="17106: Recommend using full tokens for S3 access. Accept munged ("/" => "_") tokens in S3 reques..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/199ca290ab259ba21f798bb059bb808fe3b609ba">199ca290ab259ba21f798bb059bb808fe3b609ba</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2188/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2188/">developer-run-tests: #2188 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2188" alt="" /></a></a>
<ul>
<li>Updates docs re "_" substitution</li>
<li>Accepts "_" substitution with v2-signed requests too, to simplify usage/docs</li>
</ul>
<p>One more edge that we will have to confront eventually (but doesn't affect the present issue/branch):</p>
<p>If the token is an OIDC access token rather than an Arvados token, it might contain both "/" and "_", so this substitution won't work. As the code stands now, an OIDC access token containing "_" will work (provided it doesn't happen to start with "v2_" like an arvados token), but an OIDC access token with "/" that the user has replaced with "_" will not work.</p>
<p>OIDC access tokens are allowed to have any printable char, so I think handling the general case would require escaping ("=2f"?) rather than substituting.</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=889442020-11-20T17:01:59ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>17106-s3-fed-token @ <a class="changeset" title="17106: Recommend using full tokens for S3 access. Accept munged ("/" => "_") tokens in S3 reques..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/199ca290ab259ba21f798bb059bb808fe3b609ba">199ca290ab259ba21f798bb059bb808fe3b609ba</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2188/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2188/">developer-run-tests: #2188 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2188" alt="" /></a></a></p>
<p>One more edge: if the token is an OIDC access token rather than an Arvados token, it might contain both "/" and "_", so this substitution won't work. As the code stands now, an OIDC access token containing "_" will work (provided it doesn't happen to start with "v2_" like an arvados token), but an OIDC access token with "/" that the user has replaced with "_" will not work.</p>
<p>OIDC access tokens are allowed to have any printable char, so I think handling the general case would require escaping ("=2f"?) rather than substituting.</p>
</blockquote>
<p>So as a general solution instead of substituting "_" for "/" we do URI-encode or other escaping? Seems like if we are going to change the substitution strategy we should do this now, otherwise we merge support for the underscore syntax only to pull it out again (or be stuck with it forever).</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=889522020-11-20T18:54:07ZTom Cleggtom@curii.com
<ul></ul><p>I was thinking we would continue to accept the "_" substitution for Arvados tokens ("v2_zzzzz-gj3su-..." is recognizable) even when we also accept a more general escaping mechanism for non-Arvados tokens.</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=889532020-11-20T19:00:20ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>I was thinking we would continue to accept the "_" substitution for Arvados tokens ("v2_zzzzz-gj3su-..." is recognizable) even when we also accept a more general escaping mechanism for non-Arvados tokens.</p>
</blockquote>
<p>I don't understand the benefit if we end up with two schemes and there's some ambiguity if "_" should be turned back into "/" or not.</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=890002020-11-23T19:59:46ZTom Cleggtom@curii.com
<ul></ul><p>I just meant a string like "v2_{uuid}_..." will still be unambiguous if/when we also support a more general escaping scheme, so we won't need to worry about accidentally mangling "_" chars in tokens that aren't Arvados tokens.</p>
<p>Added support for query-escaped tokens.</p>
<p>17106-s3-fed-token @ <a class="changeset" title="17106: Allow use of URL-encoded token as S3 access/secret key. Arvados-DCO-1.1-Signed-off-by: To..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0cfb2b0646ad8129c82883717af7a51d28e6876a">0cfb2b0646ad8129c82883717af7a51d28e6876a</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2193/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2193/">developer-run-tests: #2193 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2193" alt="" /></a></a></p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=890032020-11-23T20:53:24ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>I just meant a string like "v2_{uuid}_..." will still be unambiguous if/when we also support a more general escaping scheme, so we won't need to worry about accidentally mangling "_" chars in tokens that aren't Arvados tokens.</p>
<p>Added support for query-escaped tokens.</p>
<p>17106-s3-fed-token @ <a class="changeset" title="17106: Allow use of URL-encoded token as S3 access/secret key. Arvados-DCO-1.1-Signed-off-by: To..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0cfb2b0646ad8129c82883717af7a51d28e6876a">0cfb2b0646ad8129c82883717af7a51d28e6876a</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2193/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2193/">developer-run-tests: #2193 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2193" alt="" /></a></a></p>
</blockquote>
<p>1. I think you want to use "url.PathUnescape" not "url.QueryUnescape" because "QueryUnescape" treats "+" as a space and "PathEscape" does not.</p>
<p>2. The documentation explaining tokens with S3 would benefit from an example:</p>
<p>"If have an Arvados token 'v2/zzzzz-gj3su-1234567890abcde/zyxzyxzyxzyx' you can use the following to communicate with the cluster 'zzzzz':</p>
<p>access_key = <a href="https://arvadosapi.com/zzzzz-gj3su-1234567890abcde">zzzzz-gj3su-1234567890abcde</a><br />secret_key = zyxzyxzyxzyx</p>
<p>however if this is a federated token and you are communicating with a cluster other than 'zzzzz', use this:</p>
<p>access_key = v2_zzzzz-gj3su-1234567890abcde_zyxzyxzyxzyx <br />secret_key = v2_zzzzz-gj3su-1234567890abcde_zyxzyxzyxzyx"</p>
<p>Rest LGTM.</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=890242020-11-24T15:29:24ZTom Cleggtom@curii.com
<ul></ul><p>17106-s3-fed-token @ <a class="changeset" title="17106: Add examples to S3 auth instructions. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomcl..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c36bac7d8ec9f7f579ddfdc06a328fa3668e80a3">c36bac7d8ec9f7f579ddfdc06a328fa3668e80a3</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2195/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2195/">developer-run-tests: #2195 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2195" alt="" /></a></a></p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=890272020-11-24T15:47:10ZTom Cleggtom@curii.com
<ul><li><strong>File</strong> <a href="/attachments/2615">doc.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2615/doc.png">doc.png</a> added</li></ul><p><img src="https://dev.arvados.org/attachments/download/2615/doc.png" alt="" /></p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=890302020-11-24T16:42:51ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p><img src="https://dev.arvados.org/attachments/download/2615/doc.png" alt="" /></p>
</blockquote>
<p>LGTM</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=890322020-11-24T17:05:26ZAnonymous
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '17106-s3-fed-token' fixes #17106 Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@to..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/917330c81bb370225ccd0e051dbdca3d1870710e">arvados|917330c81bb370225ccd0e051dbdca3d1870710e</a>.</p> Arvados - Bug #17106: Cannot use federated tokens with keep-web S3 APIhttps://dev.arvados.org/issues/17106?journal_id=907562021-02-18T22:52:02ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> changed from <i>37</i> to <i>38</i></li></ul>