https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422019-01-10T18:50:29ZArvadosArvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=704962019-01-10T18:50:29ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed" href="/issues/13648">Idea #13648</a>: [Epic] Use one cluster configuration file for all components</i> added</li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=711632019-02-06T19:23:10ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>To Be Groomed</i> to <i>Arvados Future Sprints</i></li><li><strong>Story points</strong> set to <i>1.0</i></li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=756632019-06-19T15:31:46ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2019-07-03 Sprint</i></li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=756702019-06-19T15:35:00ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=759802019-06-27T19:22:30ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=760422019-07-03T14:43:49ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2019-07-03 Sprint</i> to <i>2019-07-17 Sprint</i></li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=763682019-07-17T14:33:02ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2019-07-17 Sprint</i> to <i>2019-07-31 Sprint</i></li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=765732019-07-31T14:52:06ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2019-07-31 Sprint</i> to <i>2019-08-14 Sprint</i></li></ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=767182019-08-06T13:16:09ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="14716: Analyzing integration test fix failure Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <l..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/2c7b49e8791154db907616243c930c552aeb3df3">2c7b49e87</a> - branch <code>14716-webdav-cluster-config</code><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1448/">https://ci.curoverse.com/job/developer-run-tests/1448/</a></p>
<p>There's a keep-web integration test that's failing and I'm not being able to tell why. It's the one testing access to a collection using <code>/collections/uuid/path</code>. Supposedly the anonymous token is used on that case ignoring all other provided credentials, I made sure that the anonymous user token is configured, however but it still fails.<br />Some Workbench integration tests also fail, and they're related to the download feature so maybe I'm seeing the same issue from 2 different points of view.</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=767392019-08-06T18:35:08ZTom Cleggtom@curii.com
<ul></ul><p>It looks like the test was relying on a bug. The "/collections/$uuid/path" form is only supposed to use the configured anonymous tokens, and ignore caller-supplied tokens. But when no anonymous tokens were configured (as in this test), keep-web was incorrectly using the supplied credentials.</p>
14716-anonymous-token @ <a class="changeset" title="14716: Fix accidentally accepting credentials at public download URL. Arvados-DCO-1.1-Signed-off..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f89544af7f3d38bd61b4216527d66897eb08dcd0">f89544af7f3d38bd61b4216527d66897eb08dcd0</a> -- <a class="external" href="https://ci.curoverse.com/view/Developer/job/developer-run-tests/1451/">https://ci.curoverse.com/view/Developer/job/developer-run-tests/1451/</a>
<ul>
<li>don't use supplied credentials at /collections/$uuid/path, even if no anon tokens are configured</li>
<li>fix test → expect 404</li>
</ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=767942019-08-08T15:18:07ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="14716: Migrates NodeProfiles to Services on arvbox. Arvados-DCO-1.1-Signed-off-by: Lucas Di Pent..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/a33badcdc6af29a87d1f960dbff8ca947329a46b">a33badcdc</a> - branch <code>14716-webdav-cluster-config</code><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1459/">https://ci.curoverse.com/job/developer-run-tests/1459/</a></p>
<ul>
<li>Merged Tom's fix so now keep-web tests pass OK</li>
<li>Updated arvbox to make keep-web work</li>
<li>Migrated deprecated NodeProfiles on arvbox cluster config</li>
</ul>
<p>There're some workbench integration tests still failing. I'm not being able to run them correctly on my local dev VM (even on master, they fail in other ways) so not sure how to debug this. I think what's happening is a configuration issue between keep-web and workbench of some kind. Already tried the new keep-web on arvbox and seem to work fine.</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=767952019-08-08T17:02:34ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>The workbench integration tests that are failing are for example (from <a class="external" href="https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1533/console">https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1533/console</a>):</p>
<pre>
13:55:05 56) Failure:
13:55:05 DownloadTest#test_download_anonymous_content_from_keep-web_by_portable_data_hash [/tmp/workspace/developer-run-tests-apps-workbench-integration/apps/workbench/test/integration/download_test.rb:84]:
13:55:05 --- expected
13:55:05 +++ actual
13:55:05 @@ -1,2 +1 @@
13:55:05 -"Hello world
13:55:05 -"
13:55:05 +nil
13:55:05
13:55:05
13:55:05 57) Failure:
13:55:05 DownloadTest#test_download_anonymous_content_from_keep-web_by_uuid [/tmp/workspace/developer-run-tests-apps-workbench-integration/apps/workbench/test/integration/download_test.rb:84]:
13:55:05 --- expected
13:55:05 +++ actual
13:55:05 @@ -1,2 +1 @@
13:55:05 -"Hello world
13:55:05 -"
13:55:05 +nil
13:55:05
13:55:05
13:55:05 58) Failure:
13:55:05 DownloadTest#test_download_from_keep-web_using_a_session_token [/tmp/workspace/developer-run-tests-apps-workbench-integration/apps/workbench/test/integration/download_test.rb:84]:
13:55:05 Expected: "w a z"
13:55:05 Actual: nil
</pre>
<p>Checking the keep-web.log from the artifacts (<a class="external" href="https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1533/artifact/tmp/keep-web.log">https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1533/artifact/tmp/keep-web.log</a>) I'm seeing that all requests that mention the 'Hello World.txt' and 'w a z' files were served correctly, that's why I'm thinking there has to be a problem in the interaction between keep-web and workbench of some sort.</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=767992019-08-08T18:52:11ZTom Cleggtom@curii.com
<ul></ul><a name="config-docs"></a>
<h2 >config docs<a href="#config-docs" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>Set <code>Services.Controller.Insecure: true</code> if your API server's TLS certificate is not signed by a recognized CA.</p>
</blockquote>
<p>should be TLS.Insecure: true</p>
<blockquote>
<p>"http://:9002/": {}</p>
</blockquote>
<p>should be "http://keep_web_hostname_goes_here:9002/": {}</p>
<blockquote>
<p>exec sudo -u nobody keep-web -config=/path/to/arvados.yml</p>
</blockquote>
<p>I wonder if we need a note that arvados.yml needs to be world-readable?</p>
<p>I wonder if we should drop <code>-config=/path/to/arvados.yml</code> from the install docs and just rely on the default, since we already instructed the reader to use the default location.</p>
<blockquote>
<p>to your Workbench cluster configuration file</p>
</blockquote>
<p>remove Workbench</p>
<blockquote>
<p>ExternalURL: "https://download.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}"</p>
</blockquote>
<p>This shouldn't have the %{uuid_or_pdh} stuff any more -- Workbench now adds the /c= stuff (or puts the UUID in the host part) depending on whether the URL has a wildcard.</p>
<blockquote>
<p>ExternalURL: "https://%{uuid_or_pdh}--collections.<span class="userinput">uuid_prefix</span>.your.domain"</p>
</blockquote>
<p>Likewise these should be "https://*--collections.uuid_prefix.your.domain" etc.</p>
<a name="code"></a>
<h2 >code<a href="#code" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>AnonymousUserToken:</p>
</blockquote>
<p>Looks like you added a second copy of this key to config.default.yml?</p>
<blockquote>
<p>(ldr *Loader) LoadDefaults()</p>
</blockquote>
<p>I think I already rejected this in <a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: [websockets] Use cluster config (Resolved)" href="https://dev.arvados.org/issues/14717#note-10">#14717#note-10</a>. You can do this instead:</p>
<pre><code>cfg, err := config.NewLoader(bytes.NewBufferString("Clusters: {zzzzz: {}}"), nil).Load()</code></pre>
<p>If you want a helper function for that, it should be in arvadostest -- cluster zzzzz is only for testing.</p>
<blockquote>
<p>yaml "gopkg.in/yaml.v2"</p>
</blockquote>
<p>should use ghodss/yaml like everywhere else</p>
<blockquote>
<p>DefaultConfig()</p>
</blockquote>
<p>this seems to have changed purpose -- rename to newConfig(*arvados.Config) and update comment?</p>
<blockquote>
<p>TestLegacyConfig</p>
</blockquote>
<p>This is in services/keep-web, but seems to be testing code in lib/config -- maybe it should move to lib/config?</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=768032019-08-08T19:57:57ZTom Cleggtom@curii.com
<ul></ul><p>this seems to fix the workbench tests: <a class="changeset" title="14716: Remove Host header rewrite in integration test proxy config. Now that keep-web and Workbe..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/050ea7fdc6317a0fa0eeed20b0e6cb0b7fd6693b">050ea7fdc6317a0fa0eeed20b0e6cb0b7fd6693b</a></p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=768222019-08-09T19:00:22ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Now that we have set up an unified anonymous token for the workbench integration tests, I'm getting this failure:</p>
<pre>
14:44:28 9) Failure:
14:44:28 JobsTest#test_view_job_with_components_as_job_reader2_user [/tmp/workspace/developer-run-tests-apps-workbench-integration/apps/workbench/test/integration/jobs_test.rb:138]:
14:44:28 Expected false to be truthy.
</pre>
<p>I think this is related to <a class="external" href="https://github.com/curoverse/arvados/blob/master/apps/workbench/app/controllers/jobs_controller.rb#L6-L9">https://github.com/curoverse/arvados/blob/master/apps/workbench/app/controllers/jobs_controller.rb#L6-L9</a> , which makes workbench ignore the provided credentials and use the anonymous token, making the test fail because it's seeing some public project job. We're hours away to be removing the jobs API but I'm not sure if this test will get removed. I think this is a bug that may be fixed by adding a check to the <code>skip_around_action</code> that no credentials are provided in addition to having an anonymous token set?</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=768632019-08-12T20:55:26ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="Merge branch 'master' into 14716-webdav-cluster-config Arvados-DCO-1.1-Signed-off-by: Lucas Di P..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/8b873a9b3b8865a4d451263e48b49122b9c32759">8b873a9b3</a><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1474/">https://ci.curoverse.com/job/developer-run-tests/1474/</a></p>
<p>My previous comments on note-15 were incorrect, they were fixed by setting the anonymous token to an empty string (initially tried that but other random test failures made me think that wasn't the issue).</p>
<p>Addressed suggestions from note-13 and merge latest master.</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=768642019-08-12T23:44:21ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Re-ran tests using the pre-master merge commit <a class="changeset" title="14716: Fixes config type init function naming. Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c8f57c52224362d7621f1271774b0f2d60c55cac">c8f57c52224362d7621f1271774b0f2d60c55cac</a><br /><a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1475/">https://ci.curoverse.com/job/developer-run-tests/1475/</a> & <a class="external" href="https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1556/">https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1556/</a> (wb retry)</p>
<p>I'm investigating why <code>go/sdk/keepclient</code> tests fail... locally they don't.</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=768652019-08-13T00:51:46ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Seems to be a false alarm, successful re-run at <a class="external" href="https://ci.curoverse.com/job/developer-run-tests-remainder/1537/">https://ci.curoverse.com/job/developer-run-tests-remainder/1537/</a></p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=768682019-08-13T14:27:34ZTom Cleggtom@curii.com
<ul></ul><p>Should remove AssertPathExists from keep-web.service (thanks to Eric for pointing this out on another cluster-config issue)</p>
<p>On install-keep-web.html, when confirming that the keep-web binary is installed -- especially now that keep-web's flags are just generic config-loading options -- showing the entire output of "keep-web -h" seems like a lot of noise. Maybe just "Usage of keep-web: [...]" would be enough.</p>
<p>The "set the cluster config file like the following" section seems misplaced between "install runit" and "create a run script for runit". It also seems unnecessary to divide the config updates into two phases. Maybe the last section should be renamed from "Tell Workbench about..." to "Configure keep-web", and should also include the config.yml changes that are currently in the "install" section.</p>
<p>Related pre-existing bug: The "install keep-web" page doesn't mention that the package installs a systemd service and you should skip the runit stuff if you have systemd. It should probably be updated to match keep-balance and dispatch-cloud ("option 1: systemd" / "option 2: runit").</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=768712019-08-13T16:07:35ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Doc fixes at <a class="changeset" title="14716: Rearranges sections: install, configure, run keep-web. Arvados-DCO-1.1-Signed-off-by: Luc..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/d83e0937bc1d93560896ae1331486c473436b4a2">d83e0937b</a> - branch <code>14716-doc-fixes</code></p>
<p>Following the above suggestions:</p>
<ul>
<li>Removed generic config loading options from the "keep-web -h" output example</li>
<li>Moved the config file description to the last section</li>
<li>Renamed last section to "configure keep-web" & adjusted it a little to be more consistent</li>
<li>Added systemd section</li>
<li>Reordered sections to be: 1) install, 2) configure and finally 3) run.</li>
</ul> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=769062019-08-13T18:11:20ZTom Cleggtom@curii.com
<ul></ul><p>Given that the example "generate anonymous token" transcript produces "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz", the AnonymousUserToken in the immediately following example file should also be "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz".</p>
<p>There are no instructions for making a SystemRootToken and I don't think keep-web even uses it -- that line should probably be removed from the example config.</p>
<p>The "config file should be world-readable" note is misplaced. Come to think of it, since systemd runs keep-web as root, maybe we should remove the "sudo -u nobody" from the runit script to match systemd, and remove the "world-readable" note.</p>
<p>The rest LGTM.</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=769222019-08-13T18:40:52ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '14716-doc-fixes' Closes #14716 Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ld..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/b64996755741a71c4f02cf3ef9ea2b7ba1d0e2e1">arvados|b64996755741a71c4f02cf3ef9ea2b7ba1d0e2e1</a>.</p> Arvados - Feature #14716: [keep-web] Use cluster config filehttps://dev.arvados.org/issues/14716?journal_id=810862020-01-21T21:20:49ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>22</i></li></ul>