Arvados

18874-merge-wb2 @ 4f06e4dc7e9eb5ad04703dbbaa47370e6baeb9a1 -- developer-run-tests: #3893

These are my observations:

• The removed config Workbench.SecretKeyBase is mentioned on a couple of places, including export.go
• The arvados-client diagnostics command could be updated to avoid checking WB1's endpoint.
• There're some pending mentions on WB1 at
  • doc/admin/config-urls.html.textile.liquid
  • doc/admin/diagnostics.html.textile.liquid
  • doc/admin/metrics.html.textile.liquid
  • doc/admin/inspect.html.textile.liquid
  • doc/admin/restricting-upload-download.html.textile.liquid
  • doc/install/install-keepproxy.html.textile.liquid (Line 24 talks about wb1's upload feature needing keepproxy)
  • sdk/cwl/arvados_cwl/arvcontainer.py Line 747
  • sdk/go/health/aggregator.go Lines 329, 386
  • tools/arvbox/lib/arvbox/docker/cluster-config.sh Line 142

I can take care of the salt installer & formula, if you like.

Lucas Di Pentima wrote in #note-13:

These are my observations:
[...]

Removed/updated all of the above.

I can take care of the salt installer & formula, if you like.

That would be great, thanks.

Currently trying to figure out whether the arvbox changes broke anything. It looks like it wasn't tested after updating from debian 10 to debian 12, so "arvbox build dev" was already failing. I think it needs to move back to debian 11 until #20846 merges.

20690-remove-wb1 @ f8ba6deb97126c786d97cf835b917837822bfeb1 -- developer-run-tests: #3908

Test failure above might be fixed by the timeout that changed in the main branch. Merged main.

20690-remove-wb1 @ 383527c82b7f8c413121165d8d6296feca9a1728 -- developer-run-tests: #3913

One more change needed to make arvbox build & start on debian 11.

20690-remove-wb1 @ e88556b084ba5af008e0cde991a4502d106e4d09

Just one comment:

I tried to start a fresh arvbox (with an image from this branch) instance and got stuck. I managed to get it unstuck by running go mod tidy -compat=1.17 while being in ~/.arvbox/arvbox/arvados (got the clue from controller's arvbox log)
After that, almost all services started except workbench2, repeatedly reporting "Waiting for workbench2 (installing ruby gems 93 of about 106) ..." (misleading message that may be easy to remove?). Eventually workbench2 finished installing/compiling dependencies and started OK.

The rest LGTM, thanks!

Tom Clegg wrote in #note-14:

I can take care of the salt installer & formula, if you like.

That would be great, thanks.

I can make a branch for this same ticket and you can review it. What I'm thinking to do is update the installer and/or formula so that everything related to workbench1 is set to be uninstalled. This way existing installations won't need manual intervention to remove packages/configurations. I think the arvados-formula has support for this.

Lucas Di Pentima wrote in #note-17:

I tried to start a fresh arvbox (with an image from this branch) instance and got stuck. I managed to get it unstuck by running go mod tidy -compat=1.17 while being in ~/.arvbox/arvbox/arvados (got the clue from controller's arvbox log)

Maybe Peter should weigh in on that -- I'm not sure what circumstances would lead to that, but I'm guessing it's something to do with an old source tree being checked out there already from previous arvbox use.

After that, almost all services started except workbench2, repeatedly reporting "Waiting for workbench2 (installing ruby gems 93 of about 106) ..." (misleading message that may be easy to remove?). Eventually workbench2 finished installing/compiling dependencies and started OK.

Hm. It looks like it only prints that if "bundle install" is running somewhere, which seems odd if the only thing it's waiting for is workbench2. Haven't figured this out yet either. I have seen the "installing ruby gems" bit come and go while waiting for the same list including api, though. Perhaps the "api" service hits "touch .../api.ready" but then restarts?

20690-remove-wb1-from-installer @ 093561a

test-provision-debian11: #558

At first I tried to make the installer ask the arvados-formula to remove Workbench1 (instead of just not mention it) so that preexisting deployments would remove the packages. The issue was that the arvados.workbench.clean state is too greedy and tries to uninstall several other dependencies that other services will need if running on the same node (for example, nginx) so I ended up changing the installer in such a way that it uses the current form of the arvados-formula that's currently in main, but doesn't mention the arvados.workbench at all to avoid trying to install it on new deployments.

• All agreed upon points are implemented / addressed.
  • Updates installer's check_command so that it doesn't fail when having deprecated config keys on the config file.
  • Removes Workbench1's secret key handling from local.params and uses a dummy value.
  • Splits the ssl hardening nginx config snippet from the passenger configuration, so that other components can include it without having to assume that passenger is installed on the node.
  • Makes the legacy workbench role only install the nginx configuration that includes all the WB1 URL redirection.
  • Updates installer documentation to reflect Workbench1's removal.
  • Adds upgrade note to reming admins to manually uninstall workbench1 packages after upgrading the installer.
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • We need to improve the arvados-formula versioning: https://dev.arvados.org/issues/21228
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • I've manually deployed a multi-host cluster successfully.
• Documentation has been updated.
  • Yes
• Behaves appropriately at the intended scale (describe intended scale).
  • N/A
• Considered backwards and forwards compatibility issues between client and server.
  • N/A
• Follows our coding standards and GUI style guidelines.
  • Yes

Lucas Di Pentima wrote in #note-20:

20690-remove-wb1-from-installer @ 093561a

This looks good, my only comment is about the upgrade note.

• I wonder whether I should remove the package before or after the upgrade
• It would be helpful to say the full package name arvados-workbench
• While I appreciate that it's very polite, I think some of that verbiage makes it less clear: does "please take into consideration" mean there are situations where I shouldn't remove the package?

I think text like this could be clearer:

"If you installed a previous version of Arvados with the Salt installer, and you upgrade your installer to upgrade the cluster, you should uninstall the arvados-workbench package from the workbench instance afterwards."

Feel free to tweak that, or adapt your version, then merge. Thanks.

Thanks! merged.