Project

General

Profile

Actions

Idea #20690

closed

Remove workbench 1 from main branch !!!!

Added by Peter Amstutz 11 months ago. Updated 26 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Workbench
Start date:
11/20/2023
Due date:
Story points:
-
Release:
Release relationship:
Auto

Description

Once we have migrated everything to use Workbench 2 only, remove Workbench 1:

  • Remove apps/workbench
  • Remove remove from run-tests and package building
  • Remove from jenkins pipeline
  • Remove from arvbox
  • Remove salt installer and formula
  • Remove from arvados-server install
  • Remove from arvados-server boot
  • Remove references to apps/workbench in documentation build

Subtasks 2 (0 open2 closed)

Task #21178: Review 20690-remove-wb1ResolvedTom Clegg11/20/2023Actions
Task #21229: Review 20690-remove-wb1-from-installerResolvedLucas Di Pentima11/28/2023Actions

Related issues

Related to Arvados Epics - Idea #17001: Arvados uses WB2 by defaultResolvedActions
Related to Arvados - Idea #20846: Support Ubuntu 22.04 LTSResolvedBrett Smith10/30/2023Actions
Actions #1

Updated by Peter Amstutz 11 months ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz 11 months ago

  • Related to Idea #17001: Arvados uses WB2 by default added
Actions #3

Updated by Peter Amstutz 8 months ago

  • Target version changed from Future to Development 2023-10-25 sprint
Actions #4

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Actions #5

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #6

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2023-11-29 sprint to Future
Actions #7

Updated by Peter Amstutz 7 months ago

  • Target version changed from Future to Development 2024-01-03 sprint
Actions #8

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2024-01-03 sprint to Development 2023-11-29 sprint
Actions #9

Updated by Tom Clegg 7 months ago

  • Related to Idea #20846: Support Ubuntu 22.04 LTS added
Actions #10

Updated by Peter Amstutz 7 months ago

  • Assigned To set to Tom Clegg
Actions #12

Updated by Tom Clegg 7 months ago

  • Status changed from New to In Progress
  • Category set to Workbench
  • Description updated (diff)

In progress:

20690-remove-wb1 @ 4bdddf57dbfe6565fe7f1a583fc15be4024a21b1 -- developer-run-tests: #3899 (retry developer-run-tests-remainder: #4099 )

  • All agreed upon points are implemented / addressed.
    • ✅ Remove source:apps/workbench
    • ✅ Remove from source:.licenseignore
    • ✅ Remove from source:build/run-tests.sh
    • ✅ Remove from package building scripts in source:build/
    • ✅ Remove jenkins jobs from run-tests and developer-run-tests pipelines
    • ☐ Delete jenkins jobs entirely
    • ✅ Remove from arvbox
    • ☐ Remove salt installer and formula
    • ✅ Remove from arvados-server install
    • ✅ Remove dependencies from arvados-server install (phantomjs, geckodriver) -- but wb2 tests still need firefox (right?)
    • ✅ Remove from arvados-server boot
    • ✅ Remove from source:sdk/python/tests/run-test-servers.py
    • ✅ Add wb1-to-wb2 path redirects (from salt) in arvados-server boot
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • n/a
  • Code is tested and passing, both automated and manual, what manual testing was done is described
    • test lib/install -check.vv -tags docker (in run-tests --interactive)
    • cd lib/install && ./arvadostest_docker_build.sh
  • Documentation has been updated.
    • ✅ Remove references to apps/workbench in documentation build
    • ✅ Add upgrade note about removing the workbench package and updating config.yml and nginx.conf
    • ☐ Add instructions for adding wb1-to-wb2 redirects to Nginx config.
  • Behaves appropriately at the intended scale (describe intended scale).
    • n/a
  • Considered backwards and forwards compatibility issues between client and server.
    • n/a
  • Follows our coding standards and GUI style guidelines.

I included a couple of commits from an unmerged #18874 branch so that package/install/boot correctly handle wb1 urls with wb2.

Also updated the arvados-package test from debian 10 to 11.

Actions #13

Updated by Lucas Di Pentima 7 months ago

These are my observations:

  • The removed config Workbench.SecretKeyBase is mentioned on a couple of places, including export.go
  • The arvados-client diagnostics command could be updated to avoid checking WB1's endpoint.
  • There're some pending mentions on WB1 at
    • doc/admin/config-urls.html.textile.liquid
    • doc/admin/diagnostics.html.textile.liquid
    • doc/admin/metrics.html.textile.liquid
    • doc/admin/inspect.html.textile.liquid
    • doc/admin/restricting-upload-download.html.textile.liquid
    • doc/install/install-keepproxy.html.textile.liquid (Line 24 talks about wb1's upload feature needing keepproxy)
    • sdk/cwl/arvados_cwl/arvcontainer.py Line 747
    • sdk/go/health/aggregator.go Lines 329, 386
    • tools/arvbox/lib/arvbox/docker/cluster-config.sh Line 142

I can take care of the salt installer & formula, if you like.

Actions #14

Updated by Tom Clegg 6 months ago

Lucas Di Pentima wrote in #note-13:

These are my observations:
[...]

Removed/updated all of the above.

I can take care of the salt installer & formula, if you like.

That would be great, thanks.

Currently trying to figure out whether the arvbox changes broke anything. It looks like it wasn't tested after updating from debian 10 to debian 12, so "arvbox build dev" was already failing. I think it needs to move back to debian 11 until #20846 merges.

20690-remove-wb1 @ f8ba6deb97126c786d97cf835b917837822bfeb1 -- developer-run-tests: #3908

Actions #15

Updated by Tom Clegg 6 months ago

Test failure above might be fixed by the timeout that changed in the main branch. Merged main.

20690-remove-wb1 @ 383527c82b7f8c413121165d8d6296feca9a1728 -- developer-run-tests: #3913

Actions #16

Updated by Tom Clegg 6 months ago

One more change needed to make arvbox build & start on debian 11.

20690-remove-wb1 @ e88556b084ba5af008e0cde991a4502d106e4d09

Actions #17

Updated by Lucas Di Pentima 6 months ago

Just one comment:

I tried to start a fresh arvbox (with an image from this branch) instance and got stuck. I managed to get it unstuck by running go mod tidy -compat=1.17 while being in ~/.arvbox/arvbox/arvados (got the clue from controller's arvbox log)
After that, almost all services started except workbench2, repeatedly reporting "Waiting for workbench2 (installing ruby gems 93 of about 106) ..." (misleading message that may be easy to remove?). Eventually workbench2 finished installing/compiling dependencies and started OK.

The rest LGTM, thanks!

Actions #18

Updated by Lucas Di Pentima 6 months ago

Tom Clegg wrote in #note-14:

I can take care of the salt installer & formula, if you like.

That would be great, thanks.

I can make a branch for this same ticket and you can review it. What I'm thinking to do is update the installer and/or formula so that everything related to workbench1 is set to be uninstalled. This way existing installations won't need manual intervention to remove packages/configurations. I think the arvados-formula has support for this.

Actions #19

Updated by Tom Clegg 6 months ago

Lucas Di Pentima wrote in #note-17:

I tried to start a fresh arvbox (with an image from this branch) instance and got stuck. I managed to get it unstuck by running go mod tidy -compat=1.17 while being in ~/.arvbox/arvbox/arvados (got the clue from controller's arvbox log)

Maybe Peter should weigh in on that -- I'm not sure what circumstances would lead to that, but I'm guessing it's something to do with an old source tree being checked out there already from previous arvbox use.

After that, almost all services started except workbench2, repeatedly reporting "Waiting for workbench2 (installing ruby gems 93 of about 106) ..." (misleading message that may be easy to remove?). Eventually workbench2 finished installing/compiling dependencies and started OK.

Hm. It looks like it only prints that if "bundle install" is running somewhere, which seems odd if the only thing it's waiting for is workbench2. Haven't figured this out yet either. I have seen the "installing ruby gems" bit come and go while waiting for the same list including api, though. Perhaps the "api" service hits "touch .../api.ready" but then restarts?

Actions #20

Updated by Lucas Di Pentima 6 months ago

20690-remove-wb1-from-installer @ 093561a

test-provision-debian11: #558

At first I tried to make the installer ask the arvados-formula to remove Workbench1 (instead of just not mention it) so that preexisting deployments would remove the packages. The issue was that the arvados.workbench.clean state is too greedy and tries to uninstall several other dependencies that other services will need if running on the same node (for example, nginx) so I ended up changing the installer in such a way that it uses the current form of the arvados-formula that's currently in main, but doesn't mention the arvados.workbench at all to avoid trying to install it on new deployments.

  • All agreed upon points are implemented / addressed.
    • Updates installer's check_command so that it doesn't fail when having deprecated config keys on the config file.
    • Removes Workbench1's secret key handling from local.params and uses a dummy value.
    • Splits the ssl hardening nginx config snippet from the passenger configuration, so that other components can include it without having to assume that passenger is installed on the node.
    • Makes the legacy workbench role only install the nginx configuration that includes all the WB1 URL redirection.
    • Updates installer documentation to reflect Workbench1's removal.
    • Adds upgrade note to reming admins to manually uninstall workbench1 packages after upgrading the installer.
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
  • Code is tested and passing, both automated and manual, what manual testing was done is described
    • I've manually deployed a multi-host cluster successfully.
  • Documentation has been updated.
    • Yes
  • Behaves appropriately at the intended scale (describe intended scale).
    • N/A
  • Considered backwards and forwards compatibility issues between client and server.
    • N/A
  • Follows our coding standards and GUI style guidelines.
    • Yes
Actions #21

Updated by Tom Clegg 6 months ago

  • Target version changed from Development 2023-11-29 sprint to Development 2024-01-03 sprint
Actions #22

Updated by Brett Smith 6 months ago

Lucas Di Pentima wrote in #note-20:

20690-remove-wb1-from-installer @ 093561a

This looks good, my only comment is about the upgrade note.

  • I wonder whether I should remove the package before or after the upgrade
  • It would be helpful to say the full package name arvados-workbench
  • While I appreciate that it's very polite, I think some of that verbiage makes it less clear: does "please take into consideration" mean there are situations where I shouldn't remove the package?

I think text like this could be clearer:

"If you installed a previous version of Arvados with the Salt installer, and you upgrade your installer to upgrade the cluster, you should uninstall the arvados-workbench package from the workbench instance afterwards."

Feel free to tweak that, or adapt your version, then merge. Thanks.

Actions #23

Updated by Lucas Di Pentima 6 months ago

Thanks! merged.

Actions #24

Updated by Lucas Di Pentima 6 months ago

  • Status changed from In Progress to Resolved
Actions #25

Updated by Peter Amstutz 26 days ago

  • Release set to 70
Actions

Also available in: Atom PDF