Idea #20690
closedRemove workbench 1 from main branch !!!!
Description
Once we have migrated everything to use Workbench 2 only, remove Workbench 1:
- Remove apps/workbench
- Remove remove from run-tests and package building
- Remove from jenkins pipeline
- Remove from arvbox
- Remove salt installer and formula
- Remove from arvados-server install
- Remove from arvados-server boot
- Remove references to apps/workbench in documentation build
Related issues
Updated by Peter Amstutz over 1 year ago
- Related to Idea #17001: Arvados uses WB2 by default added
Updated by Peter Amstutz about 1 year ago
- Target version changed from Future to Development 2023-10-25 sprint
Updated by Peter Amstutz about 1 year ago
- Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Updated by Peter Amstutz about 1 year ago
- Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Updated by Peter Amstutz about 1 year ago
- Target version changed from Development 2023-11-29 sprint to Future
Updated by Peter Amstutz about 1 year ago
- Target version changed from Future to Development 2024-01-03 sprint
Updated by Peter Amstutz 12 months ago
- Target version changed from Development 2024-01-03 sprint to Development 2023-11-29 sprint
Updated by Tom Clegg 12 months ago
- Related to Idea #20846: Support Ubuntu 22.04 LTS added
Updated by Tom Clegg 12 months ago
18874-merge-wb2 @ 4f06e4dc7e9eb5ad04703dbbaa47370e6baeb9a1 -- developer-run-tests: #3893
Updated by Tom Clegg 12 months ago
- Status changed from New to In Progress
- Category set to Workbench
- Description updated (diff)
In progress:
20690-remove-wb1 @ 4bdddf57dbfe6565fe7f1a583fc15be4024a21b1 -- developer-run-tests: #3899 (retry developer-run-tests-remainder: #4099 )
- All agreed upon points are implemented / addressed.
- ✅ Remove source:apps/workbench
- ✅ Remove from source:.licenseignore
- ✅ Remove from source:build/run-tests.sh
- ✅ Remove from package building scripts in source:build/
- ✅ Remove jenkins jobs from run-tests and developer-run-tests pipelines
- ☐ Delete jenkins jobs entirely
- ✅ Remove from arvbox
- ☐ Remove salt installer and formula
- ✅ Remove from arvados-server install
- ✅ Remove dependencies from arvados-server install (phantomjs, geckodriver) -- but wb2 tests still need firefox (right?)
- ✅ Remove from arvados-server boot
- ✅ Remove from source:sdk/python/tests/run-test-servers.py
- ✅ Add wb1-to-wb2 path redirects (from salt) in arvados-server boot
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- n/a
- Code is tested and passing, both automated and manual, what manual testing was done is described
- ✅
- ✅
test lib/install -check.vv -tags docker
(in run-tests --interactive) - ✅
cd lib/install && ./arvadostest_docker_build.sh
- Documentation has been updated.
- ✅ Remove references to apps/workbench in documentation build
- ✅ Add upgrade note about removing the workbench package and updating config.yml and nginx.conf
- ☐ Add instructions for adding wb1-to-wb2 redirects to Nginx config.
- Behaves appropriately at the intended scale (describe intended scale).
- n/a
- Considered backwards and forwards compatibility issues between client and server.
- n/a
- Follows our coding standards and GUI style guidelines.
- ✅
I included a couple of commits from an unmerged #18874 branch so that package/install/boot correctly handle wb1 urls with wb2.
Also updated the arvados-package
test from debian 10 to 11.
Updated by Lucas Di Pentima 12 months ago
These are my observations:
- The removed config
Workbench.SecretKeyBase
is mentioned on a couple of places, includingexport.go
- The
arvados-client diagnostics
command could be updated to avoid checking WB1's endpoint. - There're some pending mentions on WB1 at
doc/admin/config-urls.html.textile.liquid
doc/admin/diagnostics.html.textile.liquid
doc/admin/metrics.html.textile.liquid
doc/admin/inspect.html.textile.liquid
doc/admin/restricting-upload-download.html.textile.liquid
doc/install/install-keepproxy.html.textile.liquid
(Line 24 talks about wb1's upload feature needing keepproxy)sdk/cwl/arvados_cwl/arvcontainer.py
Line 747sdk/go/health/aggregator.go
Lines 329, 386tools/arvbox/lib/arvbox/docker/cluster-config.sh
Line 142
I can take care of the salt installer & formula, if you like.
Updated by Tom Clegg 12 months ago
Lucas Di Pentima wrote in #note-13:
These are my observations:
[...]
Removed/updated all of the above.
I can take care of the salt installer & formula, if you like.
That would be great, thanks.
Currently trying to figure out whether the arvbox changes broke anything. It looks like it wasn't tested after updating from debian 10 to debian 12, so "arvbox build dev" was already failing. I think it needs to move back to debian 11 until #20846 merges.
20690-remove-wb1 @ f8ba6deb97126c786d97cf835b917837822bfeb1 -- developer-run-tests: #3908
Updated by Tom Clegg 12 months ago
Test failure above might be fixed by the timeout that changed in the main branch. Merged main.
20690-remove-wb1 @ 383527c82b7f8c413121165d8d6296feca9a1728 -- developer-run-tests: #3913
Updated by Tom Clegg 12 months ago
One more change needed to make arvbox build & start on debian 11.
20690-remove-wb1 @ e88556b084ba5af008e0cde991a4502d106e4d09
Updated by Lucas Di Pentima 12 months ago
Just one comment:
I tried to start a fresh arvbox (with an image from this branch) instance and got stuck. I managed to get it unstuck by running go mod tidy -compat=1.17
while being in ~/.arvbox/arvbox/arvados
(got the clue from controller's arvbox log)
After that, almost all services started except workbench2
, repeatedly reporting "Waiting for workbench2 (installing ruby gems 93 of about 106) ..." (misleading message that may be easy to remove?). Eventually workbench2
finished installing/compiling dependencies and started OK.
The rest LGTM, thanks!
Updated by Lucas Di Pentima 12 months ago
Tom Clegg wrote in #note-14:
I can take care of the salt installer & formula, if you like.
That would be great, thanks.
I can make a branch for this same ticket and you can review it. What I'm thinking to do is update the installer and/or formula so that everything related to workbench1 is set to be uninstalled. This way existing installations won't need manual intervention to remove packages/configurations. I think the arvados-formula
has support for this.
Updated by Tom Clegg 12 months ago
Lucas Di Pentima wrote in #note-17:
I tried to start a fresh arvbox (with an image from this branch) instance and got stuck. I managed to get it unstuck by running
go mod tidy -compat=1.17
while being in~/.arvbox/arvbox/arvados
(got the clue from controller's arvbox log)
Maybe Peter should weigh in on that -- I'm not sure what circumstances would lead to that, but I'm guessing it's something to do with an old source tree being checked out there already from previous arvbox use.
After that, almost all services started except
workbench2
, repeatedly reporting "Waiting for workbench2 (installing ruby gems 93 of about 106) ..." (misleading message that may be easy to remove?). Eventuallyworkbench2
finished installing/compiling dependencies and started OK.
Hm. It looks like it only prints that if "bundle install" is running somewhere, which seems odd if the only thing it's waiting for is workbench2. Haven't figured this out yet either. I have seen the "installing ruby gems" bit come and go while waiting for the same list including api, though. Perhaps the "api" service hits "touch .../api.ready" but then restarts?
Updated by Lucas Di Pentima 11 months ago
20690-remove-wb1-from-installer @ 093561a
At first I tried to make the installer ask the arvados-formula
to remove Workbench1 (instead of just not mention it) so that preexisting deployments would remove the packages. The issue was that the arvados.workbench.clean
state is too greedy and tries to uninstall several other dependencies that other services will need if running on the same node (for example, nginx
) so I ended up changing the installer in such a way that it uses the current form of the arvados-formula
that's currently in main
, but doesn't mention the arvados.workbench
at all to avoid trying to install it on new deployments.
- All agreed upon points are implemented / addressed.
- Updates installer's
check_command
so that it doesn't fail when having deprecated config keys on the config file. - Removes Workbench1's secret key handling from
local.params
and uses a dummy value. - Splits the ssl hardening nginx config snippet from the passenger configuration, so that other components can include it without having to assume that passenger is installed on the node.
- Makes the legacy
workbench
role only install thenginx
configuration that includes all the WB1 URL redirection. - Updates installer documentation to reflect Workbench1's removal.
- Adds upgrade note to reming admins to manually uninstall workbench1 packages after upgrading the installer.
- Updates installer's
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- We need to improve the
arvados-formula
versioning: https://dev.arvados.org/issues/21228
- We need to improve the
- Code is tested and passing, both automated and manual, what manual testing was done is described
- I've manually deployed a multi-host cluster successfully.
- Documentation has been updated.
- Yes
- Behaves appropriately at the intended scale (describe intended scale).
- N/A
- Considered backwards and forwards compatibility issues between client and server.
- N/A
- Follows our coding standards and GUI style guidelines.
- Yes
Updated by Brett Smith 11 months ago
Lucas Di Pentima wrote in #note-20:
20690-remove-wb1-from-installer @ 093561a
This looks good, my only comment is about the upgrade note.
- I wonder whether I should remove the package before or after the upgrade
- It would be helpful to say the full package name
arvados-workbench
- While I appreciate that it's very polite, I think some of that verbiage makes it less clear: does "please take into consideration" mean there are situations where I shouldn't remove the package?
I think text like this could be clearer:
"If you installed a previous version of Arvados with the Salt installer, and you upgrade your installer to upgrade the cluster, you should uninstall the arvados-workbench
package from the workbench instance afterwards."
Feel free to tweak that, or adapt your version, then merge. Thanks.
Updated by Lucas Di Pentima 11 months ago
- Status changed from In Progress to Resolved