Support #18874
closedMerge workbench2 history into main Arvados repo
Added by Peter Amstutz over 2 years ago. Updated 4 months ago.
Description
Import entire workbench2 history into main Arvados repo.
Want to preserve the commit hashes, so wb2 history will go back to a different root commit.
Merge point will move workbench2 to arvados/apps/workbench2
Update build scripts, arvados-boot, arvbox etc to reflect new location of workbench2 inside the main tree.
From discussion:
Tom will figure out the git merge and updating run-tests to be able to perform the workbench2 tests.
Once this is ready Tom & Lucas will update Jenkins & other scripts (eg package building).
Updating services/workbench2 during transition period¶
Temporarily (after the initial subtree merge into the arvados repo, but before we have stopped pushing changes to the workbench2 repo) this is the recipe for syncing changes from main workbench2.git to main
arvados.git/services/workbench2:
$ git fetch git@git.arvados.org:arvados-workbench2 main:workbench2-main $ git subtree merge -P services/workbench2 workbench2-main
Use a commit message like this, to satisfy git.arvados.org policies:
Merge branch 'main' from workbench2.git refs #18874 Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Transition progress¶
steps:- ✅ add Jenkins jobs for testing the version of workbench2 in the arvados repo
- ✅ merge latest main branch of workbench2 into the arvados tree at services/workbench2/ using git-subtree
- ✅ use the new
arvados-server install
to rebuild the machine image jenkins uses to run arvados tests (see ENOSPC note above) - ✅ run Jenkins jobs manually, fix any other problems that still prevent services/workbench2 tests from passing
- ✅ configure Jenkins to run the new workbench2 tests as part of the developer-run-tests and run-tests pipelines
- ✅ update arvbox to use checked-out services/workbench2 instead of arvados-workbench2.git
- ✅ update debian package-building to build workbench2 packages from arvados.git/services/workbench2
- ✅ stop building packages from arvados-workbench2.git
- ✅ merge latest main branch from arvados-workbench2.git into arvados.git/services/workbench2
- ✅ prohibit pushing updates to
main branch atarvados-workbench2.git (from now on devs will push to arvados.git/services/workbench2 instead)
Related issues
Updated by Peter Amstutz over 2 years ago
- Blocked by Support #18873: Add hooks to limit which paths of arvados repo non-curii employees can push to added
Updated by Peter Amstutz over 2 years ago
- Target version changed from Kanban to 2022-04-27 Sprint
- Project changed from 40 to Arvados
Updated by Peter Amstutz over 2 years ago
- Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Updated by Peter Amstutz over 2 years ago
- Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Updated by Peter Amstutz over 2 years ago
- Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Updated by Peter Amstutz over 2 years ago
- Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-06-22 Sprint to 2022-07-06
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-07-06 to 2022-07-20
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-07-20 to 2022-08-03 Sprint
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-08-03 Sprint to 2022-08-17 sprint
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-08-17 sprint to 2022-08-31 sprint
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-08-31 sprint to 2022-09-14 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-09-14 sprint to 2022-09-28 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-09-28 sprint to 2022-10-12 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-10-26 sprint to 2022-11-09 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-11-09 sprint to 2022-11-23 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-11-23 sprint to 2022-12-07 Sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-12-07 Sprint to 2022-12-21 Sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-12-21 Sprint to 2023-01-18 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2023-01-18 sprint to 2023-02-01 sprint
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2023-02-01 sprint to 2023-02-15 sprint
Updated by Peter Amstutz over 1 year ago
- Target version changed from 2023-02-15 sprint to 2023-02-01 sprint
Updated by Peter Amstutz over 1 year ago
- Target version changed from 2023-02-01 sprint to Future
Updated by Lucas Di Pentima over 1 year ago
- Assigned To set to Lucas Di Pentima
Updated by Lucas Di Pentima over 1 year ago
- Story points set to 1.0
The most potentially complex task in my eyes would be the repository merge, but it seems it's totally doable. This SO question has some clues about how to do it:
https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories
In the past we talked about using the wb2 repo as a submodule of the arvados repo, but I think it would be more useful in this case to just merge it like explained in this answer:
https://stackoverflow.com/a/10548919
...the feature explained here seemingly allows keeping both histories.
So with that out of the way, I think it would take a full point to update the build scripts, CI pipelines, and so on.
Updated by Peter Amstutz about 1 year ago
- Related to Idea #17001: Arvados uses WB2 by default added
Updated by Peter Amstutz about 1 year ago
- Story points changed from 1.0 to 3.0
- Target version changed from Future to Development 2023-08-30
- Assigned To changed from Lucas Di Pentima to Tom Clegg
Updated by Tom Clegg almost 1 year ago
18874-merge-wb2 @ 102fefc02fecec15d801d3d457aad8148be89dbc -- developer-run-tests: #3795
18874-merge-wb2 @ bfaee40696c3c15556ef089e69da47bb832b08db -- developer-run-tests: #3801
18874-merge-wb2 @ 608f3e19607e00b76ef9fb8cedef8124a31ceff7 -- developer-run-tests: #3802
18874-merge-wb2 @ 608f3e19607e00b76ef9fb8cedef8124a31ceff7 -- developer-run-tests: #3804
Status:
Jenkins "remainder" test configs are updated to skip wb2.
Jenkins jobs for testing wb2 are added (but not yet attached to the main run-tests jobs as dependencies).
wb2 unit tests pass.
wb2 integration tests fail:
Error: ENOSPC: System limit for number of file watchers reached, watch '/tmp/workspace/developer-run-tests-services-workbench2/services/workbench2/public/webshell/README'
I updated arvados-server install
to do the "fs.inotify.max_user_watches thing":Running tests to avoid that.
I think it would make sense to merge this to main, then update the Jenkins worker image using the new arvados-server install
in order to get the fs.inotify.max_user_watches thing fixed, then see what else needs to be done to make integration tests pass.
Updated by Tom Clegg 12 months ago
Now that this branch merges a subtree and then main, I don't know a single git command to show the succinct diff. But this gives the changes I made to the workbench2 tree:
git diff 2f83fcd45b...18874-merge-wb2
...and this shows the changes I made to the arvados tree:
git diff 29ad3b7333...18874-merge-wb2 lib build .licenseignore
I made two changes to the workbench2 test scripts that might affect other dev workflows.
I modified Makefile
so make unit-tests
and make yarn-install
don't implicitly run arvados-server install
, which requires root. I updated the *-tests-in-docker
rules so they run make arvados-server-install
in the container explicitly.
I modified tools/run-integration-tests.sh
so it doesn't try to build arvados-server
or run arvados-server install
if the arvados-server binary already exists in GOPATH. I suspect these steps shouldn't be there at all -- shouldn't the caller be running "make arvados-server-install" explicitly if they want to update apt dependencies etc.?
Updated by Peter Amstutz 12 months ago
- Target version changed from Development 2023-08-30 to Development 2023-09-13 sprint
Updated by Tom Clegg 12 months ago
Offline we decided we don't want to merge this before 2.7. It wouldn't affect anything functionally, but we might as well avoid creating extra risk of confusion by having workbench2 code in the arvados source tree that's tagged 2.7 but is actually a bit older/different than the released version 2.7 of workbench2.
Updated by Peter Amstutz 12 months ago
- Target version changed from Development 2023-09-13 sprint to Development 2023-09-27 sprint
Updated by Peter Amstutz 12 months ago
- Target version changed from Development 2023-09-27 sprint to To be scheduled
Updated by Peter Amstutz 11 months ago
- Target version changed from To be scheduled to Development 2023-10-11 sprint
Updated by Tom Clegg 11 months ago
merging changes on wb2 main branch since the initial subtree merge:
18874-merge-wb2 @ 0f5b0542513b572959e39400bae42e69aeb1a7b6 -- developer-run-tests: #3838
- All agreed upon points are implemented / addressed.
- There will be more steps after merging this branch.
- ✅ add Jenkins jobs for testing the version of workbench2 in the arvados repo
- ▢ (you are here) merge latest main branch of workbench2 into the arvados tree at services/workbench2/ using git-subtree
- ▢ use the new
arvados-server install
to rebuild the machine image jenkins uses to run arvados tests (see ENOSPC note above) - ▢ run Jenkins jobs manually, fix any other problems that still prevent services/workbench2 tests from passing
- ▢ configure Jenkins to run the new workbench2 tests as part of the developer-run-tests and run-tests pipelines
- ▢ update arvbox to use checked-out services/workbench2 instead of arvados-workbench2.git
- ▢ update debian package-building to use arvados.git/services/workbench2 instead of arvados-workbench2.git
- ▢ merge latest main branch from arvados-workbench2.git into arvados.git/services/workbench2
- ▢ prohibit pushing updates to main branch at arvados-workbench2.git (from now on devs will push to arvados.git/services/workbench2 instead)
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- I think we can use this ticket to track all the above steps?
- Code is tested and passing, both automated and manual, what manual testing was done is described
- The non-workbench2 tests pass, and the workbench2 tests aren't enabled yet because jenkins image isn't suitable yet (see ENOSPC)
- Documentation has been updated.
- todo
- Behaves appropriately at the intended scale (describe intended scale).
- N/A
- Considered backwards and forwards compatibility issues between client and server.
- N/A
- Follows our coding standards and GUI style guidelines.
- ✅
Updated by Lucas Di Pentima 11 months ago
Just one observation:
I think the original idea was to put WB2 inside apps/workbench2
so just commenting this in case it was overlooked.
Other than that, it LGTM
Updated by Tom Clegg 11 months ago
side note: in order to push the main branch even though it contains old commits that don't satisfy our copyright-header requirements, I temporarily monkeypatched /data/usr-local-arvados-dev/git/hooks/check-copyright-headers.rb on git.arvados.org with the merge commit hash:
def check_copyright_headers
if ($newrev[0,6] == '000000')
# A branch is being deleted. Do not check old commits for DCO signoff!
all_objects = []
commits = []
+ elsif $newrev =~ /^df1ebc0e31/
+ # workbench2 subtree merge
+ all_objects = []
+ commits = []
elsif ($oldrev[0,6] == '000000')
Updated by Peter Amstutz 11 months ago
- Target version changed from Development 2023-10-11 sprint to Development 2023-10-25 sprint
Updated by Peter Amstutz 10 months ago
- Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Updated by Peter Amstutz 10 months ago
- Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Updated by Tom Clegg 10 months ago
18874-merge-wb2 @ 6ff39245b93dd55e60c7433af64f2244b0091c6e
developer-run-tests-services-workbench2: #23
On jenkins, all tests consistently pass except process.spec.js "displays IO parameters with no value" consistently fails (when setting the token at the beginning of the test).
AssertionError: object tested must be an array, a map, an object, a set, a string, or a weakset, but undefined given
On my dev machine, all tests consistently pass except process.spec.js "displays IO parameters with keep links and previews" consistently fails (much later in the test, not when setting the token at the beginning). Multiple times in both headless and interactive modes.
Updated by Tom Clegg 10 months ago
I don't understand it, but this seems to work great:
cy.visit(`/token/?api_token=${user.token}`);
- cy.url({ timeout: 10000 }).should("contain", "/projects/");
+ cy.url().should("contain", "/projects/");
cy.get("div#root").should("contain", "Arvados Workbench (zzzzz)");
Updated by Tom Clegg 9 months ago
- Description updated (diff)
18874-merge-wb2 @ f5fd7e2dc13b912e1032a54119f3fb60973f77e2 -- developer-run-tests: #3900
developer-run-tests-services-workbench2: #28
It seems #note-50 did not work great, the next two attempts were just lucky. The next two after that were not lucky, and failed. I increased the timeout to 15s and the next build passed.
Changes:- Fix lib/install to pass a real git hash to the wb2 build, if available
- Fix wb2 integration test script to use the current arvados source tree instead of cloning fresh
- Fix lib/boot workbench2 script failing in dev/test mode
- Update cmd/arvados-package to test on debian:11 instead of debian:10
- wb2 integration tests pass on jenkins (assuming the 15s timeout mentioned above is really enough)
Updated by Tom Clegg 9 months ago
- Add workbench2 to package-building scripts in source:build/ →
packages/*/arvados-workbench2_3.0.0~rc0-1_amd64.deb
- Update scripts to fix a couple of build problems
The new Dockerfiles for debian12 and ubuntu2204 in #20846 (unmerged) will need to have these nodejs/npm changes applied as well. Waiting to see which branch merges first.
I think coordinating this will look like- disable the current workbench2 package-building job in jenkins config
- merge this branch
- verify the new dev package gets built
- update documented release process
Updated by Lucas Di Pentima 9 months ago
Tom Clegg wrote in #note-54:
18874-merge-wb2 @ 5917dffb6d14ec54ff2afa0f2dcc16095609e0dd
- Add workbench2 to package-building scripts in source:build/ →
packages/*/arvados-workbench2_3.0.0~rc0-1_amd64.deb
- Update scripts to fix a couple of build problems
I've found just one issue that was fixed with:
diff --git a/build/run-library.sh b/build/run-library.sh
index 3eab0f506..dd878d847 100755
--- a/build/run-library.sh
+++ b/build/run-library.sh
@@ -124,6 +124,7 @@ package_workbench2() {
local version="$(version_from_git)"
cd "$WORKSPACE/$src"
rm -rf ./build
+ NODE_ENV=production yarn install
VERSION="$VERSION" BUILD_NUMBER="$(default_iteration "$pkgname" "$version" yarn)" GIT_COMMIT="$(git rev-parse HEAD | head -c9)" yarn build
cd "$WORKSPACE/packages/$TARGET"
fpm_build "${WORKSPACE}/$src" "${WORKSPACE}/$src/build/=$dst" "$pkgname" dir "$version" \
The rest LGTM, thanks.
Updated by Peter Amstutz 8 months ago
- Target version changed from Development 2024-01-03 sprint to Development 2024-01-17 sprint