Arvados

18874-merge-wb2 @ 102fefc02fecec15d801d3d457aad8148be89dbc -- developer-run-tests: #3795

18874-merge-wb2 @ bfaee40696c3c15556ef089e69da47bb832b08db -- developer-run-tests: #3801

18874-merge-wb2 @ 608f3e19607e00b76ef9fb8cedef8124a31ceff7 -- developer-run-tests: #3802

18874-merge-wb2 @ 608f3e19607e00b76ef9fb8cedef8124a31ceff7 -- developer-run-tests: #3804

Status:

Jenkins "remainder" test configs are updated to skip wb2.

Jenkins jobs for testing wb2 are added (but not yet attached to the main run-tests jobs as dependencies).

wb2 unit tests pass.

wb2 integration tests fail:

Error: ENOSPC: System limit for number of file watchers reached, watch '/tmp/workspace/developer-run-tests-services-workbench2/services/workbench2/public/webshell/README'

I updated arvados-server install to do the "fs.inotify.max_user_watches thing":Running tests to avoid that.

I think it would make sense to merge this to main, then update the Jenkins worker image using the new arvados-server install in order to get the fs.inotify.max_user_watches thing fixed, then see what else needs to be done to make integration tests pass.

Now that this branch merges a subtree and then main, I don't know a single git command to show the succinct diff. But this gives the changes I made to the workbench2 tree:

git diff 2f83fcd45b...18874-merge-wb2

...and this shows the changes I made to the arvados tree:

git diff 29ad3b7333...18874-merge-wb2 lib build .licenseignore

I made two changes to the workbench2 test scripts that might affect other dev workflows.

I modified Makefile so make unit-tests and make yarn-install don't implicitly run arvados-server install, which requires root. I updated the *-tests-in-docker rules so they run make arvados-server-install in the container explicitly.

I modified tools/run-integration-tests.sh so it doesn't try to build arvados-server or run arvados-server install if the arvados-server binary already exists in GOPATH. I suspect these steps shouldn't be there at all -- shouldn't the caller be running "make arvados-server-install" explicitly if they want to update apt dependencies etc.?

Offline we decided we don't want to merge this before 2.7. It wouldn't affect anything functionally, but we might as well avoid creating extra risk of confusion by having workbench2 code in the arvados source tree that's tagged 2.7 but is actually a bit older/different than the released version 2.7 of workbench2.

merging changes on wb2 main branch since the initial subtree merge:

18874-merge-wb2 @ 0f5b0542513b572959e39400bae42e69aeb1a7b6 -- developer-run-tests: #3838

• All agreed upon points are implemented / addressed.
  • There will be more steps after merging this branch.
  • ✅ add Jenkins jobs for testing the version of workbench2 in the arvados repo
  • ▢ (you are here) merge latest main branch of workbench2 into the arvados tree at services/workbench2/ using git-subtree
  • ▢ use the new arvados-server install to rebuild the machine image jenkins uses to run arvados tests (see ENOSPC note above)
  • ▢ run Jenkins jobs manually, fix any other problems that still prevent services/workbench2 tests from passing
  • ▢ configure Jenkins to run the new workbench2 tests as part of the developer-run-tests and run-tests pipelines
  • ▢ update arvbox to use checked-out services/workbench2 instead of arvados-workbench2.git
  • ▢ update debian package-building to use arvados.git/services/workbench2 instead of arvados-workbench2.git
  • ▢ merge latest main branch from arvados-workbench2.git into arvados.git/services/workbench2
  • ▢ prohibit pushing updates to main branch at arvados-workbench2.git (from now on devs will push to arvados.git/services/workbench2 instead)
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • I think we can use this ticket to track all the above steps?
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • The non-workbench2 tests pass, and the workbench2 tests aren't enabled yet because jenkins image isn't suitable yet (see ENOSPC)
• Documentation has been updated.
  • todo
• Behaves appropriately at the intended scale (describe intended scale).
  • N/A
• Considered backwards and forwards compatibility issues between client and server.
  • N/A
• Follows our coding standards and GUI style guidelines.
  • ✅

Just one observation:

I think the original idea was to put WB2 inside apps/workbench2 so just commenting this in case it was overlooked.

Other than that, it LGTM

side note: in order to push the main branch even though it contains old commits that don't satisfy our copyright-header requirements, I temporarily monkeypatched /data/usr-local-arvados-dev/git/hooks/check-copyright-headers.rb on git.arvados.org with the merge commit hash:

 def check_copyright_headers
   if ($newrev[0,6] ==  '000000')
     # A branch is being deleted. Do not check old commits for DCO signoff! 
     all_objects = []
     commits = []
+  elsif $newrev =~ /^df1ebc0e31/
+    # workbench2 subtree merge
+    all_objects = []
+    commits = []
   elsif ($oldrev[0,6] ==  '000000')

18874-merge-wb2 @ 6ff39245b93dd55e60c7433af64f2244b0091c6e

developer-run-tests-services-workbench2: #23

On jenkins, all tests consistently pass except process.spec.js "displays IO parameters with no value" consistently fails (when setting the token at the beginning of the test).

     AssertionError: object tested must be an array, a map, an object, a set, a string, or a weakset, but undefined given

On my dev machine, all tests consistently pass except process.spec.js "displays IO parameters with keep links and previews" consistently fails (much later in the test, not when setting the token at the beginning). Multiple times in both headless and interactive modes.

I don't understand it, but this seems to work great:

     cy.visit(`/token/?api_token=${user.token}`);
-    cy.url({ timeout: 10000 }).should("contain", "/projects/");
+    cy.url().should("contain", "/projects/");
     cy.get("div#root").should("contain", "Arvados Workbench (zzzzz)");

developer-run-tests-services-workbench2: #24

developer-run-tests-services-workbench2: #25

18874-merge-wb2 LGTM, thanks.

• Add workbench2 to package-building scripts in source:build/ → packages/*/arvados-workbench2_3.0.0~rc0-1_amd64.deb
• Update scripts to fix a couple of build problems

The new Dockerfiles for debian12 and ubuntu2204 in #20846 (unmerged) will need to have these nodejs/npm changes applied as well. Waiting to see which branch merges first.

1. disable the current workbench2 package-building job in jenkins config
2. merge this branch
3. verify the new dev package gets built
4. update documented release process

Tom Clegg wrote in #note-54:

18874-merge-wb2 @ 5917dffb6d14ec54ff2afa0f2dcc16095609e0dd

• Add workbench2 to package-building scripts in source:build/ → packages/*/arvados-workbench2_3.0.0~rc0-1_amd64.deb
• Update scripts to fix a couple of build problems

I've found just one issue that was fixed with:

diff --git a/build/run-library.sh b/build/run-library.sh
index 3eab0f506..dd878d847 100755
--- a/build/run-library.sh
+++ b/build/run-library.sh
@@ -124,6 +124,7 @@ package_workbench2() {
     local version="$(version_from_git)" 
     cd "$WORKSPACE/$src" 
     rm -rf ./build
+    NODE_ENV=production yarn install
     VERSION="$VERSION" BUILD_NUMBER="$(default_iteration "$pkgname" "$version" yarn)" GIT_COMMIT="$(git rev-parse HEAD | head -c9)" yarn build
     cd "$WORKSPACE/packages/$TARGET" 
     fpm_build "${WORKSPACE}/$src" "${WORKSPACE}/$src/build/=$dst" "$pkgname" dir "$version" \

The rest LGTM, thanks.