Story #16029

Investigate Cypress testing framework for Workench2 integration tests

Added by Peter Amstutz 7 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Tests
Target version:
Start date:
03/14/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Some of the more important WB1's integration tests are:

  • Login
    • auto-login with api token works after redirect (check that resulting url doesn’t include the api token)
    • using expired token redirects to login page
    • access with no token should show the login page
  • Collections
    • Copy to project
    • Sharing
    • Combine collections into new collection
    • Combine files into new collection
    • URL with PDH with multiple matches
    • Remove & rename files
    • Remove/rename not available on readonly collections
    • Create collection uploading files
    • Upload errors
  • Projects
    • Anonymous access
      • Project available actions
      • Access to shared/public project’s collections
      • Access to shared/public project’s subprojects
    • Rename projects
    • Create project; move it inside other project
    • Sharing
      • Publicly share option not available when anonymous browsing disabled
      • Publicly share option available when anon browsing enabled
      • Project owner shares with another user & group
      • ’Share with group’ listing doesn’t offer projects
    • Copy/move/delete collections between projects
  • Search
    • Search for a owned project, an invalid uuid, an publicly shared project
  • Trash
    • Check deleted items on trash page
    • Test untrash button and confirm untrashed items are now listed
    • Test trash page search feature
  • Processes
    • Test ‘all processes’ page
    • Test cancel button when displaying a queued/running process
    • Run a process from a registered workflows
    • Test CR state display
  • Admin
    • Users
      • Create new user
      • Setup/unsetup active user
    • Groups
      • Create new group
      • Add/remove users to groups

Subtasks

Task #16031: Review 16029-cypress-testingResolvedPeter Amstutz

Task #16257: Review 16029-arvboot-wss-fixResolvedLucas Di Pentima


Related issues

Blocks Arvados - Story #15335: [Workbench2] Integration testingNew

Associated revisions

Revision 99d3277a
Added by Lucas Di Pentima 5 months ago

Merge branch '16029-arvboot-wss-fix'. Refs #16029

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

Revision 58c54049
Added by Lucas Di Pentima 4 months ago

Merge branch '16029-arvboot-listenhost-ssl-altname'
Refs #16029

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

Revision 27f584f8
Added by Lucas Di Pentima 3 months ago

Merge branch '16029-cypress-testing'
Closes #16029

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

History

#1 Updated by Lucas Di Pentima 7 months ago

  • Related to Story #15335: [Workbench2] Integration testing added

#2 Updated by Piotr Mrzygłowski 7 months ago

Integration with Jenkins might be conducted using example jenkinsfile:
https://docs.cypress.io/guides/guides/continuous-integration.html#Examples

#3 Updated by Piotr Mrzygłowski 7 months ago

  • Subject changed from Investigate Cyprus testing framework for Workench2 integration tests to Investigate Cypress testing framework for Workench2 integration tests

#4 Updated by Peter Amstutz 6 months ago

  • Target version changed from 2020-01-29 Sprint to 2020-02-12 Sprint

#5 Updated by Peter Amstutz 6 months ago

  • Target version changed from 2020-02-12 Sprint to 2020-02-26 Sprint

#6 Updated by Lucas Di Pentima 6 months ago

  • Assigned To changed from Piotr Mrzygłowski to Lucas Di Pentima

#7 Updated by Lucas Di Pentima 6 months ago

  • Related to deleted (Story #15335: [Workbench2] Integration testing)

#8 Updated by Lucas Di Pentima 6 months ago

  • Blocks Story #15335: [Workbench2] Integration testing added

#9 Updated by Lucas Di Pentima 6 months ago

  • Status changed from New to In Progress

#10 Updated by Peter Amstutz 5 months ago

  • Target version changed from 2020-02-26 Sprint to 2020-03-11 Sprint

#11 Updated by Peter Amstutz 5 months ago

  • Target version changed from 2020-03-11 Sprint to 2020-03-25 Sprint

#12 Updated by Lucas Di Pentima 5 months ago

  • Description updated (diff)

Reviewed the existing wb1's integration tests, listed on the description the more important ones in my opinion.

#13 Updated by Lucas Di Pentima 5 months ago

Fixed Websocket Server's ExternalURL config on arvados-boot to use wss:// instead of https:// at ef109c816 - branch 16029-arvboot-wss-fix
Test run: https://ci.arvados.org/job/developer-run-tests/1774/

#15 Updated by Tom Clegg 5 months ago

16029-arvboot-wss-fix LGTM, thanks

#16 Updated by Peter Amstutz 4 months ago

  • Target version changed from 2020-03-25 Sprint to 2020-04-08 Sprint

#17 Updated by Lucas Di Pentima 4 months ago

  • Target version changed from 2020-04-08 Sprint to 2020-04-22

#18 Updated by Lucas Di Pentima 4 months ago

Updates at commit 9bd1a28a - branch 16029-cypress-testing (wb2 repo)
Test run: https://ci.arvados.org/view/Developer/job/developer-tests-workbench2/20/ (with new workbench2-build docker image)

  • Adds Cypress as an end-to-end test runner.
  • Adds e2e tests for login operations (passing a token for now, when PAM login form is merged we could add those too).
  • Does not rely on pre-existing fixtures, but creates users via the /auth/controller/callback private endpoint.
  • Uses new arvados-server install & arvados-server boot features to have a testing cluster to test against.
  • Updates Dockerfile to make a tester image with current arvados-server boot requirements. (already built on CI)

How to manually run tests?

  • One option would be to create a docker image and use it like Jenkins does: by calling "make integration-tests" inside it. It's kind of slow because it would clone the arvados repo every time.
  • Other option would be to run arvados-server boot on a certain port, also WB2 in dev mode (yarn start) and then run CYPRESS_system_token=xxx CYPRESS_controller_url=https://localhost:port yarn run cypress run

Pending

  • Update WB2 hacking wiki to explain how to use Cypress.
  • Cypress creates screenshots (on failed tests) and a video of the entire run that would be interesting to save as a job artifact, I don't know how to configure that yet.

#19 Updated by Peter Amstutz 4 months ago

The Dockerfile has ward@curoverse as the maintainer. It should probably be "Arvados team" () or Curii ().

The testing instructions in README.md should describe how to run tests manually, starting with building the Docker image.

The Dockerfile should go in its own subdirectory, when you do "docker build" it makes a copy of the entire workbench2 directory tree but doesn't actually access any of it.

After building the docker image, here's how I ran it (starting from the workbench2 directory):

$ docker run -ti -v$PWD:$PWD -w$PWD workbench2-build make integration-tests

There's a point where you see:

Compiled successfully!

You can now view arvados-workbench-2 in the browser.

  Local:            https://localhost:38894/
  On Your Network:  https://172.17.0.2:38894/

Note that the development build is not optimized.
To create a production build, use yarn build.

and then it waits for a rather long time without any indication something is happening. Looks like it is about two minutes:

Done in 120.13s.

It should probably print a message if it is waiting for something.

It did run the tests, but everything failed with "Timed out"

  1) Login tests shows login page on first visit:
     CypressError: Timed out retrying: expected '<div#root>' to contain 'Please log in'

Now I am suspicious that when it is waiting for Arvados services to start and is done after almost exactly 2 minutes is because it timed out and not because that's how long it took. It might need a health check before proceeding.

On re-running I notice it reinstalls Cypress along with re-cloning Arvados. We should have a way of mounting those from the outside so they don't need to be redownloaded every time.

On re-running I think I confirmed that the problem it is timing out waiting for the API server to come up, it is still compiling the sassc gem (which means API server isn't up yet) when it decided to proceed with running the tests.

If I am developing and want to re-run tests with minimum turnaround time, what is going to be the right way to do that?

#20 Updated by Lucas Di Pentima 4 months ago

Peter Amstutz wrote:

The Dockerfile has ward@curoverse as the maintainer. It should probably be "Arvados team" () or Curii ().
The testing instructions in README.md should describe how to run tests manually, starting with building the Docker image.

Ok, will do!

The Dockerfile should go in its own subdirectory, when you do "docker build" it makes a copy of the entire workbench2 directory tree but doesn't actually access any of it.

I've just started a container with this image and no volumes attached and can't find wb2 sources in it, but sure I can move the file to a subdir.

After building the docker image, here's how I ran it (starting from the workbench2 directory):
[...]
Now I am suspicious that when it is waiting for Arvados services to start and is done after almost exactly 2 minutes is because it timed out and not because that's how long it took. It might need a health check before proceeding.

The best way to run it with short turnaround times is by doing what I described on note-18, second bulletpoint on 'How to manually run tests?', the first one is the way Jenkins uses.
As we discussed on chat, arvados-boot & wb2 instances are launched as background jobs, and then waited for their URLs to be valid by using wait-on (see tools/run-integration-tests.sh script).
Arvados is waited at the discovery doc's URL to ensure both controller and railsAPI are up.

If arvados-boot fails, it will exit the process and never answer its URL so I think wait-on is correct about continuing, what I can do is making the timeout a little bigger and check for wait-on's exit status so that it doesn't try to run the tests.

On re-running I notice it reinstalls Cypress along with re-cloning Arvados. We should have a way of mounting those from the outside so they don't need to be redownloaded every time.

This is because is designed to be run by Jenkins.

On re-running I think I confirmed that the problem it is timing out waiting for the API server to come up, it is still compiling the sassc gem (which means API server isn't up yet) when it decided to proceed with running the tests.

If I am developing and want to re-run tests with minimum turnaround time, what is going to be the right way to do that?

The easiest way I think would be:
  • Manually start an arvados boot instance: go run ./cmd/arvados-server boot -type test -config ~/arvados-workbench2/tools/arvados_cluster.yml -own-temporary-database -controller-address :8000 -listen-host localhost
  • Start wb2 dev instance: yarn start
  • Start Cypress interactive mode: CYPRESS_system_token=xxx CYPRESS_controller_url=https://localhost:8000 yarn run cypress open

As I commented on chat, I think it would be nice to make the 'reset db' API to accept an optional param to just empty the database without loading the fixtures, wdyt?

#21 Updated by Peter Amstutz 4 months ago

Lucas Di Pentima wrote:

The Dockerfile should go in its own subdirectory, when you do "docker build" it makes a copy of the entire workbench2 directory tree but doesn't actually access any of it.

I've just started a container with this image and no volumes attached and can't find wb2 sources in it, but sure I can move the file to a subdir.

I mean, the way docker works is it makes a copy of the whole directory tree under where the Dockerfile sits in before building so you can use "ADD" and "COPY" to put files into the container, but if you don't do use ADD or COPY, it just wastes time/disk space making a copy you don't use and is thrown away after building the image. I don't mean it copies it into the container.

The best way to run it with short turnaround times is by doing what I described on note-18, second bulletpoint on 'How to manually run tests?', the first one is the way Jenkins uses.

Ok. At least when using Docker this should still be streamlined into a script, but that might be a different script from the jenkins one.

As we discussed on chat, arvados-boot & wb2 instances are launched as background jobs, and then waited for their URLs to be valid by using wait-on (see tools/run-integration-tests.sh script).
Arvados is waited at the discovery doc's URL to ensure both controller and railsAPI are up.

If arvados-boot fails, it will exit the process and never answer its URL so I think wait-on is correct about continuing, what I can do is making the timeout a little bigger and check for wait-on's exit status so that it doesn't try to run the tests.

That sounds right. The other thing to try is waiting on the health check aggregator which would only return a good status if when everything is up.

The easiest way I think would be:
  • Manually start an arvados boot instance: go run ./cmd/arvados-server boot -type test -config ~/arvados-workbench2/tools/arvados_cluster.yml -own-temporary-database -controller-address :8000 -listen-host localhost
  • Start wb2 dev instance: yarn start
  • Start Cypress interactive mode: CYPRESS_system_token=xxx CYPRESS_controller_url=https://localhost:8000 yarn run cypress open

Yes, so there should a second different script to start the interactive case.

As I commented on chat, I think it would be nice to make the 'reset db' API to accept an optional param to just empty the database without loading the fixtures, wdyt?

I think we discussed this before and already agreed to do it, so yes.

#22 Updated by Peter Amstutz 4 months ago

Some more notes:

  • As previously discussed, run-integration-tests.sh instead of "wait-for" it should use the health check aggregator endpoint to determine if the cluster is ready to run, and have a longer time out (at least 5 minutes) to accommodate packages being downloaded/installed in a fresh install
  • It's essential to either do "REACT_APP_ARVADOS_API_HOST=... yarn start" and/or to update public/config.json, otherwise workbench2 won't know how to contact the test API server
  • To run a container that is capable of running cypress in interactive mode, you need to do something like this:
    docker run -ti \
        -w$PWD \
        --env="DISPLAY" \
        --volume=$PWD:$PWD \
        --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
        --volume="$HOME/work/arvados:$HOME/work/arvados:rw" \
        --shm-size 2g \
        workbench2-build \
        /bin/bash
    
  • To give the browser access to your screen, on your host you need to do something like:
xhost +local:`docker inspect --format='{{ .Config.Hostname }}' docker_container_name`

Tomorrow I can finally get on to reviewing the actual code of the actual tests :-)

#23 Updated by Peter Amstutz 4 months ago

The tests look good. I am very happy at how short they are. After struggling with the orchestration it is nice that the actual tests are easy.

I have one more comment. How hard would it be to convert the tests and support commands to Typescript? Cypress supports it:

https://docs.cypress.io/guides/tooling/typescript-support.html

#24 Updated by Lucas Di Pentima 4 months ago

Made SSL certificate's subjectAltName to include IP or DNS depending on the -listen-host argument of arvados-server boot at 7eb530d63 - branch 16029-arvboot-listenhost-ssl-altname

#25 Updated by Lucas Di Pentima 4 months ago

Peter Amstutz wrote:

The tests look good. I am very happy at how short they are. After struggling with the orchestration it is nice that the actual tests are easy.

I have one more comment. How hard would it be to convert the tests and support commands to Typescript? Cypress supports it:

https://docs.cypress.io/guides/tooling/typescript-support.html

I did tried a little bit but failed and didn't wanted to add more complexity to the task. I agree that this would be ideal, I can give it another look.

#26 Updated by Lucas Di Pentima 4 months ago

Update at b191ca173 - branch 16029-arvboot-health-externalurl Makes the health aggregator accessible from the outside of arvados boot. This is to make the integration test script able to ask for its port by accessing the exported config endpoint.

#27 Updated by Lucas Di Pentima 4 months ago

Branch on previous comment isn't needed as arvados boot returns its URL when all the health checks are OK.
Updated orchestration script to follow the pattern of https://github.com/arvados/arvados/blob/master/lib/boot/example.sh at commit 855193b8 (wb2 repo)
Test run: https://ci.arvados.org/view/Developer/job/developer-tests-workbench2/21/

#28 Updated by Lucas Di Pentima 4 months ago

Update at commit 0c1be947 (wb2 repo)
Test run: https://ci.arvados.org/view/Developer/job/developer-tests-workbench2/22/

  • Adds options to the orchestration script to allow interactive mode (useful for developing)
Usage: run-integration-tests.sh [options]
Options:
  -i            Run Cypress in interactive mode.
  -a PATH       Arvados dir. If PATH doesn't exist, a repo clone is downloaded there.
  -w PATH       Workbench2 dir. Default: Current working directory

#29 Updated by Lucas Di Pentima 3 months ago

  • Target version changed from 2020-04-22 to 2020-05-06 Sprint
  • Category set to Tests

#30 Updated by Peter Amstutz 3 months ago

Awesome. This looks good to me. I pushed a couple of minor tweaks to 16029-cypress-testing -- now f3569ce613f0e19a63149c08be68e0bcbb7eaf92

Please merge and let's get it working on Jenkins.

#31 Updated by Anonymous 3 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF