Idea #17535
closedJenkins step to test provision.sh + salt formula
Added by Peter Amstutz over 3 years ago. Updated almost 3 years ago.
Description
Somewhere in the Jenkins pipeline after development packages are built:
1) Spin up a Jenkins worker from a slim image (possibly multiple OS versions)
2) Run provision.sh / salt to install and configure Arvados on the worker
3) Run the diagnostics + smoke tests (hasher pipeline)
4) Report results
5) Shut everything down
Related issues
Updated by Javier Bértoli over 3 years ago
- Status changed from New to In Progress
Updated by Javier Bértoli over 3 years ago
- Subject changed from Jenkins step to test terraform + provision.sh + salt formula to Jenkins step to test provision.sh + salt formula
Added a test Saltstack arvados-formula pipeline, with multi-configuration matrix (to enable testing Debian and Centos if desired), using a tests-2.1
worker node.
The job:
- copies the 'single_host/multi_hostname' configuration examples
- copies the 'local.params' example and modifies it to be usable (same as the local Vagrant script does)
- runs the provision script
- runs the
runs-test.sh
script in the @arvados/tools/salt-install/tests' directory, which:- verifies the SSL snake-oil certificates are in place
- creates the Arvados Standard Docker Images project
- uploads an
arvados/jobs
image to the project - creates the initial user if it does not exist
- activates the user
- runs the
hasher-workflow.cwl
using the user's credentials
Run successfully an initial test with the current branch I'm testing for #17246
Still pending to add this job in one of the pipelines.
Updated by Javier Bértoli over 3 years ago
Added the job in the Arvados Build Pipeline, as a downstream job after build-packages-debian10 success.
Modified the "Execute Shell" step to:
- use the development release (so it the formula uses the
arvados-dev
repo) - set VERSION to "latest" (so it picks the newest package in the repo
Updated by Peter Amstutz over 3 years ago
- Target version changed from 2021-04-28 bughunt sprint to 2021-05-12 sprint
Updated by Javier Bértoli over 3 years ago
The jenkins run is failing because the job is misconfigured:
- To test packages built in the
development
pipeline (https://ci.arvados.org/job/test-saltstack-arvados-formula/, triggered after https://ci.arvados.org/job/build-packages-debian10/), it should:- pull packages from the
-dev
repo - install latest package available.
- pull packages from the
- To test the
release
pipeline (to test release candidates, jenkins job to be created, to be triggered after https://ci.arvados.org/view/Release%20Pipeline/job/build-publish-packages-testing-debian10/), it should:- pull packages from the
-testing
repo - install latest package available.
- pull packages from the
Updated by Javier Bértoli over 3 years ago
Fixed the development run (sed regex issue in the job configuration), rerun latest and passed
Updated by Peter Amstutz over 3 years ago
- Target version changed from 2021-05-12 sprint to 2021-05-26 sprint
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-05-26 sprint to 2021-06-09 sprint
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-06-09 sprint to 2021-06-23 sprint
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-06-23 sprint to 2021-07-07 sprint
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-07-07 sprint to 2021-07-21 sprint
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-07-21 sprint to 2021-08-04 sprint
Updated by Javier Bértoli about 3 years ago
- Added a new cloud of instances in Jenkins, with "vanilla images" for the OSes we support, named gce2-vanilla
- Added a Centos 7 job https://ci.arvados.org/view/All/job/test-provision-centos7/
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-08-04 sprint to 2021-08-18 sprint
Updated by Javier Bértoli about 3 years ago
- Blocked by Bug #17990: [deployment][arvados-formula] ubuntu 18.04 needs to install using rvm for api and workbench added
Updated by Javier Bértoli about 3 years ago
All is working now
Added a multi-job pipeline, test-provision, which accepts two parameters:
- git_hash, arvados commit (as we do with all our other jobs) and
- ARVADOS_FORMULA_BRANCH, to pass the arvados-formula branch to pass to the provision.sh script.
after "build-packages-multijob" completes successfully.
Updated by Javier Bértoli about 3 years ago
Added the jobs to the "release pipeline"
Updated by Javier Bértoli about 3 years ago
- Status changed from In Progress to Feedback
Updated by Javier Bértoli about 3 years ago
84e52fd52, branch 17535-test-provision-jenkins
Updated by Javier Bértoli about 3 years ago
- Status changed from Feedback to In Progress
Updated by Peter Amstutz about 3 years ago
- Target version changed from 2021-08-18 sprint to 2021-09-01 sprint
Updated by Ward Vandewege about 3 years ago
I'm wondering about the duplication of all those nginx sls files: there are three copies of each under `tools/salt-install/config_examples`, and another copy in the arvados-formula repo under `test/salt/pillar/examples`. I looked at some of the differences, I think there may be some room to make all copies more consistent with each other.
And maybe some trimming/consolidation is also in order. Anyway, none of that should block this, LGTM.
Updated by Javier Bértoli about 3 years ago
I had the same thoughts regarding duplication. I just hate it.
The examples in the formula are there for those that expect to use the formula without using the provision scripts.
And the duplication in the provision scripts are because each directory is self-contained and to simplify the deployment process, I suggested users just choose a configuration variant and copy the dir to a 'working dir'.
De-duplicating means we will not allow users to just 'copy the whole dir to a place, modify it and use it', unless we find an alternative way to do this?
Updated by Javier Bértoli about 3 years ago
- % Done changed from 33 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|d3b76845c740935f7474f535d308303c748b0b4a.
Updated by Javier Bértoli about 3 years ago
- Status changed from Resolved to In Progress
Merged. It's working
Updated by Peter Amstutz almost 3 years ago
- Target version changed from 2021-09-01 sprint to 2021-09-15 sprint
Updated by Javier Bértoli almost 3 years ago
- Status changed from In Progress to Resolved