Support #16873

Jenkins job that uses "kitchen test" to test salt formulas against all supported distros

Added by Peter Amstutz about 1 year ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Deployment
Target version:
Start date:
06/30/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

A new jenkins pipeline job which takes a given salt formula version and a given Arvados version (including dev) and performs installation and correctness checking using Dockerized environments of each supported distro.


Subtasks

Task #16880: Review #16873 - Jenkins job that uses "kitchen test" to test salt formulas against all supported distrosResolvedJavier Bértoli

Task #17844: [jenkins] create an worker image to run arvados-formula testingResolvedJavier Bértoli

History

#1 Updated by Peter Amstutz about 1 year ago

  • Description updated (diff)
  • Subject changed from Jenkins job that test-runs salt install in Docker against all supported distros to Jenkins job that uses "kitchen test" to test salt formulas against all supported distros
  • Tracker changed from Bug to Support

#2 Updated by Peter Amstutz about 1 year ago

  • Assigned To set to Javier Bértoli

#3 Updated by Peter Amstutz 12 months ago

  • Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint

#4 Updated by Peter Amstutz 12 months ago

  • Target version changed from 2020-10-21 Sprint to 2020-11-04 Sprint

#5 Updated by Peter Amstutz 11 months ago

  • Target version changed from 2020-11-04 Sprint to 2020-11-18

#6 Updated by Peter Amstutz 11 months ago

  • Target version deleted (2020-11-18)

#8 Updated by Javier Bértoli 3 months ago

There's a WIP, disabled

#9 Updated by Javier Bértoli 3 months ago

  • Target version set to 2021-07-07 sprint
  • Status changed from New to In Progress
  • Category set to Deployment

#10 Updated by Javier Bértoli 3 months ago

  • Status changed from In Progress to Feedback

Enabled the jenkins job

It uses the jenkins-image-arvados-formula-tests-{date} image created on #17844, through a cloud worker, using the label saltstack-tests.

It runs kitchen to verify the follow combinations:

$ kitchen list "debian-10-3003|centos-7-3003" 
Instance                        Driver  Provisioner  Verifier  Transport  Last Action    Last Error
api-debian-10-3003-0-py3        Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>
api-centos-7-3003-0-py3         Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>
workbench-debian-10-3003-0-py3  Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>
workbench-centos-7-3003-0-py3   Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>
shell-debian-10-3003-0-py3      Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>
shell-centos-7-3003-0-py3       Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>
keepstore-debian-10-3003-0-py3  Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>
keepstore-centos-7-3003-0-py3   Docker  SaltSolo     Inspec    Ssh        <Not Created>  <None>

(run kitchen list to see all the available)

The naming of the suites follow the kitchen-ci format: suite_name-platform, which are built following the Saltstack-formulas images (for more details see the suites and platforms sections in the kitchen.yml file)

Note that some arvados components are applied and tested on the same image, to speed up testing.

Also, tests are run concurrently for the same reason.

At the moment, centos tests are failing because of #17750, #17787, which is expected.

Didn't enable the ubuntu testing because debian should cover it, but can be enabled if desired. It's just a matter of adding an entry in the regex

#11 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2021-07-07 sprint to 2021-07-21 sprint

#12 Updated by Nico César 3 months ago

At the end there is some extra info:

>>>>>>     Converge failed on instance <workbench-centos-7-3003-0-py3>.  Please see .kitchen/logs/workbench-centos-7-3003-0-py3.log for more details
>>>>>>     Converge failed on instance <api-centos-7-3003-0-py3>.  Please see .kitchen/logs/api-centos-7-3003-0-py3.log for more details

are the .kitchen/logs/* files lost in the worker instance that runs it? if that's the case, can we add it to "workspace files" to keep them?

#13 Updated by Nico César 3 months ago

I also noticed that "Color ANSI Console Output" is enabled for that job, butsomehow the colors are not there and the escape characters are... is there an easy way to disable colors all together?

#15 Updated by Javier Bértoli 3 months ago

Nico César wrote:

I also noticed that "Color ANSI Console Output" is enabled for that job, butsomehow the colors are not there and the escape characters are... is there an easy way to disable colors all together?

There are ways and I tried a few (with --no-color, txt, yaml)

It seems to be an issue with the ANSIColor plugin, because other parts of the output are correctly colored and when the run ends OK, the output is not garbled even where it fails in the other cases.

I'll enable "yaml", as it seems to be the only one that is correctly managed by the ANSIcolor plugin.

#16 Updated by Javier Bértoli 3 months ago

Nico César wrote:

I also noticed that "Color ANSI Console Output" is enabled for that job, butsomehow the colors are not there and the escape characters are... is there an easy way to disable colors all together?

Kept the formula code as-is, so running the formula on a local workspace works as usual, but modified the jenkins' job configuration to add a kitchen.local.yml" file with a @yaml outputter, so it's readable in jenkins.

A weird twist is that, adding an invalid output module, produces a yaml output as the one above, but that is parseable by jenkins and, therefore, colored

#17 Updated by Javier Bértoli 3 months ago

Nico César wrote:

At the end there is some extra info:

[...]

are the .kitchen/logs/* files lost in the worker instance that runs it? if that's the case, can we add it to "workspace files" to keep them?

The .kitchen/logs/* files are a per-suite file with the output already seen in jenkins' output. I've used them in very rare situations, but we can add them, I guess as artifacts?

#18 Updated by Peter Amstutz 2 months ago

  • Target version changed from 2021-07-21 sprint to 2021-08-04 sprint

#19 Updated by Javier Bértoli 2 months ago

Modified the job to save the logs as artifacts as suggested

#20 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2021-08-04 sprint to 2021-08-18 sprint

#21 Updated by Javier Bértoli about 2 months ago

Added a few more changes to fix Nginx/Passenger/Arvados-{api,workbench} issues related to bundler version.

arvados-formula:commit:f3e5a4b branch fix-rvm-deps-in-centos.

#22 Updated by Lucas Di Pentima about 2 months ago

This LGTM, thanks!

#23 Updated by Javier Bértoli about 2 months ago

  • Status changed from Feedback to Resolved

Merged

Also available in: Atom PDF