Story #17246

Single node salt install improvements

Added by Peter Amstutz 3 months ago. Updated about 20 hours ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Start date:
03/31/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

From discussion 7 January 2021

  • Instead of arva2, choose a random cluster id or get one from the user
  • Instead of using arva2.arv.local the provision script should use the host's FQDN or get the hostname from the user
  • To avoid the need for virtual hosts, put workbench (1 or 2) on the https port and put other services on alternate ports
  • Confirm that running the provision script on an basic EC2 instance results in an arvados install with a workbench that can be logged into & can run stuff

Subtasks

Task #17509: review 17246-salt-install-improvementsResolvedLucas Di Pentima


Related issues

Blocked by Arvados - Bug #17398: [crunch-dispatch-local] [crunch-run] error starting gateway server: missing port in addressResolved02/18/2021

Blocks Arvados - Support #17320: Explain what additonal configuration is needed for provision.sh to go to productionIn Progress

Associated revisions

Revision 677877d5 (diff)
Added by Javier Bértoli 3 months ago

feat(provision): allow to install individual roles

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 102277e6 (diff)
Added by Javier Bértoli 3 months ago

feat(provision): manage setup params from local file

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision d8241cba (diff)
Added by Javier Bértoli 3 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 90980819 (diff)
Added by Javier Bértoli 3 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 2a662b83 (diff)
Added by Javier Bértoli 3 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 3aa47e60 (diff)
Added by Javier Bértoli 3 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 45b514e1 (diff)
Added by Javier Bértoli 3 months ago

feat(provision): check the cluster name provided is exactly 5 chars long

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision b57db2bc (diff)
Added by Javier Bértoli 3 months ago

feat(provision): allow to install individual roles

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 2b935043 (diff)
Added by Javier Bértoli 3 months ago

feat(provision): manage setup params from local file

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 33160dea (diff)
Added by Javier Bértoli 3 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 6c058acc (diff)
Added by Javier Bértoli 3 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision c590159b (diff)
Added by Javier Bértoli 3 months ago

feat(provision): check the cluster name provided is exactly 5 chars long

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision ffd332a1 (diff)
Added by Javier Bértoli 2 months ago

feat(provision): refactor to manage different infrastructure configurations

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision d6ec1985 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): refactor single host architectures

Allow to use a single-host/single-hostname or single-host/multiple-hostnames setup
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision a396085e (diff)
Added by Javier Bértoli 2 months ago

fix(provision): add missing postgres dependency for arvados-api-server

refs #17246 & #17352
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision d0218fb8 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): check salt-call is installed

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 8579c7a8 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): remove deprecated parameter

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision f44a0323 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): add only specific pillars for each role

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 9d52d145 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): make the salt-call binary check compatible with sh

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 86d1e1b0 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): add missing roles sections

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision dda76502 (diff)
Added by Javier Bértoli 2 months ago

refactor(provision): naming consistency

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 9074ec89 (diff)
Added by Javier Bértoli 2 months ago

feat(provision): allow to install individual roles

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision c0bfa951 (diff)
Added by Javier Bértoli 2 months ago

feat(provision): manage setup params from local file

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision a8ccabb1 (diff)
Added by Javier Bértoli 2 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision f023eb51 (diff)
Added by Javier Bértoli 2 months ago

feat(provision): refactor to add other setup examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 55d74d65 (diff)
Added by Javier Bértoli 2 months ago

feat(provision): check the cluster name provided is exactly 5 chars long

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 8195246d (diff)
Added by Javier Bértoli 2 months ago

feat(provision): refactor to manage different infrastructure configurations

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision d10dcb43 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): refactor single host architectures

Allow to use a single-host/single-hostname or single-host/multiple-hostnames setup
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision efd412f4 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): add missing postgres dependency for arvados-api-server

refs #17246 & #17352
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision de02a06e (diff)
Added by Javier Bértoli 2 months ago

fix(provision): check salt-call is installed

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 9aa498ee (diff)
Added by Javier Bértoli 2 months ago

fix(provision): remove deprecated parameter

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 67151930 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): add only specific pillars for each role

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 7a35170a (diff)
Added by Javier Bértoli 2 months ago

fix(provision): make the salt-call binary check compatible with sh

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 963b5ff6 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): add missing roles sections

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 60a59df7 (diff)
Added by Javier Bértoli 2 months ago

refactor(provision): naming consistency

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 1b9d2df0 (diff)
Added by Javier Bértoli 2 months ago

fix(provision): force user to properly set cluster & domain parameters

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 1a0cdc10 (diff)
Added by Javier Bértoli about 1 month ago

fix(provision): force user to properly set cluster & domain parameters

Also improved single host installation options (single, multiple hostnames)

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 332a26eb (diff)
Added by Javier Bértoli 30 days ago

fix(provision): improve single host installation and documentation

Also improved single host installation options (single, multiple hostnames)

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 9191a9a5 (diff)
Added by Javier Bértoli 30 days ago

fix(provision): add multi hosts installation examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision e7d59680 (diff)
Added by Javier Bértoli 28 days ago

docs(provision): add salt usage with roles in multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 609d60c0 (diff)
Added by Javier Bértoli 22 days ago

docs(provision): add salt usage with roles in multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 6587faf6 (diff)
Added by Javier Bértoli 21 days ago

docs(provision): add salt usage with roles in multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision f82bd0f6 (diff)
Added by Javier Bértoli 8 days ago

docs(provision): add salt usage with roles in multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision dde8a2e6 (diff)
Added by Javier Bértoli 8 days ago

docs(provision): add salt usage with roles in multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 675a683d (diff)
Added by Javier Bértoli 8 days ago

fix(provision): use nginx-formula PR branch until changes are merged

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision f5ed56bf (diff)
Added by Javier Bértoli 8 days ago

feat(provision): add local.params example for multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 9f1a7c27 (diff)
Added by Javier Bértoli 8 days ago

fix(provision): temp fix for nginx formula issue with snippets

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 77da50d7 (diff)
Added by Javier Bértoli 5 days ago

fix(provision): use uniform database parameters across examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 5fe88fae (diff)
Added by Javier Bértoli 5 days ago

chore(provision): set postgresql cidr in single host examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision c281c353 (diff)
Added by Javier Bértoli 5 days ago

fix(provision): run hosts/snakeoil states before nginx so it does not fail on missing deps

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 4c4af3ff (diff)
Added by Javier Bértoli 5 days ago

refactor(provision): use uniform database parameters across examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision cf249360 (diff)
Added by Javier Bértoli 5 days ago

chore(provision): ignore some files in git

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision f47b71a4 (diff)
Added by Javier Bértoli 5 days ago

refactor(provision): delete parameters moved to config files

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 13e45cca (diff)
Added by Javier Bértoli 5 days ago

feat(provision): add some checks before run

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 89507291 (diff)
Added by Javier Bértoli 5 days ago

docs(provision): explain options in config examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision cd391a3b (diff)
Added by Javier Bértoli 5 days ago

chore(provision): enable both examples in vagrant

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 317c18d0 (diff)
Added by Javier Bértoli 5 days ago

refactor(provision): add host entries in single host multiple hostnames

We could use the example states in the formula, but added them here
for the sake of completeness

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 8a5388bb (diff)
Added by Javier Bértoli 5 days ago

docs(provision): improve the documentation in the examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 1e67d66f (diff)
Added by Javier Bértoli 5 days ago

refactor(provision): add snakeoil certs in single host multiple hostnames

We could use the example states in the formula, but added them here
for the sake of completeness

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision ea846236 (diff)
Added by Javier Bértoli 2 days ago

docs(provision): improve the documentation in the examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 86c7588a (diff)
Added by Javier Bértoli 2 days ago

fix(provision): default to using the master branch of arvados-formula

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 329b8285 (diff)
Added by Javier Bértoli 2 days ago

feat(provision): pin arvados version to latest stable

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision e3e255e8 (diff)
Added by Javier Bértoli about 20 hours ago

feat(provision): pin arvados version to latest stable

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 42fff421
Added by Javier Bértoli about 20 hours ago

Merge branch '17246-salt-install-improvements'

closes #17246

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

History

#1 Updated by Peter Amstutz 3 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#4 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2021-01-20 Sprint to 2021-02-03 Sprint

#6 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#7 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2021-02-03 Sprint to 2021-02-17 sprint

#8 Updated by Javier Bértoli 2 months ago

  • % Done changed from 0 to 80

Uploaded code, commit dda765028@arvados, branch 17246-salt-install-improvements.

The script expects the user to create a local.params file where these parameters can be set.

Added support to specify individual 'arvados roles' to deploy to a node, allowing to use this script to do a multi-hosts deploy.

#9 Updated by Javier Bértoli 2 months ago

  • Blocked by Bug #17398: [crunch-dispatch-local] [crunch-run] error starting gateway server: missing port in address added

#10 Updated by Javier Bértoli 2 months ago

  • Blocks Support #17320: Explain what additonal configuration is needed for provision.sh to go to production added

#11 Updated by Peter Amstutz 2 months ago

  • Target version changed from 2021-02-17 sprint to 2021-03-03 sprint

#12 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2021-03-03 sprint to 2021-03-17 sprint

#13 Updated by Javier Bértoli about 1 month ago

  • Target version changed from 2021-03-17 sprint to 2021-03-31 sprint

#14 Updated by Javier Bértoli 28 days ago

Updated code and documentation for multi-host installation (commit e7d596806@arvados, branch 17246-salt-install-improvements)

#15 Updated by Javier Bértoli 21 days ago

  • % Done changed from 80 to 100
  • Target version changed from 2021-03-31 sprint to 2021-04-14 sprint
  • Status changed from In Progress to Resolved

Single node install works correctly, expecting review.

#16 Updated by Javier Bértoli 21 days ago

  • Status changed from Resolved to In Progress

#17 Updated by Lucas Di Pentima 20 days ago

Reviewing 6587faf - branch 17246-salt-install-improvements

Tried to follow the instructions for a single host / single hostname install by creating a new debian 10 VM.

  • In file doc/install/salt.html.textile.liquid there's a mention of arvados-formula URL from the arvados repo, but in the README.md file still points to the saltstack-formulas repo.
  • There's a typo at doc/install/salt-single-host.html.textile.liquid line 40: 'asigning'
  • Instructions say to run the provision.sh script without any param, but I found that I needed to add -c local.params for it to run. At least I think the script should error out when the CONFIG_FILE var is empty string instead of just trying to run source ''
  • Eventually I got the following error:
...
[INFO    ] Fetching file from saltenv 'base', ** done ** 'arvados/dispatcher/service/file.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'arvados/dispatcher/service/running.sls'
local:
    Data failed to compile:
----------
    No matching sls found for 'nginx.passenger' in env 'base'
+ '[' xyes = xyes ']'
+ echo 'Removing .psql file'
Removing .psql file
+ rm /root/.psqlrc
+ '[' x = xyes ']'
+ echo 'Copying the Arvados CA certificate to the installer dir, so you can import it'
Copying the Arvados CA certificate to the installer dir, so you can import it
+ '[' x = xyes ']'
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem /home/lucas/arvados/tools/salt-install/lucas.arvados.local-arvados-snakeoil-ca.pem
cp: cannot stat '/etc/ssl/certs/arvados-snakeoil-ca.pem': No such file or directory
+ '[' x = xyes ']'

#18 Updated by Javier Bértoli 8 days ago

commit dde8a2e6a, branch 17246-salt-install-improvements

#19 Updated by Peter Amstutz 7 days ago

  • Target version changed from 2021-04-14 sprint to 2021-04-28 bughunt sprint

#20 Updated by Lucas Di Pentima about 20 hours ago

17246-salt-install-improvements LGTM, thanks!

#21 Updated by Javier Bértoli about 20 hours ago

  • Status changed from In Progress to Resolved

Merged

Also available in: Atom PDF