Feature #16048

Automatically reload/restart services on configuration change

Added by Peter Amstutz 10 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
04/14/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Subtasks

Task #16322: Review 16048-reload-configIn ProgressTom Clegg


Related issues

Related to Arvados Epics - Story #15941: arvados-bootIn Progress03/15/202107/01/2021

Related to Arvados - Bug #8374: [KEEPSTORE] detection of new drives Closed02/04/2016

Related to Arvados - Story #16069: [boot] start a dev clusterNew

Associated revisions

Revision ea6f25f0
Added by Tom Clegg 7 months ago

Merge branch '16048-reload-config'

closes #16048

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz 10 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 10 months ago

  • Status changed from In Progress to New
  • Tracker changed from Bug to Feature

#3 Updated by Peter Amstutz 10 months ago

#4 Updated by Peter Amstutz 10 months ago

  • Related to Bug #8374: [KEEPSTORE] detection of new drives added

#5 Updated by Tom Clegg 10 months ago

#6 Updated by Peter Amstutz 8 months ago

  • Target version set to 2020-04-22

#7 Updated by Tom Clegg 8 months ago

  • Assigned To set to Tom Clegg

#8 Updated by Tom Clegg 8 months ago

  • Status changed from New to In Progress

#9 Updated by Tom Clegg 8 months ago

16048-reload-config @ 45c57dc86010f8fd9ae862d91a710e0752ceda4a -- https://ci.arvados.org/view/Developer/job/developer-run-tests/1807/

If AutoReloadConfig is true in the config (see doc/examples/config/zzzzz.yml), arvados-server boot notices when the config file changes. If it changes in a meaningful way (as opposed to just changing YAML formatting, adding/removing a value that's already the default, etc) then it shuts down and restarts all services. This takes about 15s on my machine.

With a minimal config like zzzzz.yml that relies on auto-assigned port numbers, restarting means assigning port numbers. This means you should either:
  • configure (at least) the controller URL explicitly instead of relying on auto-assigned ports, so ports stay stable across restarts, or
  • expect to reconfigure/restart any clients after updating config, so they use the new controller, keepproxy, keepstore, etc. endpoints, or
  • not use AutoReloadConfig.

#10 Updated by Lucas Di Pentima 8 months ago

Sorry for the delay, yesterday I wasn’t being able to run arvados-server boot on my usual dev VM because of some bundle version issue. Had to make a new one for it to work.

What would be the use case for this feature?
  • I’m thinking that if the idea is to be able to do integration testing with different cluster configurations, maybe we should preserve the postgresql database? I’ve done some manual testing and it seems that it’s more of a complete cluster reset and not just a restart/reload feature.
  • If the idea is to just have an automatic cluster reset feature, it LGTM.

#11 Updated by Tom Clegg 7 months ago

Agree, when using -own-temporary-database the side effect of resetting the database is probably a drawback more often than a feature. And the big shutdown/startup is slow. For now this just gives us more convenience than restarting everything manually.

In future we'll surely want to make it more efficient -- and perhaps avoid downtime entirely when port #s aren't changing, making it usable in production -- by reloading/restarting only the services that are potentially affected by the change.

#12 Updated by Anonymous 7 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#13 Updated by Peter Amstutz about 2 months ago

  • Release set to 25

Also available in: Atom PDF