Project

General

Profile

Actions

Feature #16048

closed

Automatically reload/restart services on configuration change

Added by Peter Amstutz over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Subtasks 1 (1 open0 closed)

Task #16322: Review 16048-reload-configIn ProgressTom Clegg04/14/2020Actions

Related issues

Related to Arvados Epics - Idea #15941: arvados-bootNewActions
Related to Arvados - Bug #8374: [KEEPSTORE] detection of new drives Closed02/04/2016Actions
Related to Arvados - Idea #16069: [boot] start a dev clusterNewActions
Actions #1

Updated by Peter Amstutz over 4 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz over 4 years ago

  • Status changed from In Progress to New
  • Tracker changed from Bug to Feature
Actions #3

Updated by Peter Amstutz over 4 years ago

Actions #4

Updated by Peter Amstutz over 4 years ago

  • Related to Bug #8374: [KEEPSTORE] detection of new drives added
Actions #5

Updated by Tom Clegg about 4 years ago

  • Related to Idea #16069: [boot] start a dev cluster added
Actions #6

Updated by Peter Amstutz about 4 years ago

  • Target version set to 2020-04-22
Actions #7

Updated by Tom Clegg about 4 years ago

  • Assigned To set to Tom Clegg
Actions #8

Updated by Tom Clegg about 4 years ago

  • Status changed from New to In Progress
Actions #9

Updated by Tom Clegg about 4 years ago

16048-reload-config @ 45c57dc86010f8fd9ae862d91a710e0752ceda4a -- developer-run-tests: #1807

If AutoReloadConfig is true in the config (see doc/examples/config/zzzzz.yml), arvados-server boot notices when the config file changes. If it changes in a meaningful way (as opposed to just changing YAML formatting, adding/removing a value that's already the default, etc) then it shuts down and restarts all services. This takes about 15s on my machine.

With a minimal config like zzzzz.yml that relies on auto-assigned port numbers, restarting means assigning port numbers. This means you should either:
  • configure (at least) the controller URL explicitly instead of relying on auto-assigned ports, so ports stay stable across restarts, or
  • expect to reconfigure/restart any clients after updating config, so they use the new controller, keepproxy, keepstore, etc. endpoints, or
  • not use AutoReloadConfig.
Actions #10

Updated by Lucas Di Pentima about 4 years ago

Sorry for the delay, yesterday I wasn’t being able to run arvados-server boot on my usual dev VM because of some bundle version issue. Had to make a new one for it to work.

What would be the use case for this feature?
  • I’m thinking that if the idea is to be able to do integration testing with different cluster configurations, maybe we should preserve the postgresql database? I’ve done some manual testing and it seems that it’s more of a complete cluster reset and not just a restart/reload feature.
  • If the idea is to just have an automatic cluster reset feature, it LGTM.
Actions #11

Updated by Tom Clegg about 4 years ago

Agree, when using -own-temporary-database the side effect of resetting the database is probably a drawback more often than a feature. And the big shutdown/startup is slow. For now this just gives us more convenience than restarting everything manually.

In future we'll surely want to make it more efficient -- and perhaps avoid downtime entirely when port #s aren't changing, making it usable in production -- by reloading/restarting only the services that are potentially affected by the change.

Actions #12

Updated by Anonymous about 4 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #13

Updated by Peter Amstutz over 3 years ago

  • Release set to 25
Actions

Also available in: Atom PDF