Arvados

I think the binary packages should only install init support that works with the system that the package supports, but that there should be more options included in (or built by) the source distribution, but they probably shouldn't be installed unless the user asks for it explicitly. It seems like it could be very confusing for a system to have multiple different sets of init scripts installed, only one of which (presumably) is actually in use.

However, I also think it makes sense to think of this more generically as support for "process control/supervision" systems rather than being limited to those systems that can be used as an 'init'. One case is as above for packages on supported systems, but another case is for users installing from source or who for whatever reason don't want to use the system init system to manage the services. One good feature of runit is that, although it can be used as an init, it can also be used on top of another init. However, a drawback is that it still assumes it will be run as root (otherwise chpst to another user wouldn't work).

Something that supervisord does that the inits don't (generally?) do is to provide a daemon that can be queried/controlled remotely. We currently use it along with the nodervisor dashboard (https://github.com/TAKEALOT/nodervisor) to monitor services on a number of machines from one interface.

Our main (non-Arvados) cluster is also running the Apache mesos framework, on top of which we use marathon to monitor long running processes. Marathon could be thought of as a cluster-level init. Support for mesos/marathon is probably outside the scope of this planning story, but it is relevant to consider how supporting something like marathon in the future would fit in with these stories. The idea of a system like marathon is that you ask for a service to be run somewhere on the mesos cluster, such that it can make sure it keeps running even if the host it is originally started on disappeared (obviously this also requires some form of dynamic networking configuration in order to route the services to appropriate locations). This has the added benefit that you can also configure marathon to scale up/down the number of backends for each service for which that is appropriate.

On our (non-Arvados) clusters we currently run supervisord to manage mesos/marathon/chronos and then run other services (such as web backends) on top of marathon. We do not run any of those services as root, and it is nice that supervisord does not require being run as root either. There would be some advantage to us to be able to use supervisord to manage Arvados services as well, because we could manage them from the same dashboard. I'm also interested in eventually removing any artificial dependencies Arvados components have that require them to be run as root (although that support itself is also probably outside the scope of this plan). I would think it should be possible to install Arvados from source in a non-standard location and to run most parts of the system as a regular user, and it would be nice to support some process control systems that don't need to be run as root so that there isn't an artificial dependency on being root just in order to manage process control. If there is a real dependency on being root (or some other user such as git) then that can be made explicit.

I liked Tom's suggestion (in https://arvados.org/issues/6663#note-31) that the way that the configuration/scripts are written would ideally include components that are shared between them and which could also be usefully called from the command line. Ideally all daemon stories would include a generic shell script that just starts the daemon in the foreground (this could either be built alongside other init scripts or used as a component of them). If something like chpst to a particular user would normally be required, this script could just check if it is being run as the correct user and complain if not. Likewise for environment variables that are required to be set for it to work (and for which there is no sane default).

Nico notes that pleaserun is good for generating skeleton scripts, but it's not sharp enough to include the output verbatim in packages.

We had some discussion about this in a planning meeting.

There's broad support for having packages install a service definition for the distro's native init. It can also include definitions for other inits in a place where they won't be used but are easy to find and copy; e.g., /usr/share/doc.

Of course, we can do this incrementally. This means that a version 1 might just support sysvinit and systemd (assuming I'm right that Ubuntu releases that use Upstart has sysvinit compatibility that we can hook into). We'll probably include runit scripts for a lot of our packages, since we already have a lot of them written. Others can be added as they're developed.

Upstart provides sysvinit compatibility by including a job /etc/init/rc.conf that runs /etc/init.d/rc in the usual way. I think this will let us de-prioritize develpoment of Upstart definitions. Yes, we'll be able to use more features when we write those, like more active process monitoring; but we're not going to have those on pure sysvinit systems anyway, so I don't see much harm in living without it on more modern distros for now. And I'm not eager to invest much effort in Upstart given that it's on the way out in favor of systemd. Not saying we can't or won't do it; just that it seems lower priority to me.

http://upstart.ubuntu.com/cookbook/#the-rc-job is the cite for the above.