Arvados

Here's how RailsAPI is supposed to read cluster configuration:

1. nginx clears the environment except for TZ: "By default, nginx removes all environment variables inherited from its parent process except the TZ variable."
2. passenger_load_shell_envvars is on by default, which means "applications are loaded by running them with bash -l -c." Note that Passenger's architecture documentation says this is just for usability, not a technical requirement: "This causes bash to load its startup files, e.g. bashrc, profile, etc, after which it executes the SpawnPreparer with the given parameters. The reason why we do this is because a lot of users try to set environment variables in their bashrc, and they expect these environment variables to be picked up by applications spawned by Phusion Passenger. Unfortunately environment variables don’t work that way, but we support it anyway because it is good for usability."
3. source:services/api/config/arvados_config.rb runs Open3.capture3("arvados-server", "config-dump", "-config=-", "-skip-legacy", stdin_data: "Clusters: {xxxxx: {}}"). It finds arvados-server via the PATH variable set in step 2.

We recently observed a situation on a couple different clusters running Ubuntu 20.04 where unattended-upgrades automatically upgraded glibc, systemd, and nginx. After nginx got the reload/restart signal, something happened so that Passenger did not pick up environment variables from step 2 like it usually does. It tried to start RailsAPI with PATH=/var/www/arvados-api/shared/vendor_bundle/ruby/2.7.0/bin, which caused the Open3.capture3 expression to raise an exception, app initialization to fail, and the cluster to go down.

We still do not have a full explanation for why step 2 didn't work as it normally does in this case. However, we can reduce the number of moving pieces here by setting passenger_load_shell_envvars off and set the environment variables we need with passenger_env_var. Doing so will make RailsAPI deployment work more consistently by isolating them from configuration changes to the system shell, and it's a relatively straightforward change, so it
seems worth doing.