Arvados

Consider this function in source:build/rails-package-scripts/postinst.sh which gets called in the main body:

prepare_database() {
  # Prevent PostgreSQL from trying to page output
  unset PAGER
  DB_MIGRATE_STATUS=`"$BUNDLE" exec bin/rake db:migrate:status 2>&1 || true`
  if echo "$DB_MIGRATE_STATUS" | grep -qF 'Schema migrations table does not exist yet.'; then
      # The database exists, but the migrations table doesn't.
      run_and_report "Setting up database" "$BUNDLE" exec bin/rake db:schema:load db:seed
  elif echo "$DB_MIGRATE_STATUS" | grep -q '^database: '; then
      run_and_report "Running db:migrate" "$BUNDLE" exec bin/rake db:migrate
  elif echo "$DB_MIGRATE_STATUS" | grep -q 'database .* does not exist'; then
      run_and_report "Running db:setup" "$BUNDLE" exec bin/rake db:setup
  else
      # We don't have enough configuration to even check the database.
      return 1
  fi
}

Imagine you install arvados-api-server on a system where all the prerequisites are met (it has a good cluster configuration with an existing database and valid credentials, etc.) EXCEPT there is no smtpd listening on port 25. The first time you try to install the package, prepare_database will go down the first branch, and the schema will get loaded, but then db:seed will fail because of #22703. (There might be other ways to get db:seed to fail but this is the one in front of me now.)

The next time you try to install arvados-api-server, it will go down the second branch, try to run db:migrate, that'll be a noop, and it will say that everything is good. However, not everything is good: you still have not seeded the database, so you don't have records like a root user, and your RailsAPI server is functionally useless. If you're being good and following the install docs, you won't notice this until you run arvados-client diagnostics, it fails on the "get current user" step, and you go to track down why.

This function should do more to check whether the database is seeded and retry the rake task if not. I realize this is a much more delicate operation than it sounds and probably would benefit from some team discussion about the "right" way to do this.