Arvados

It's apparently #20862. test-provision succeeded on the 2.7.0 release, then fails on 80794f079f005fd3d927b9d330a46bcc96a1a132, which is the merge immediately after. And this actually makes more sense with the error: I'm guessing the gem shuffling causes a deployment problem for RailsAPI, and this means the up test never passes.

Brett Smith wrote in #note-1:

Why is debian11 passing when the others aren't? This isn't super surprising since it's closest to the platform where we do development. Is there some key version difference that debian11 is quietly relying on? Or did something get updated for debian11 that needs to be updated for other targets as well?

Judging by the logs, the failing platforms install Ruby from RVM, while debian11 uses packaged Ruby. Seems likely to be relevant. This was incorrect, ubuntu2004 is failing but also uses packaged Ruby.

debian11 installs arvados-login-sync where the others do not: This was also a red herring, this is only because debian11 doesn't fail before this point.

Comparing the states that are run via rg --color=never -o '\b\S+ state [^]]+\]' on the Jenkins logs, the only differences between ubuntu2004 and debian11 are what you would expect: names, PostgreSQL version number, etc. There's no difference at the deployment level that would explain why debian11 succeeds and ubuntu2004 fails. It's something lower-level.

I was able to install the arvados-api-server development package in my own Ubuntu 20.04 container. Which is what you'd expect, Jenkins can do that too, but I was keeping an eye out for any error messages that Jenkins might be ignoring and didn't see any.

Trying to replicate this further requires actually running the services, which is more challenging in a container. So either I'd need a way to get into the running Jenkins worker and debug, or else recreate the environment in a VM myself. That's expensive enough that I'm not going to do it without someone actually asking me.

So, I'm blocked on this unless someone wants to give me an idea to pursue; SSH access to Jenkins workers; or permission to go deep on this.

The issue I'm observing is the following: test-provision-debian11: #539

...
15:32:03 Running bundle config set --local path /var/www/arvados-api/shared/vendor_bundle... done.
15:32:03 Running bundle install... done.
15:32:03 Ensuring directory and file permissions ...... done.
15:32:03 Setting up database...Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache does not exist
15:32:03 rake aborted!
15:32:03 Don't know how to build task 'db:structure:load' (See the list of available tasks with `rake --tasks`)
15:32:03 Did you mean?  db:structure:dump
15:32:03 
15:32:03 (See full trace by running task with --trace)
15:32:03  failed.
15:32:03 dpkg: error processing package arvados-api-server (--configure):
15:32:03  installed arvados-api-server package post-installation script subprocess returned error exit status 1
...

Since Rails 6.1 the db:structure:{load, dump} rake task have been deprecated: https://github.com/rails/rails/pull/39470 -- It seems we should be using db:schema:{load, dump} now, with config.active_record.schema_format set to :sql (as we already do)

Updates at d6e41e6 - branch 21031-test-provision-fix
Package test run: developer-build-packages-debian11: #15

• Replaces db:structure:load with db:schema:load in RailsAPI package build script.
• Updates postinst.sh script for Rails apps packages to reflect the above.

Given that the above test passed, I'll be merging this to main so that we can build packages & test them on the test-provision jobs.

Build packages pipeline: build-packages-multijob: #3811

Test provision job still failing with the same message: test-provision-debian11: #541