Arvados

For what it's worth arvados-cwl-runner seems to be the only Python blocker. I could install everything else in a Python 3.12 virtualenv:

$ ~/Scratch/arv312/bin/pip3.12 install --upgrade --upgrade-strategy=eager sdk/python/ services/fuse/ tools/crunchstat-summary/ services/dockercleaner/ tools/user-activity/
[…]
Successfully installed arvados-docker-cleaner-2.8.0.dev20240429172114 arvados-llfuse-1.5.2 arvados-python-client-2.8.0.dev20240429172114 arvados-user-activity-2.8.0.dev20240429172114 arvados_fuse-2.8.0.dev20240429172114 cachetools-5.3.3 certifi-2024.2.2 charset-normalizer-3.3.2 ciso8601-2.3.1 crunchstat_summary-2.8.0.dev20240429172114 docker-7.0.0 docutils-0.21.2 google-api-core-2.19.0 google-api-python-client-2.127.0 google-auth-2.29.0 google-auth-httplib2-0.2.0 googleapis-common-protos-1.63.0 httplib2-0.22.0 idna-3.7 lockfile-0.12.2 packaging-24.0 prometheus-client-0.20.0 proto-plus-1.23.0 protobuf-4.25.3 pyasn1-0.6.0 pyasn1-modules-0.4.0 pycurl-7.45.3 pyparsing-3.1.2 python-daemon-3.0.1 requests-2.31.0 rsa-4.9 setuptools-69.5.1 uritemplate-4.1.1 urllib3-2.2.1 websockets-12.0

And then I successfully ran and used the installed arv-mount.

I had a bunch of meetings yesterday so I started putting together the package build. I wasn't intending to actually get the job done, more just exploring to see what kinds of issues we might run into. That revealed #21742. There might be a couple other minor things but at least so far it didn't reveal any major issues like the OpenSSL 3.0 transition was.

WIP branch 21363-ubuntu2404-package-build @ 14c22e7c214a24db8dc3b66c809b86de1ac38416

SLURM may not support Ubuntu 24.04 yet. Here's their list of supported platforms. Maybe Ubuntu 23.04 is a typo? But it's not a given, that is a real release.

START: arvados-api-server test on arvados/package-test:ubuntu2404                                                                        
ERROR: arvados-api-server test on arvados/package-test:ubuntu2404 failed with exit status 100                                            

START: arvados-docker-cleaner test on arvados/package-test:ubuntu2404
ERROR: arvados-docker-cleaner test on arvados/package-test:ubuntu2404 failed with exit status 100

START: python3-arvados-cwl-runner test on arvados/package-test:ubuntu2404
ERROR: python3-arvados-cwl-runner test on arvados/package-test:ubuntu2404 failed with exit status 100

START: python3-cwltest test on arvados/package-test:ubuntu2404
Traceback (most recent call last): 
  File "/usr/bin/cwltest", line 5, in <module>
    from cwltest.main import main
  File "/usr/lib/python3-cwltest/lib/python3.12/site-packages/cwltest/main.py", line 15, in <module>
    from cwltest.argparser import arg_parser
  File "/usr/lib/python3-cwltest/lib/python3.12/site-packages/cwltest/argparser.py", line 5, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
ERROR: python3-cwltest test on arvados/package-test:ubuntu2404 failed with exit status 1

I believe 100 is dpkg's generic "postinst failed" exit code. The cwltest failure is because the specific version we package doesn't work on Python 3.12.

21363-ubuntu2404-package-build @ ac27c2f61542d0d5a4c2a212da3d9fd4b2819c8c

• All agreed upon points are implemented / addressed.
  • This branch is all that can be done in the arvados repo for now. After this, we set up the repository, then set up the Jenkins jobs to build and upload packages. And that work is a prerequisite for work on the Salt installer etc.
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • 🤷 We're at least on the same page that this ticket will not be closed by merging this branch, this is just the first step.
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • We have a chicken and egg problem that this needs to be merged before we can set up Jenkins jobs that would do testing. In this branch, I can successfully run the following from my arvados/build directory:

    WORKSPACE="$(dirname "$PWD")" ./run-build-packages-one-target.sh --target ubuntu2404 --arch amd64
    WORKSPACE="$(dirname "$PWD")" ./run-build-packages-one-target.sh --target ubuntu2404 --arch amd64 --test-packages

• Documentation has been updated.
  • N/A, that comes later
• Behaves appropriately at the intended scale (describe intended scale).
  • No change in scale
• Considered backwards and forwards compatibility issues between client and server.
  • Strictly a forwards-compatibility improvement
• Follows our coding standards and GUI style guidelines.
  • N/A (we don't have coding standards for shell or Dockerfiles)

21363-ubuntu2404-package-build LGTM, thanks.

I just created the apt repository following the wiki instructions. Just for posterity, as a follow-up to #21652, I just double-checked and confirmed that there are no extra spaces in the "arm64" repository that would break things:

% curl -fsSL --head 'https://apt.arvados.org/noble/dists/noble-dev/main/binary-arm64/'
HTTP/1.1 200 OK
Date: Fri, 07 Jun 2024 12:58:55 GMT
Server: Apache
Upgrade: h2
Connection: Upgrade
Content-Type: text/html;charset=UTF-8

Packages are build on Jenkins: build-packages-ubuntu2404: #1 - added this job to build-packages-multijob

Based on https://repo.saltproject.io/salt/py3/ubuntu/24.04/amd64/ it looks like we'll need to upgrade the Salt installer to Salt 3007 to support Ubuntu 24.04.

Adding Salt installer support is not straightforward because the Salt bootstrap script has not been updated to support noble yet.

Here's a pull request to do it. Note the commit —like so much else of Salt, it's just adding to a lookup table. Note also the code below the new lines, where if it doesn't recognize the version of Ubuntu you're on, it just assumes you're on trusty (‽‽‽). Note also that, despite the fact that this pull request is trivial and Ubuntu 24.04 has been out for at least one whole month already, is has not gotten any attention in ~two weeks.

Honestly I kind of want to ditch the bootstrap script. Well, that's not the whole truth. What I honestly want to do is ditch Salt, because I'm constantly running into issues like this that give me the strong impression that community support for the project is waning. But if we're not ready to make a switch, it doesn't seem too difficult to ditch the bootstrap script. The reason it's big and complicated is because it's trying to support an entire matrix of OSes (including non-Linuxes like BSDs) and install methods. If you don't care about that whole matrix, then the whole install process is 'add an apt repository with its associated key and install some packages', which is the most boring possible task for any Debian sysadmin and should be no big deal for us to write ourselves. We could support all the Debian-based distros our installer currently supports without any conditionals or branching, just judicious use of /etc/os-release.

But, that's still a semi-big task, and I'm not going to do it without someone else signing off. But if we don't, then Ubuntu 24.04 support is blocked on bootstrap support.

Brett Smith wrote in #note-19:

Honestly I kind of want to ditch the bootstrap script. Well, that's not the whole truth. What I honestly want to do is ditch Salt, because I'm constantly running into issues like this that give me the strong impression that community support for the project is waning. But if we're not ready to make a switch, it doesn't seem too difficult to ditch the bootstrap script. The reason it's big and complicated is because it's trying to support an entire matrix of OSes (including non-Linuxes like BSDs) and install methods. If you don't care about that whole matrix, then the whole install process is 'add an apt repository with its associated key and install some packages', which is the most boring possible task for any Debian sysadmin and should be no big deal for us to write ourselves. We could support all the Debian-based distros our installer currently supports without any conditionals or branching, just judicious use of /etc/os-release.

I like the idea of eliminating a dependency like the bootstrap installer, and only supporting the distros that Arvados currently supports.

That pull request is frustrating because it's failing checks due to entirely unrelated brokenness and then github is also blocking the merge because the commit "needs to be signed" which is super frustrating because that's something that github usually does for you except when it doesn't and then figuring out what it wants you to do feels impossible.

I am in agreement that getting rid of the dependency on the bootstrap script and doing the simple thing that works with the distros we actually support is the way to go here.

Regarding ditching salt - #21880

Bigger, more challenging issue: Phusion Passenger does not distribute noble packages yet. Upstream bug report but no real activity yet (but it's only five days old).

21363-ubuntu2404-salt-installer @ 6b126f512c1f5f33d22abe29369d1e8e12e58642

This branch upgrades us to Salt 3007, because that's the current major release and the only one that supports Ubuntu noble. It also stops using the bootstrap script on Debian-based systems per discussion above. (Doing that on Red Hat-based systems can be done as part of a general Red Hat support ticket, if we do that before we migrate away from Salt.)

Job 884 on https://ci.arvados.org/view/All/job/test-provision/ shows the branch working on all of our currently supported distros (which is a requirement).

test-provision-ubuntu2404: #4 shows that it successfully installs and starts Salt on Ubuntu 24.04. Provisioning fails because Passenger has not published noble packages yet. It's possible this code will "just work" after that becomes available.

Merging the branch would improve the installer and doesn't leave us any worse off functionality-wise than we already are, so we might as well do it now.

• All agreed upon points are implemented / addressed.
  • See above
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • This branch will not resolve the ticket but gets us closer
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • See above
• Documentation has been updated.
  • N/A - that can only happen after test-provision-ubuntu2404 is succeeding
• Behaves appropriately at the intended scale (describe intended scale).
  • No change in scale
• Considered backwards and forwards compatibility issues between client and server.
  • Tests show that backwards compatibility with older distros is fine while improving support for newer ones
• Follows our coding standards and GUI style guidelines.
  • N/A (no shell coding guidelines)

This LGTM. It seems that noble packages should be available really soon, they already added support for it 3 weeks ago https://github.com/phusion/passenger_apt_automation/commit/278eb35a3d525bae344974e549d22f5706c790f4

Brett Smith wrote in #note-23:

Bigger, more challenging issue: Phusion Passenger does not distribute noble packages yet. Upstream bug report but no real activity yet (but it's only five days old).

noble packages just went up

test-provision-ubuntu2404: #5 says it can't find package libcurl4. This is probably just distro name drift.

Brett Smith wrote in #note-31:

test-provision-ubuntu2404: #5 says it can't find package libcurl4. This is probably just distro name drift.

It is now called libcurl4t64 because of the 64-bit time transition. It's probably not the only one unfortunately. But it should just be a bunch of name transitions.

arvados-formula branch 21363-time64-pkgs @ 027e8ea06ddf40d947839278869b1a6edd827685

Gets test-provision-ubuntu2404 passing: test-provision-ubuntu2404: #6

• test-provision-debian11: #771
• test-provision-debian12: #90
• test-provision-ubuntu2204: #82
• (test-provision-ubuntu2004 failed because of a temporary lock issue, I hope this is proof enough.)

This change updates our arvados Salt formula to work on distros that have done Debian's 64-bit time transition. See commit message for implementation details.

• All agreed upon points are implemented / addressed.
  • It gets test-provision passing, which is an key milestone.
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • There is still documentation updates to do, but that'll happen in a separate branch in the main arvados repository.
    There are also a lot of opportunities to clean up the arvados formula by removing support for older distributions, etc. I didn't do that here because I suspect the formula's main branch is still used for things like 2.7 testing, so I didn't want to risk breaking that. I can make a ticket to do that cleanup after we release 3.0 and stop doing 2.7 releases.
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • See above
• Documentation has been updated.
  • Not yet, happens separately
• Behaves appropriately at the intended scale (describe intended scale).
  • No change in scale
• Considered backwards and forwards compatibility issues between client and server.
  • Still works with pre-transition distros, see commit message
• Follows our coding standards and GUI style guidelines.
  • N/A (no Salt style guide)

21363-time64-pkgs LGTM, thanks!

21363-ubuntu2404-docs @ 09cc25b7a11a9bfafc38dc1380115f34dfa9509a - developer-run-tests: #4384

I considered adding an upgrade note, but it doesn't look like we've ever done that before when we add support for a distribution. I guess it just gets highlighted in the release notes instead (which I can see the logic of).

• All agreed upon points are implemented / addressed.
  • Yes, this is the last part of the ticket
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • N/A
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • See above. This is a relatively tiny branch compared to others that have been tested and merged previously.
• Documentation has been updated.
  • Yes
• Behaves appropriately at the intended scale (describe intended scale).
  • No change in scale
• Considered backwards and forwards compatibility issues between client and server.
  • Retains support for all supported distributions, just adds and documents Ubuntu 24.04 support
• Follows our coding standards and GUI style guidelines.
  • Yes

21363-ubuntu2404-docs LGTM too, thanks!