Support #21855
closed
Migrate "CentOS" path prefix on rpm.arvados.org to something else
Description
Because CentOS has mutated from being a stable RHEL-compatible distro to a rolling distro that is no longer a suitable development target, we will stop supporting it in Arvados 3.0. That makes it weird to have the repository path https://rpm.arvados.org/CentOS
This should be renamed https://rpm.arvados.org/RHEL and the relevant package build scripts and documentation updated to use that. (I spent an inordinate amount of time trying to figure out if there was a naming scheme we should be following but came up empty, so RHEL is the most straightforward name to use here.)
For backwards compatibility, https://rpm.arvados.org/CentOS should continue to exist and serve already-released packages. For simplicity we could link CentOS and RHEL to the same directory, but that will eventually lead to weird things like packages showing up in a "/CentOS/9" directory when CentOS 9 doesn't exist.
Also, should we be distributing against point releases? E.g. perhaps our packages should be "/RHEL/8.6" because we are using python 3.9 packages that were not in the 8.0 release but became available in a point release later. (I don't know what the best practice is here, it seems that the client is configured with just the /RHEL/ path and selects the appropriate version on its own).
For example, https://download.rockylinux.org/pub/rocky/ lists individual point releases. It looks like the point releases are point-in-time, and the major release simply tracks the most recent point version. So for our own packaging, where our packages are going to be the same for all supported point releases within a major version, there's probably no reason to break it out by separate RHEL point releases.
Updated by Peter Amstutz about 1 month ago
- Target version changed from 439 to Development 2024-06-19 sprint
Updated by Peter Amstutz 14 days ago
- Target version changed from Development 2024-06-19 sprint to Development 2024-07-03 sprint
Updated by Peter Amstutz 13 days ago
- Assigned To changed from Lucas Di Pentima to Brett Smith
Updated by Brett Smith 12 days ago
Peter Amstutz wrote:
Also, should we be distributing against point releases?
We should maybe make some symlinks but I don't see a need for more than that. Quoting the yum documentation:
$relasever: You can use this variable to reference the release version of Red Hat Enterprise Linux. Yum obtains the value of $releasever from the distroverpkg=value line in the /etc/yum.conf configuration file. If there is no such line in /etc/yum.conf, then yum infers the correct value by deriving the version number from the redhat-release-server package. The value of $releasever typically consists of the major release number and the variant of Red Hat Enterprise Linux, for example 6Client, or 6Server.
So when you install one of these distributions, out of the box $releasever=8, and on the server side the 8 repository always seemed to be mapped to the latest point release. If ABI compatibility is not enough for you, and you need 100% bug-for-bug compatibility with a specific point release, you configure $releasever=8.N to stay there.
Since our packages are good for all releases from 8.6 on, we could symlink all those releases to our current 8 directory, and that would make life a little easier for any administrators who have pinned $releasever. At the same time, they can also just change their baseurl for Arvados to hardcode 8, and that works just as well. So I just don't think this is a big deal either way.
Updated by Peter Amstutz 12 days ago
Brett Smith wrote in #note-8:
Since our packages are good for all releases from 8.6 on, we could symlink all those releases to our current
8directory, and that would make life a little easier for any administrators who have pinned$releasever. At the same time, they can also just change theirbaseurlfor Arvados to hardcode8, and that works just as well. So I just don't think this is a big deal either way.
Ok, so the least-effort thing to do on our part would be to just set the directory to 8 and leave it at that.
Updated by Brett Smith 12 days ago
Peter Amstutz wrote in #note-9:
Ok, so the least-effort thing to do on our part would be to just set the directory to
8and leave it at that.
Yes. This does mean that our documentation for adding the repository do not work on systems where the administrator has pinned releasever. But I don't know how common that is. So I would say, let's wait for someone to actually complain about it. If they do, setting up the symlink tree is probably the next easiest thing to do. (Versus having more documentation branches or whatever.)
Updated by Brett Smith 12 days ago
- Status changed from New to In Progress
Implementation plan:
- In Jenkins, acquire the rocky8 upload lock
- On the rpm server:
This is the simplest way to maintain URL backwards compatibility while avoiding the "CentOS 9" problem.mv CentOS RHEL mkdir CentOS env -C CentOS ln -st . ../RHEL/*/CentOS/8will continue to get new packages as we build them, but/CentOSwill not get any new releases or other subdirectories. - In Jenkins, release the rocky8 upload lock
s/CentOS/RHEL/ginrun_upload_packages.py. This is a very literal search-and-replace.- Prepare a branch with the same for the repository setup documentation.
After the first part is done and the lock is released, it's okay if time passes between the following steps, the urgency goes away.
Updated by Lucas Di Pentima 12 days ago
Brett Smith wrote in #note-11:
Implementation plan:
- In Jenkins, acquire the rocky8 upload lock
- On the rpm server: [...]This is the simplest way to maintain URL backwards compatibility while avoiding the "CentOS 9" problem.
/CentOS/8will continue to get new packages as we build them, but/CentOSwill not get any new releases or other subdirectories.- In Jenkins, release the rocky8 upload lock
s/CentOS/RHEL/ginrun_upload_packages.py. This is a very literal search-and-replace.- Prepare a branch with the same for the repository setup documentation.
After the first part is done and the lock is released, it's okay if time passes between the following steps, the urgency goes away.
This plan looks good to me.
Updated by Brett Smith 12 days ago
21855-rpm-url-update @ 3f229334baa4580bbca5a92271727e18fbc86ff8 - developer-run-tests: #4304 - developer-run-tests-remainder rerun developer-run-tests-remainder: #4520
Please check anything you want, but IMO a nice thing to do in this review would just be to double-check that I didn't miss any references to the old URL. It's not very substantial otherwise.
- All agreed upon points are implemented / addressed.
- Yes
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
- Code is tested and passing, both automated and manual, what manual testing was done is described
- Confirmed the updated documentation lets you install the
arvados-clientpackage from a fresh Rocky 8.8 container.
- Confirmed the updated documentation lets you install the
- Documentation has been updated.
- Yes
- Behaves appropriately at the intended scale (describe intended scale).
- No change in scale
- Considered backwards and forwards compatibility issues between client and server.
- Maintained backwards compatibility with the symlinks. This includes compatibility with our internal tooling as well as clients.
- Follows our coding standards and GUI style guidelines.
- N/A (no documentation style guide)
Updated by Lucas Di Pentima 12 days ago
Changes LGTM, I've just found one missing thing: The arvados-formula also points to the old RPM URLs.
Updated by Brett Smith 12 days ago
Lucas Di Pentima wrote in #note-14:
Changes LGTM, I've just found one missing thing: The arvados-formula also points to the old RPM URLs.
Thanks, done over there in db50f104ac3af5cd3134f82e99384fc7b084be53.
In the spirit of modernizing URLs, I also removed /RHEL/RPM-GPG-KEY-curoverse, since Curoverse isn't a thing anymore and nobody has had a chance to use that URL. The symlink does still exist under /CentOS.