debian package repos: switch from freight to something faster
Freight works great, except that it is extremely slow to add packages to an existing repository, the speed of addition is O(n) for n the number of packages in the repository.
From preliminary testing, aptly seems like a good replacement candidate (it doesn't have this particular problem).
#3 Updated by Ward Vandewege 5 months ago
Aptly is much more strict about duplicate packages. Namely, from https://wiki.debian.org/DebianRepository/Format?action=show&redirect=RepositoryFormat#Duplicate_Packages:
A repository must not include different packages (different content) with the same package name, version, and architecture. When a repository is meant to be used as a supplement to another repository this should hold for the joint main+supplement repository as well.
#4 Updated by Ward Vandewege 5 months ago
Almost ready for the migration.
As part of this, I am changing the Origin and Label fields of our apt repos. They currently say 'Freight', and that will become 'Arvados'.
Updating the buster-dev repo now takes on the order of 30 seconds, as opposed to ~1000 seconds with freight.
The functionality can be tested on the (temporary) apt-dev host. E.g. put this in /etc/sources.list.d/arvados.list
deb http://apt-dev.arvados.org/buster buster main
(or use stretch, jessie, wheezy, bionic, xenial, trusty or precise)
Because we have to change the URLs in people's sources.list files, they will get a 404 on the old URLs after we flip the switch. It was not possible to rewrite the URLs automatically and make apt happy, because the `Packages` file relies on relative URLs.
Unfortunately, it seems to be impossible to get some sort of warning through to the command line when `apt` gets a 404. Apt even rewrites most error codes... sigh. So, I did the next best thing, and put a file here: http://apt-dev.arvados.org/REPOSITORIES_HAVE_BEEN_REORGANIZED.txt (this file will appear at http://apt.arvados.org/REPOSITORIES_HAVE_BEEN_REORGANIZED.txt) when the switch is flipped.
There is one more loose end - the set of scripts that move files from the -dev to -attic repos will need to be updated. They are not checked into our code tree. I will correct that, fix and simplify, as part of ticket #17219.
This is the forum post that I would like to put under 'Announcements':
Debian/Ubuntu apt repository changes
The Arvados project has reorganized its package repositories for Debian and Ubuntu.
If you installed Arvados before 2020-12-22 on a Debian/Ubuntu system using the Arvados package repositories, you need to update your apt config, because `apt-get update` will print output similar to this:
# apt-get update ... Err http://apt.arvados.org buster/main amd64 Packages 404 Not Found ... W: Failed to fetch http://apt.arvados.org/dists/buster/main/binary-amd64/Packages 404 Not Found E: Some index files failed to download. They have been ignored, or old ones used instead.
The layout of the repositories has changed. The new repository URLs are:
|Debian 10 ("buster")||deb http://apt.arvados.org/buster buster main|
|Debian 9 ("stretch")||deb http://apt.arvados.org/stretch stretch main|
|Ubuntu 18.04 ("bionic")||deb http://apt.arvados.org/bionic bionic main|
|Ubuntu 16.04 ("xenial")||deb http://apt.arvados.org/xenial xenial main|
The repository URLs for older, no longer supported distributions has also changed:
|Debian 8 ("jessie")||deb http://apt.arvados.org/jessie jessie main|
|Debian 7 ("wheezy")||deb http://apt.arvados.org/wheezy wheezy main|
|Ubuntu 14.04 ("trusty")||deb http://apt.arvados.org/trusty trusty main|
|Ubuntu 12.04 ("precise")||deb http://apt.arvados.org/precise precise main|
After updating your `/etc/sources.list.d/arvados.list` file, it is necessary to run `apt-get update`.
There are 2 branches to review:
- arvados: 40500de7ecb33a0f2197b2fe2449e6fc14b835a4 on branch 17218-repository-update
- arvados-dev: 680f3a3c0e6092013126959188a06c1b8bc5ce4c on branch 17218-repository-update