Feature #17218

debian package repos: switch from freight to something faster

Added by Ward Vandewege 5 months ago. Updated about 17 hours ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Story points:
-
Release relationship:
Auto

Description

Freight works great, except that it is extremely slow to add packages to an existing repository, the speed of addition is O(n) for n the number of packages in the repository.

From preliminary testing, aptly seems like a good replacement candidate (it doesn't have this particular problem).


Related issues

Related to Arvados - Bug #17219: refactor scripts that move old packages to attic reposResolved01/04/2021

Associated revisions

Revision 40500de7 (diff)
Added by Ward Vandewege 5 months ago

We've made a change in the structure of our apt repositories.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 680f3a3c (diff)
Added by Ward Vandewege 5 months ago

We've made a change in the structure of our apt repositories.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 85b3e1e4
Added by Ward Vandewege 5 months ago

Merge branch '17218-repository-update'

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 533f0c79
Added by Ward Vandewege 5 months ago

Merge branch '17218-repository-update'

closes #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision b09f5cbc (diff)
Added by Ward Vandewege 5 months ago

We've made a change in the structure of our apt repositories.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 4c0f702a (diff)
Added by Ward Vandewege 5 months ago

Do not try to add packages to the repository that are already in it.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision c16c3f3e (diff)
Added by Ward Vandewege 5 months ago

Fix detection of packages that were already built.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision d0157c3b
Added by Javier Bértoli 5 months ago

Merge branch '17218-update-apt-repos'

closes #17218

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <>

Revision 76e93fdb (diff)
Added by Ward Vandewege 3 months ago

Bugfix for the `testing_to_stable_publish_packages.sh` script to reflect
our switch to aptly.

Some refactoring to remove exitcode nonsense.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 38c4ad77 (diff)
Added by Ward Vandewege 3 months ago

When a package can't be published, print which distribution the error is
with.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

History

#1 Updated by Ward Vandewege 5 months ago

  • Assigned To set to Ward Vandewege
  • Status changed from New to In Progress

#2 Updated by Ward Vandewege 5 months ago

  • Target version set to 2021-01-06 Sprint

#3 Updated by Ward Vandewege 5 months ago

Aptly is much more strict about duplicate packages. Namely, from https://wiki.debian.org/DebianRepository/Format?action=show&redirect=RepositoryFormat#Duplicate_Packages:

A repository must not include different packages (different content) with the same package name, version, and architecture. When a repository is meant to be used as a supplement to another repository this should hold for the joint main+supplement repository as well.

#4 Updated by Ward Vandewege 5 months ago

Almost ready for the migration.

As part of this, I am changing the Origin and Label fields of our apt repos. They currently say 'Freight', and that will become 'Arvados'.

Updating the buster-dev repo now takes on the order of 30 seconds, as opposed to ~1000 seconds with freight.

The functionality can be tested on the (temporary) apt-dev host. E.g. put this in /etc/sources.list.d/arvados.list

deb http://apt-dev.arvados.org/buster buster main

(or use stretch, jessie, wheezy, bionic, xenial, trusty or precise)

Because we have to change the URLs in people's sources.list files, they will get a 404 on the old URLs after we flip the switch. It was not possible to rewrite the URLs automatically and make apt happy, because the `Packages` file relies on relative URLs.

Unfortunately, it seems to be impossible to get some sort of warning through to the command line when `apt` gets a 404. Apt even rewrites most error codes... sigh. So, I did the next best thing, and put a file here: http://apt-dev.arvados.org/REPOSITORIES_HAVE_BEEN_REORGANIZED.txt (this file will appear at http://apt.arvados.org/REPOSITORIES_HAVE_BEEN_REORGANIZED.txt) when the switch is flipped.

There is one more loose end - the set of scripts that move files from the -dev to -attic repos will need to be updated. They are not checked into our code tree. I will correct that, fix and simplify, as part of ticket #17219.

This is the forum post that I would like to put under 'Announcements':

==================================

Debian/Ubuntu apt repository changes

The Arvados project has reorganized its package repositories for Debian and Ubuntu.

If you installed Arvados before 2020-12-22 on a Debian/Ubuntu system using the Arvados package repositories, you need to update your apt config, because `apt-get update` will print output similar to this:

  # apt-get update
  ...
  Err http://apt.arvados.org buster/main amd64 Packages                 
    404  Not Found
  ...
  W: Failed to fetch http://apt.arvados.org/dists/buster/main/binary-amd64/Packages  404  Not Found

  E: Some index files failed to download. They have been ignored, or old ones used instead.

The layout of the repositories has changed. The new repository URLs are:

Distribution
- -
Debian 10 ("buster") deb http://apt.arvados.org/buster buster main
Debian 9 ("stretch") deb http://apt.arvados.org/stretch stretch main
Ubuntu 18.04 ("bionic") deb http://apt.arvados.org/bionic bionic main
Ubuntu 16.04 ("xenial") deb http://apt.arvados.org/xenial xenial main

The repository URLs for older, no longer supported distributions has also changed:

Distribution
- -
Debian 8 ("jessie") deb http://apt.arvados.org/jessie jessie main
Debian 7 ("wheezy") deb http://apt.arvados.org/wheezy wheezy main
Ubuntu 14.04 ("trusty") deb http://apt.arvados.org/trusty trusty main
Ubuntu 12.04 ("precise") deb http://apt.arvados.org/precise precise main

After updating your `/etc/sources.list.d/arvados.list` file, it is necessary to run `apt-get update`.

===========================================

There are 2 branches to review:

  • arvados: 40500de7ecb33a0f2197b2fe2449e6fc14b835a4 on branch 17218-repository-update
  • arvados-dev: 680f3a3c0e6092013126959188a06c1b8bc5ce4c on branch 17218-repository-update

#5 Updated by Ward Vandewege 5 months ago

  • Related to Bug #17219: refactor scripts that move old packages to attic repos added

#6 Updated by Ward Vandewege 5 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#7 Updated by Ward Vandewege 5 months ago

commit:a83c3cfdfe62d6a01e1aa33360e11908cbdbd4f1 on the arvados-formula repo is ready for review.

#8 Updated by Javier Bértoli 5 months ago

Merged and updated the salt formula (new version, 1.1.4).

updated the provision script (commit 3d7250d6e, branch 17218-update-apt-repos)

#9 Updated by Peter Amstutz about 17 hours ago

  • Release set to 38

Also available in: Atom PDF