Feature #17218


debian package repos: switch from freight to something faster

Added by Ward Vandewege over 3 years ago. Updated almost 3 years ago.

Assigned To:
Target version:
Story points:
Release relationship:


Freight works great, except that it is extremely slow to add packages to an existing repository, the speed of addition is O(n) for n the number of packages in the repository.

From preliminary testing, aptly seems like a good replacement candidate (it doesn't have this particular problem).

Related issues

Related to Arvados - Bug #17219: refactor scripts that move old packages to attic reposResolvedWard Vandewege01/04/2021Actions
Actions #1

Updated by Ward Vandewege over 3 years ago

  • Assigned To set to Ward Vandewege
  • Status changed from New to In Progress
Actions #2

Updated by Ward Vandewege over 3 years ago

  • Target version set to 2021-01-06 Sprint
Actions #3

Updated by Ward Vandewege over 3 years ago

Aptly is much more strict about duplicate packages. Namely, from

A repository must not include different packages (different content) with the same package name, version, and architecture. When a repository is meant to be used as a supplement to another repository this should hold for the joint main+supplement repository as well.

Actions #4

Updated by Ward Vandewege over 3 years ago

Almost ready for the migration.

As part of this, I am changing the Origin and Label fields of our apt repos. They currently say 'Freight', and that will become 'Arvados'.

Updating the buster-dev repo now takes on the order of 30 seconds, as opposed to ~1000 seconds with freight.

The functionality can be tested on the (temporary) apt-dev host. E.g. put this in /etc/sources.list.d/arvados.list

deb buster main

(or use stretch, jessie, wheezy, bionic, xenial, trusty or precise)

Because we have to change the URLs in people's sources.list files, they will get a 404 on the old URLs after we flip the switch. It was not possible to rewrite the URLs automatically and make apt happy, because the `Packages` file relies on relative URLs.

Unfortunately, it seems to be impossible to get some sort of warning through to the command line when `apt` gets a 404. Apt even rewrites most error codes... sigh. So, I did the next best thing, and put a file here: (this file will appear at when the switch is flipped.

There is one more loose end - the set of scripts that move files from the -dev to -attic repos will need to be updated. They are not checked into our code tree. I will correct that, fix and simplify, as part of ticket #17219.

This is the forum post that I would like to put under 'Announcements':


Debian/Ubuntu apt repository changes

The Arvados project has reorganized its package repositories for Debian and Ubuntu.

If you installed Arvados before 2020-12-22 on a Debian/Ubuntu system using the Arvados package repositories, you need to update your apt config, because `apt-get update` will print output similar to this:

  # apt-get update
  Err buster/main amd64 Packages                 
    404  Not Found
  W: Failed to fetch  404  Not Found

  E: Some index files failed to download. They have been ignored, or old ones used instead.

The layout of the repositories has changed. The new repository URLs are:

- -
Debian 10 ("buster") deb buster main
Debian 9 ("stretch") deb stretch main
Ubuntu 18.04 ("bionic") deb bionic main
Ubuntu 16.04 ("xenial") deb xenial main

The repository URLs for older, no longer supported distributions has also changed:

- -
Debian 8 ("jessie") deb jessie main
Debian 7 ("wheezy") deb wheezy main
Ubuntu 14.04 ("trusty") deb trusty main
Ubuntu 12.04 ("precise") deb precise main

After updating your `/etc/sources.list.d/arvados.list` file, it is necessary to run `apt-get update`.


There are 2 branches to review:

  • arvados: 40500de7ecb33a0f2197b2fe2449e6fc14b835a4 on branch 17218-repository-update
  • arvados-dev: 680f3a3c0e6092013126959188a06c1b8bc5ce4c on branch 17218-repository-update
Actions #5

Updated by Ward Vandewege over 3 years ago

  • Related to Bug #17219: refactor scripts that move old packages to attic repos added
Actions #6

Updated by Ward Vandewege over 3 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #7

Updated by Ward Vandewege over 3 years ago

commit:a83c3cfdfe62d6a01e1aa33360e11908cbdbd4f1 on the arvados-formula repo is ready for review.

Actions #8

Updated by Javier Bértoli over 3 years ago

Merged and updated the salt formula (new version, 1.1.4).

updated the provision script (commit 3d7250d6e, branch 17218-update-apt-repos)

Actions #9

Updated by Peter Amstutz almost 3 years ago

  • Release set to 38

Also available in: Atom PDF