Arvados

Aptly is much more strict about duplicate packages. Namely, from https://wiki.debian.org/DebianRepository/Format?action=show&redirect=RepositoryFormat#Duplicate_Packages:

A repository must not include different packages (different content) with the same package name, version, and architecture. When a repository is meant to be used as a supplement to another repository this should hold for the joint main+supplement repository as well.

Almost ready for the migration.

As part of this, I am changing the Origin and Label fields of our apt repos. They currently say 'Freight', and that will become 'Arvados'.

Updating the buster-dev repo now takes on the order of 30 seconds, as opposed to ~1000 seconds with freight.

The functionality can be tested on the (temporary) apt-dev host. E.g. put this in /etc/sources.list.d/arvados.list

deb http://apt-dev.arvados.org/buster buster main

(or use stretch, jessie, wheezy, bionic, xenial, trusty or precise)

Because we have to change the URLs in people's sources.list files, they will get a 404 on the old URLs after we flip the switch. It was not possible to rewrite the URLs automatically and make apt happy, because the `Packages` file relies on relative URLs.

Unfortunately, it seems to be impossible to get some sort of warning through to the command line when `apt` gets a 404. Apt even rewrites most error codes... sigh. So, I did the next best thing, and put a file here: http://apt-dev.arvados.org/REPOSITORIES_HAVE_BEEN_REORGANIZED.txt (this file will appear at http://apt.arvados.org/REPOSITORIES_HAVE_BEEN_REORGANIZED.txt) when the switch is flipped.

There is one more loose end - the set of scripts that move files from the -dev to -attic repos will need to be updated. They are not checked into our code tree. I will correct that, fix and simplify, as part of ticket #17219.

This is the forum post that I would like to put under 'Announcements':

==================================

Debian/Ubuntu apt repository changes

The Arvados project has reorganized its package repositories for Debian and Ubuntu.

If you installed Arvados before 2020-12-22 on a Debian/Ubuntu system using the Arvados package repositories, you need to update your apt config, because `apt-get update` will print output similar to this:

  # apt-get update
  ...
  Err http://apt.arvados.org buster/main amd64 Packages                 
    404  Not Found
  ...
  W: Failed to fetch http://apt.arvados.org/dists/buster/main/binary-amd64/Packages  404  Not Found

  E: Some index files failed to download. They have been ignored, or old ones used instead.

The layout of the repositories has changed. The new repository URLs are:

The repository URLs for older, no longer supported distributions has also changed:

After updating your `/etc/sources.list.d/arvados.list` file, it is necessary to run `apt-get update`.

===========================================

There are 2 branches to review:

• arvados: 40500de7ecb33a0f2197b2fe2449e6fc14b835a4 on branch 17218-repository-update
• arvados-dev: 680f3a3c0e6092013126959188a06c1b8bc5ce4c on branch 17218-repository-update

commit:a83c3cfdfe62d6a01e1aa33360e11908cbdbd4f1 on the arvados-formula repo is ready for review.

Merged and updated the salt formula (new version, 1.1.4).

updated the provision script (commit 3d7250d6e, branch 17218-update-apt-repos)