Arvados

Peter Amstutz wrote:

Docker 25 changed the format of "docker save", this is already causing us problems with #21417, determine if there are any other backwards/forwards compatibility issues using images produced by "docker save" and newer/older versions of Docker.

According to the implementing pull request, the correct way to read a saved Docker image is to start with manifest.json and work through that. If you do that properly, images should be compatible in both directions: older versions can load the OCI format, and newer versions can load the old format. So I don't think we have anything to worry about there.

Reading the rest of the release notes, nothing jumped out at me as an obvious problem. Although, in fairness, I'm not sure I would've caught the docker image save issue just from reading either. But I don't see anything I could make a concrete ticket out of.

• I did searches for a couple removed features that I thought we might be using somewhere (devicemapper storage engine, oom-score-adjust), but didn't find any results.
• This release "deprecate[s] API versions older than 1.24," we're standardized on 1.35 so it seems like we're still in a healthy support window there. It seems like the main motivation for this deprecation is getting rid of pre-versioned APIs, there's not any discussion or timetable offered for deprecating later versions like we're on.

Peter Amstutz wrote:

Confirm that older docker images (pre Docker 25) works with Docker 25.

Confirm that newer docker images (produced by Docker 25) works with older Docker.

I installed Docker CE 25 on my own system, and then used the Docker running on ci (Debian 10, Docker 17) as my target "old Docker," since I figured that was the oldest we were likely to support. Everything worked in both directions.

Saving an image on the old Docker, confirming the format, and loading a new image:

ci:~ % sudo docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:09:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:09:06 2017
 OS/Arch:      linux/amd64
 Experimental: false

ci:~ % sudo docker save debian:bullseye-slim | xz > Scratch/bullseye-slim.tar.xz
ci:~ % tar -tf Scratch/bullseye-slim.tar.xz | tail
634e16cb03ae927ff6e94ef9db8adae6749a3c9d9cc2e8c6e880ba83a291a85d.json
f74164fbfb1039d10771b8bfbb7a95fca0c4aefd10bd2016de074005e41138c4/
f74164fbfb1039d10771b8bfbb7a95fca0c4aefd10bd2016de074005e41138c4/VERSION
f74164fbfb1039d10771b8bfbb7a95fca0c4aefd10bd2016de074005e41138c4/json
f74164fbfb1039d10771b8bfbb7a95fca0c4aefd10bd2016de074005e41138c4/layer.tar
manifest.json
repositories
ci:~ % xzcat Scratch/rocky8.tar.xz | sudo docker load
a648944b2005: Loading layer [==================================================>]  95.89MB/95.89MB
Loaded image: rockylinux:8.8-minimal

Saving an image on the new Docker, confirming the OCI format, and loading an old image:

arvados % docker version
Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

arvados % docker save rockylinux:8.8-minimal | xz > ~/Scratch/rocky8.tar.xz
arvados % tar -tf ~/Scratch/rocky8.tar.xz| tail
blobs/
blobs/sha256/
blobs/sha256/1e6dbe6f1f8e186febb5423ff3955bb2bd7d0af30d488209c169cd6225d2334f
blobs/sha256/8b19ff2cb66f753d86583a14a4244f2f5f994ab3822943aac9388c1b9d5d79b3
blobs/sha256/a648944b2005cfdd129626ab738c178518b2450f0a7fa48a25cf166ee9ba0361
blobs/sha256/aed560c442ad04230131a438fd832adf9988b074901cb77cde68bfe03084830f
index.json
manifest.json
oci-layout
repositories
arvados % xzcat ~/Scratch/bullseye-slim.tar.xz| docker load
Loaded image: debian:bullseye-slim

I think we can probably mark this as resolved, then.