Arvados

Last month a change was introduced in containerd that uses unix.Mknodat() to create files.

FUSE doesn't support this so we have seen errors like the following on workflows that mount Keep as read-write:

2024-10-01T13:47:48.741927761Z crunch-run 2.7.4 (go1.20.6) started
2024-10-01T13:47:48.743440645Z crunch-run process has uid=0(root) gid=0(root) groups=0(root)
2024-10-01T13:47:49.172962531Z Using FUSE mount: /usr/bin/arv-mount 2.7.4
2024-10-01T13:47:49.182025067Z Using container runtime: docker Engine 27.3.1, containerd 1.7.22, runc 1.1.14, docker-init 0.19.0
2024-10-01T13:47:49.182054510Z Executing container: zzzzz-dz642-xuhaivxsc4hqsju
...
2024-10-01T13:47:49.288954818Z Running [arv-mount --foreground --read-write --storage-classes default --crunchstat-interval=10 --allow-other --disk-cache --disk-cache-dir /tmp/crunch-run.zzzzz-dz642-xuhaivxsc4hqsju.2266811772/keepcache303192798 --file-cache 2147483648 --mount-tmp tmp0 --mount-by-pdh by_id --disable-event-listening --mount-by-id by_uuid /tmp/crunch-run.zzzzz-dz642-xuhaivxsc4hqsju.2266811772/keep4225894812]
2024-10-01T13:47:49.778525558Z Fetching Docker image from collection '2ed1bc9f0a11a6973076269fbad024a1+302'
2024-10-01T13:47:49.834876928Z Using Docker image id "sha256:6132fae1763d234fc4f28018ed7caf579e138caaa17c242106a29a54346ea7bb" 
2024-10-01T13:47:49.834909340Z Loading Docker image from keep
2024-10-01T13:47:49.836269198Z Creating Docker container
2024-10-01T13:47:50.225643161Z Starting container
2024-10-01T13:47:50.583857471Z error in Run: could not start container: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/tmp/crunch-run.zzzzz-dz642-xuhaivxsc4hqsju.2266811772/keep4225894812/by_id/4b113ad7ff8cd8f2abd25d9bdbe43de2+112/download.sh" to rootfs at "/var/spool/cwl/download.sh": create mount destination for /var/spool/cwl/download.sh mount: create target of file bind-mount: mknod regular file /tmp/docker-data/overlay2/35897e3b35b5ae55a61c943fec29c96f71b097bfefda61a762eab80211efca2a/merged/var/spool/cwl/download.sh: function not implemented: unknown
...