Arvados » Arvados Epics

Notes from recent discussions, especially the 2024-03-06 engineering meeting

Reasons to start prioritizing this:

• Heavier user of arv-mount is consistently hitting concurrency and performance issues
• python-llfuse is in maintenance mode - They did a release in November 2023 that should keep us set for a while but the writing's on the wall.

Potential milestones:

1. Go mount can serve crunch-run's purposes
2. Go mount can serve users' purposes for read-only mounts
   • Support as many command line options as practical (see below)
   • Bounded memory use
   • Disk caching of data
3. Go mount adds --read-write support

Quoting lib/crunchrun/crunchrun.go, here are the exact options crunch-run can currently call arv-mount with:

• --foreground
• --read-write (I think because of --mount-tmp)
• --storage-classes
• --crunchstat-interval
• --allow-other (since the compute work may run as another user inside the container)
• --disk-cache
• --disk-cache-dir
• --file-cache
• --ram-cache
• --mount-tmp
• --mount-by-pdh
• --disable-event-listening (not totally clear why, just trying to reduce network traffic?)
• --mount-by-id
• --unmount-timeout, --unmount (cleanup after the job is done, IMO I think we could implement this with standard tools, see below)
• --version (basic "can run" check)

Implementation notes about specific options:

• Please use a GNU-style argument parser so --long-options still work. Our users use Linux, not Plan 9, there's no good reason to force them to s/--/-/g in all their tooling.
• By the time we get to phase #2 and start offering this to users, for a combination of reasons of "it's required" or "it's too useful not too have" or "it's very low effort to implement after you have the previous options," we expect to support everything except possibly the following:
  • --read-write (you'll need to support writable tmp mounts for crunch-run but updating projects and collections is the next phase)
  • --foreground: IMO the new mount should always run foreground, this should become a noop option for compatibility and we tell users "if you want it to daemonize run it with systemd-run"
  • --replace, --subtype, --unmount-*: I think the main reason these options exist is because the code exists to support --exec so we might as well expose it. If we end up reimplementing that same pattern, that's fine. But if there's no need, I think it's low risk to remove these options and tell users to use fusermount -u, findmnt, and other existing tools. I don't think we need to reimplement those just for compatibility.
• Consider this alternative, more ergonomic spelling for --exec: arvados-client mount MOUNTDIR [COMMAND ARG ...]
• Consider #19934 for --exec (start the process with the mount point as the working directory)