Arvados

On HPC, accounting and quotas are based on the user submitting the job.

Current Arvados deployment uses a single "crunch" user for everything.

As a result, multiple Arvados users may end up throttled to the allocation for a single "crunch" user.

If the job can be submitted on behalf of the user, with their own account, then HPC quotas and accounting works as intended.

Questions to resolve:

• Mechanics of submitting as a specific user on supported HPC systems
  • requires dispatcher to be granted some kind of elevated access
  • probably want to run actual the container as the regular user
• How to protect privileged resources from regular users
  • running local keepstore, don't want to expose keepstore directory or object store credentials
  • don't expose Arvados configuration file
  • other secrets, such as system-wide dispatcher token that shouldn't be visible to regular users

I suspect we'll need a split permission architecture where some parts are suid and run as the crunch user, but as much as possible runs as the regular user.