Arvados

Brett Smith wrote:

Tom Clegg wrote:

In the meantime, there might be an effective workaround, like running crunch-dispatch with umask=002 and the same GID as the API server process.

This seems to have some promise. Cache files currently in production have mode 0644, where www-data's umask is 022. It looks like the Rails cache code is not imposing a mode stricter than the umask.

I dug into this to verify it. Here's Rails' code to write a cache file. Here's the atomic_write method called there. atomic_write specifically uses the default ownership and permissions of files in that directory. So setting tmp/cache setgid and umask 002 should be sufficient to fix the issue.

For what it's worth: the tracebacks we get from this refer to groups_for_user files. These are caches specifically created in the User model, calling methods on Rails cache. So yet another option might be to set up a different cache (another FileStore backed by a different directory?) when we're running crunch-dispatch, although I'm less sure how to do that nicely.

Manual test procedure:

• Start an API server without crunch-dispatch. Put at least one job in its queue.
• Stop the API server.
• Empty Rails' tmp/cache directory.
• Start crunch-dispatch.
• Confirm that cache files created by crunch-dispatch are writable by the Web server user (www-data on Debian).

on arvados dev reviewing b3badf7..e48701c only 1 line change in

jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh

LGTM!

Test process:

1. Temporarily add "User.first.group_permissions" to #run method in crunch-dispatch to cause a cache to be written (I couldn't figure out the conditions that cause crunch-dispatch to create cache files consistently on startup)
2. Run sudo -u root -g peter crunch-dispatch.rb
3. This causes a groups_for_user file to be created.
4. Without umask:
   

   -rw-r--r-- 1 root peter  530 Sep 23 12:40 groups_for_user_bseta-tpzed-000000000000000
   

   
   API server breaks because it tries to delete groups_for_user_bseta-tpzed-000000000000000 and it can't
5. With umask:
   

   -rw-rw-r-- 1 root peter  531 Sep 23 12:41 groups_for_user_bseta-tpzed-000000000000000
   

   
   API server works because it is able to delete groups_for_user_bseta-tpzed-000000000000000

Branch is 7228-crunch-dispatch-umask

reviewed 27cb821..4a1fdbc

LGTM