Idea #21020
closed
Support XDG base directory envvars throughout the Python SDK
Added by Brett Smith 10 months ago.
Updated about 2 months ago.
Description
We should support XDG Base Directory environment variables throughout the Python SDK. Things like XDG_CONFIG_PATH and XDG_CACHE_HOME. Note that this includes validations like ensuring the paths are absolute. There are probably packages on PyPI that can help us with this.
Known specific instances:
arvados.config.initialize should default to config_file=None. When the value is None, the function should search XDG_CONFIG_PATH to find arvados/settings.conf.arvados.api.http_cache should create and write to to $XDG_CACHE_HOME/arvados/SUBDIR.$ git grep -Fe .cache -e .config --line-number sdk/python/ sdk/cwl/ services/fuse/ services/dockercleaner/ tools/user-activity/ tools/crunchstat-summary/
sdk/python/arvados/commands/keepdocker.py:186: os.path.join('.cache', 'arvados', 'docker'), 0o700)
sdk/python/arvados/commands/put.py:354: CACHE_DIR = '.cache/arvados/arv-put'
sdk/python/arvados/commands/put.py:436: CACHE_DIR = '.cache/arvados/arv-put'
sdk/python/arvados/keep.py:179: self._disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
sdk/python/tests/arvados_testutil.py:285: disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
services/fuse/arvados_fuse/command.py:308: help="Filesystem cache location (default `~/.cache/arvados/keep`)",
services/fuse/tests/mount_test_base.py:31: disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
- Related to Idea #21323: System services use cache/config directories indicated by XDG env vars / systemd directives added
- Target version changed from Future to Development 2024-05-22 sprint
- Assigned To set to Brett Smith
- Description updated (diff)
- Status changed from New to In Progress
- Target version changed from Development 2024-05-22 sprint to Development 2024-06-05 sprint
21020-pysdk-env-paths @ 21beff7a6713d5a2ea0fc7d9b88bd4c51119d737 - developer-run-tests: #4255 
- All agreed upon points are implemented / addressed.
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- Code is tested and passing, both automated and manual, what manual testing was done is described
- Tests have been added for all places where functionality changed, both to cover existing as well as new functionality.
- Documentation has been updated.
- Behaves appropriately at the intended scale (describe intended scale).
- Considered backwards and forwards compatibility issues between client and server.
- To a fault. Class variables and function arguments all have backwards-compatible types, and functions have backwards-compatible signatures, even though some of these make a lot less sense when we're searching for filesystem paths dynamically.
- Follows our coding standards and GUI style guidelines.
Comments for review @ 21beff7
- I think
arvados.commannds._util.make_home_conf_dir() could be removed since it seems to be dead code now.
- In the edge case of a previously existing installation where the user has their config at
~/.config/arvados/settings.conf but their XDG_CONFIG_HOME var points to a different place, if the user upgrades the PySDK, it would suddenly stop working. Should the SDK warn about that? Or maybe should we have an upgrade note to warn users of this?
- These files might need some updating to reflect this new behavior:
doc/sdk/python/api-client.html.textile.liquiddoc/user/reference/api-tokens.html.textile.liquiddoc/user/topics/arv-copy.html.textile.liquiddoc/sdk/fuse/options.html.textile.liquid
21020-pysdk-env-paths @ df989fd01534296d1c147987ab86b61c4ab7dc17 - developer-run-tests: #4263 
In order to make the requested documentation updates, first I had to do some tangentially related documentation refreshing. Hope that's okay. I tried to keep the commits organized.
Recording discussion at standup: customizing XDG_CONFIG_HOME is expected to be uncommon, and something only advanced users do. Outside reference documentation, we do not provide complete instructions for handling this case. We only flag it with a high-level description of what to change. Any users in this situation are expected to be able to handle that.
This LGTM, thanks for the orderly commit strategy.
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by 
Updated by