Idea #21020
closed
Support XDG base directory envvars throughout the Python SDK
Description
We should support XDG Base Directory environment variables throughout the Python SDK. Things like XDG_CONFIG_PATH and XDG_CACHE_HOME. Note that this includes validations like ensuring the paths are absolute. There are probably packages on PyPI that can help us with this.
Known specific instances:
arvados.config.initializeshould default toconfig_file=None. When the value isNone, the function should searchXDG_CONFIG_PATHto findarvados/settings.conf.arvados.api.http_cacheshould create and write to to$XDG_CACHE_HOME/arvados/SUBDIR.$ git grep -Fe .cache -e .config --line-number sdk/python/ sdk/cwl/ services/fuse/ services/dockercleaner/ tools/user-activity/ tools/crunchstat-summary/ sdk/python/arvados/commands/keepdocker.py:186: os.path.join('.cache', 'arvados', 'docker'), 0o700) sdk/python/arvados/commands/put.py:354: CACHE_DIR = '.cache/arvados/arv-put' sdk/python/arvados/commands/put.py:436: CACHE_DIR = '.cache/arvados/arv-put' sdk/python/arvados/keep.py:179: self._disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep") sdk/python/tests/arvados_testutil.py:285: disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep") services/fuse/arvados_fuse/command.py:308: help="Filesystem cache location (default `~/.cache/arvados/keep`)", services/fuse/tests/mount_test_base.py:31: disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
Updated by Tom Clegg about 1 year ago
- Related to Idea #21323: System services use cache/config directories indicated by XDG env vars / systemd directives added
Updated by Peter Amstutz 9 months ago
- Target version changed from Future to Development 2024-05-22 sprint
Updated by Peter Amstutz 8 months ago
- Target version changed from Development 2024-05-22 sprint to Development 2024-06-05 sprint
Updated by Brett Smith 8 months ago
21020-pysdk-env-paths @ 21beff7a6713d5a2ea0fc7d9b88bd4c51119d737 - developer-run-tests: #4255
- All agreed upon points are implemented / addressed.
- Yes
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
- Code is tested and passing, both automated and manual, what manual testing was done is described
- Tests have been added for all places where functionality changed, both to cover existing as well as new functionality.
- Documentation has been updated.
- Yes
- Behaves appropriately at the intended scale (describe intended scale).
- No change in scale
- Considered backwards and forwards compatibility issues between client and server.
- To a fault. Class variables and function arguments all have backwards-compatible types, and functions have backwards-compatible signatures, even though some of these make a lot less sense when we're searching for filesystem paths dynamically.
- Follows our coding standards and GUI style guidelines.
- Yes
Updated by Lucas Di Pentima 8 months ago
Comments for review @ 21beff7
- I think
arvados.commannds._util.make_home_conf_dir()could be removed since it seems to be dead code now. - In the edge case of a previously existing installation where the user has their config at
~/.config/arvados/settings.confbut theirXDG_CONFIG_HOMEvar points to a different place, if the user upgrades the PySDK, it would suddenly stop working. Should the SDK warn about that? Or maybe should we have an upgrade note to warn users of this? - These files might need some updating to reflect this new behavior:
doc/sdk/python/api-client.html.textile.liquiddoc/user/reference/api-tokens.html.textile.liquiddoc/user/topics/arv-copy.html.textile.liquiddoc/sdk/fuse/options.html.textile.liquid
Updated by Brett Smith 8 months ago
21020-pysdk-env-paths @ df989fd01534296d1c147987ab86b61c4ab7dc17 - developer-run-tests: #4263
In order to make the requested documentation updates, first I had to do some tangentially related documentation refreshing. Hope that's okay. I tried to keep the commits organized.
Recording discussion at standup: customizing XDG_CONFIG_HOME is expected to be uncommon, and something only advanced users do. Outside reference documentation, we do not provide complete instructions for handling this case. We only flag it with a high-level description of what to change. Any users in this situation are expected to be able to handle that.
Updated by Lucas Di Pentima 8 months ago
This LGTM, thanks for the orderly commit strategy.
Updated by Brett Smith 8 months ago
- Status changed from In Progress to Resolved
Updated by Brett Smith about 2 months ago
- Related to Bug #22350: LSF crunch-run using $HOME for the cache added