Idea #21020: Support XDG base directory envvars throughout the Python SDK - Arvados

Actions

Copy link

Idea #21020

closed

Support XDG base directory envvars throughout the Python SDK

Added by Brett Smith 10 months ago. Updated about 2 months ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Brett Smith

Category:

SDKs

Target version:

Development 2024-06-05 sprint

Start date:

05/31/2024

Due date:

Story points:

Description

We should support XDG Base Directory environment variables throughout the Python SDK. Things like XDG_CONFIG_PATH and XDG_CACHE_HOME. Note that this includes validations like ensuring the paths are absolute. There are probably packages on PyPI that can help us with this.

Known specific instances:

arvados.config.initialize should default to config_file=None. When the value is None, the function should search XDG_CONFIG_PATH to find arvados/settings.conf.
arvados.api.http_cache should create and write to to $XDG_CACHE_HOME/arvados/SUBDIR.

$ git grep -Fe .cache -e .config --line-number sdk/python/ sdk/cwl/ services/fuse/ services/dockercleaner/ tools/user-activity/ tools/crunchstat-summary/
sdk/python/arvados/commands/keepdocker.py:186:        os.path.join('.cache', 'arvados', 'docker'), 0o700)
sdk/python/arvados/commands/put.py:354:    CACHE_DIR = '.cache/arvados/arv-put'
sdk/python/arvados/commands/put.py:436:    CACHE_DIR = '.cache/arvados/arv-put'
sdk/python/arvados/keep.py:179:            self._disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
sdk/python/tests/arvados_testutil.py:285:        disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
services/fuse/arvados_fuse/command.py:308:            help="Filesystem cache location (default `~/.cache/arvados/keep`)",
services/fuse/tests/mount_test_base.py:31:        disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")

Subtasks 1 (0 open — 1 closed)

Related issues

Actions

Copy link

Updated by Tom Clegg 7 months ago

Related to Idea #21323: System services use cache/config directories indicated by XDG env vars / systemd directives added

Actions

Copy link

Updated by Peter Amstutz 3 months ago

Target version changed from Future to Development 2024-05-22 sprint

Actions

Copy link

Updated by Peter Amstutz 3 months ago

Assigned To set to Brett Smith

Actions

Copy link

Updated by Brett Smith 2 months ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz 2 months ago

Status changed from New to In Progress

Actions

Copy link

Updated by Peter Amstutz 2 months ago

Target version changed from Development 2024-05-22 sprint to Development 2024-06-05 sprint

Actions

Copy link

Updated by Brett Smith 2 months ago

21020-pysdk-env-paths @ 21beff7a6713d5a2ea0fc7d9b88bd4c51119d737 - developer-run-tests: #4255

All agreed upon points are implemented / addressed.
- Yes
Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
Code is tested and passing, both automated and manual, what manual testing was done is described
- Tests have been added for all places where functionality changed, both to cover existing as well as new functionality.
Documentation has been updated.
- Yes
Behaves appropriately at the intended scale (describe intended scale).
- No change in scale
Considered backwards and forwards compatibility issues between client and server.
- To a fault. Class variables and function arguments all have backwards-compatible types, and functions have backwards-compatible signatures, even though some of these make a lot less sense when we're searching for filesystem paths dynamically.
Follows our coding standards and GUI style guidelines.
- Yes

Actions

Copy link

Updated by Lucas Di Pentima about 2 months ago

Comments for review @ 21beff7

I think arvados.commannds._util.make_home_conf_dir() could be removed since it seems to be dead code now.
In the edge case of a previously existing installation where the user has their config at ~/.config/arvados/settings.conf but their XDG_CONFIG_HOME var points to a different place, if the user upgrades the PySDK, it would suddenly stop working. Should the SDK warn about that? Or maybe should we have an upgrade note to warn users of this?
These files might need some updating to reflect this new behavior:
- doc/sdk/python/api-client.html.textile.liquid
- doc/user/reference/api-tokens.html.textile.liquid
- doc/user/topics/arv-copy.html.textile.liquid
- doc/sdk/fuse/options.html.textile.liquid

Actions

Copy link

#10

Updated by Brett Smith about 2 months ago

21020-pysdk-env-paths @ df989fd01534296d1c147987ab86b61c4ab7dc17 - developer-run-tests: #4263

In order to make the requested documentation updates, first I had to do some tangentially related documentation refreshing. Hope that's okay. I tried to keep the commits organized.

Recording discussion at standup: customizing XDG_CONFIG_HOME is expected to be uncommon, and something only advanced users do. Outside reference documentation, we do not provide complete instructions for handling this case. We only flag it with a high-level description of what to change. Any users in this situation are expected to be able to handle that.

Actions

Copy link

#11