Idea #21020


Support XDG base directory envvars throughout the Python SDK

Added by Brett Smith 9 months ago. Updated 13 days ago.

Assigned To:
Start date:
Due date:
Story points:


We should support XDG Base Directory environment variables throughout the Python SDK. Things like XDG_CONFIG_PATH and XDG_CACHE_HOME. Note that this includes validations like ensuring the paths are absolute. There are probably packages on PyPI that can help us with this.

Known specific instances:

  • arvados.config.initialize should default to config_file=None. When the value is None, the function should search XDG_CONFIG_PATH to find arvados/settings.conf.
  • arvados.api.http_cache should create and write to to $XDG_CACHE_HOME/arvados/SUBDIR.
  • $ git grep -Fe .cache -e .config --line-number sdk/python/ sdk/cwl/ services/fuse/ services/dockercleaner/ tools/user-activity/ tools/crunchstat-summary/
    sdk/python/arvados/commands/        os.path.join('.cache', 'arvados', 'docker'), 0o700)
    sdk/python/arvados/commands/    CACHE_DIR = '.cache/arvados/arv-put'
    sdk/python/arvados/commands/    CACHE_DIR = '.cache/arvados/arv-put'
    sdk/python/arvados/            self._disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
    sdk/python/tests/        disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")
    services/fuse/arvados_fuse/            help="Filesystem cache location (default `~/.cache/arvados/keep`)",
    services/fuse/tests/        disk_cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "arvados", "keep")

Subtasks 1 (0 open1 closed)

Task #21755: Review 21020-pysdk-env-pathsResolvedBrett Smith05/31/2024Actions

Related issues

Related to Arvados - Idea #21323: System services use cache/config directories indicated by XDG env vars / systemd directivesNewActions
Actions #1

Updated by Tom Clegg 6 months ago

  • Related to Idea #21323: System services use cache/config directories indicated by XDG env vars / systemd directives added
Actions #3

Updated by Peter Amstutz about 2 months ago

  • Target version changed from Future to Development 2024-05-22 sprint
Actions #4

Updated by Peter Amstutz about 1 month ago

  • Assigned To set to Brett Smith
Actions #5

Updated by Brett Smith 28 days ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz 27 days ago

  • Status changed from New to In Progress
Actions #7

Updated by Peter Amstutz 26 days ago

  • Target version changed from Development 2024-05-22 sprint to Development 2024-06-05 sprint
Actions #8

Updated by Brett Smith 24 days ago

21020-pysdk-env-paths @ 21beff7a6713d5a2ea0fc7d9b88bd4c51119d737 - developer-run-tests: #4255

  • All agreed upon points are implemented / addressed.
    • Yes
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • N/A
  • Code is tested and passing, both automated and manual, what manual testing was done is described
    • Tests have been added for all places where functionality changed, both to cover existing as well as new functionality.
  • Documentation has been updated.
    • Yes
  • Behaves appropriately at the intended scale (describe intended scale).
    • No change in scale
  • Considered backwards and forwards compatibility issues between client and server.
    • To a fault. Class variables and function arguments all have backwards-compatible types, and functions have backwards-compatible signatures, even though some of these make a lot less sense when we're searching for filesystem paths dynamically.
  • Follows our coding standards and GUI style guidelines.
    • Yes
Actions #9

Updated by Lucas Di Pentima 20 days ago

Comments for review @ 21beff7

  • I think arvados.commannds._util.make_home_conf_dir() could be removed since it seems to be dead code now.
  • In the edge case of a previously existing installation where the user has their config at ~/.config/arvados/settings.conf but their XDG_CONFIG_HOME var points to a different place, if the user upgrades the PySDK, it would suddenly stop working. Should the SDK warn about that? Or maybe should we have an upgrade note to warn users of this?
  • These files might need some updating to reflect this new behavior:
    • doc/sdk/python/api-client.html.textile.liquid
    • doc/user/reference/api-tokens.html.textile.liquid
    • doc/user/topics/arv-copy.html.textile.liquid
    • doc/sdk/fuse/options.html.textile.liquid
Actions #10

Updated by Brett Smith 18 days ago

21020-pysdk-env-paths @ df989fd01534296d1c147987ab86b61c4ab7dc17 - developer-run-tests: #4263

In order to make the requested documentation updates, first I had to do some tangentially related documentation refreshing. Hope that's okay. I tried to keep the commits organized.

Recording discussion at standup: customizing XDG_CONFIG_HOME is expected to be uncommon, and something only advanced users do. Outside reference documentation, we do not provide complete instructions for handling this case. We only flag it with a high-level description of what to change. Any users in this situation are expected to be able to handle that.

Actions #11

Updated by Lucas Di Pentima 18 days ago

This LGTM, thanks for the orderly commit strategy.

Actions #12

Updated by Brett Smith 17 days ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF