Story #14923

[CWL] Reduce memory footprint by using webdav to get file/dir listings in collections (instead of Python SDK)

Added by Tom Clegg 5 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

The hope is to reduce arvados-cwl-runner's memory requirement enough to run several a-c-r containers at once on a single low-cost node (see #14922).

Depending on how much work this is to implement/test, it may be worth first instrumenting a-c-r and predicting the impact on memory footprint and runtime (e.g., webdav cache hits will be slower than in-process Python SDK cache hits).

Also available in: Atom PDF