Arvados

The scratch size used to choose a suitable node type is computed by the dispatcher (it's a function of "tmp" mount sizes and approximate docker image size) which isn't quite the same thing as the CWL ResourceRequirement number.

It looks like Workbench could get the ResourceRequirement number by looking at the size specified for the /var/spool/cwl tmp mount. In the given example, the API response tab shows this was 135524253696 (~126 GiB).

Currently the minimum scratch space number used to choose a VM type isn't saved or logged anywhere (unless slurm is in charge of choosing node types, in which case it's mentioned in the sbatch ... --tmp=%d ... command line in system logs). We could log it to system logs in the cloud scenario too easily enough -- but in both cases logging it to the container log where a user could see it will require a bit more work because currently the dispatcher doesn't have a way to write to the container log.

Would it be enough to show the CWL ResourceRequirement, or do we also need to log/show the actual minimum scratch space used to choose a node type?

Tom Clegg wrote:

Would it be enough to show the CWL ResourceRequirement, or do we also need to log/show the actual minimum scratch space used to choose a node type?

I think both would be useful in this case. I would like to know what I asked for to make sure it matches what my CWL says, and also what the difference was for what I asked for and what was actually computed to choose a node type. That way its very obvious if I'm hitting some edge case where if I lower the scratch space ResourceRequirement by n bytes, I can be on a lower cost node type.