Arvados

We could also consider naming the mount dirs according to purpose instead of content -- e.g., mount at "/mnt/bamFile" instead of "/keep/d41d8cd98f00b204e9800998ecf8427e+0".

That might also make container logs easier to read: "/mnt/tumor/sample1234.bam" and "/mnt/normal/sample5678.bam" instead of "/keep/pdh78af66a/sample1234.bam" and "/keep/pdh83b2f9a/sample5678.bam".

[Tom replied after I'd composed this reply, but before I submitted it. I like this idea, but still think the questions below are valid.]

Am I wrong in thinking that any change to the "containers API" automatically invalidates all jobs which have been run to-date from a reusability point of view?

Can you expand on the pros and cons that you perceive for such a change?

Tom Clegg wrote:

We could also consider naming the mount dirs according to purpose instead of content -- e.g., mount at "/mnt/bamFile" instead of "/keep/d41d8cd98f00b204e9800998ecf8427e+0".

That might also make container logs easier to read: "/mnt/tumor/sample1234.bam" and "/mnt/normal/sample5678.bam" instead of "/keep/pdh78af66a/sample1234.bam" and "/keep/pdh83b2f9a/sample5678.bam".

Yes, we could take the input parameter name into account when determining the mount path. However it would require a bit of work since that information currently isn't easily available to the part of the code that decides where to mount things.

Tom Morris wrote:

[Tom replied after I'd composed this reply, but before I submitted it. I like this idea, but still think the questions below are valid.]

Am I wrong in thinking that any change to the "containers API" automatically invalidates all jobs which have been run to-date from a reusability point of view?

You are not wrong. This would change mount points which would invalidate job reuse.

Can you expand on the pros and cons that you perceive for such a change?

Pros: solves a user problem with a commonly-used tool.

Cons: accommodating badly-behaved tools is a slippery slope, latest version of tool doesn't have the problem, reasonable workarounds (e.g. putting the code in Docker instead of Keep) are available.

Just a thought: if it's too much work under the hood to use symbolic names like "bamFile", we could consider "/mnt/d41d8cd9".

(Having "-" and "+" PDH forms in various places seems like a confusing road to go down. Keep-web is forced into it because it really needs to communicate the whole PDH, but in this case we don't actually need it to be a PDH at all: it just has to be deterministic so re-use works, and it's best if it doesn't make the logs too hard for a human to read.)