Writable FUSE mount¶

Problem¶

When running a job in order to write output to Keep it is necessary to either a) use the Keep API in the Python SDK or b) write to a staging directory and then use arv-put to upload the contents. Both of these have disadvantages; using the Python API directly is only practical if the user script itself is written in Python and can easily accommodate writing its output to Keep (unfortunately arvados.StreamFileReader does not conform to the Python File object interface and cannot be used as a drop-in replacement.) Writing to a staging directory requires sufficient scratch space on the compute node, and cannot stream the output to Keep servers as it is being written, instead requiring a potentially time consuming upload step after the actual job is finished.

Goals¶

• Accommodate the vast majority of scripts and tools that naturally write their output as regular files
• Improve performance by asynchronously streaming blocks to Keep as they are written, instead of waiting until the job is finished

Proposal¶

We already support a read-only fuse mount; this has proven to be a key piece of infrastructure that greatly simplifies