Arvados » Arvados Epics

Right now, the feature of automatic HTTP download in cwl-runner is effectively fulfilling this function for users (although it copies it into the local keepstore). Users would probably like it if it were expanded to also support copying s3:// URLs.

However, the big idea for this epic is on-demand retrieval from external storage -- we fetch the data from the external system on demand.

Previous designs involved reading all the data to generate content hashes.

The current design is outlined in https://dev.arvados.org/issues/21936 and involves storing locators to external data in the manifest. The block identifiers are based on hashing the locator (and other metadata) instead of the content.