Idea #10996
Updated by Tom Clegg about 7 years ago
h3. Background
In sdk/go/keepclient, @(*keepclient.KeepClient)CollectionFileReader(collection map[string]interface{}, filename string)@ returns a keepclient.Reader, which implements io.Reader, io.Seeker, and io.Closer but not io.Writer.
In services/crunch-run, @(*CollectionWriter).Open(path string)@ returns an io.WriteCloser, i.e., it can write but not read or seek.
h3. Improvement
In sdk/go/arvados, a Collection should have Open, Create, and OpenFile methods that correspond to os.Open, os.Create, and os.OpenFile.
Open returns a read-only file, so it can be implemented by calling CollectionFileReader.
For now, As a starting point, Create and OpenFile can be implemented by refactoring the CollectionWriter bits into the SDK, but the returned object would not be readable or seekable. See #11249.
Then the reading+seeking and writing code need to be reconciled so OpenFile and Create return an object that can read, write, and seek.
h3. Optimization ideas
Presumably not implemented at the same time, but can be kept in mind so we don't burn too many bridges:
* BlockCache could use AsyncStream, e.g. 10990-keep-web-ranges-pa
* Could cache smaller portions of blocks