Arvados

Background¶

In sdk/go/keepclient, (*keepclient.KeepClient)CollectionFileReader(collection map[string]interface{}, filename string) returns a keepclient.Reader, which implements io.Reader, io.Seeker, and io.Closer but not io.Writer.

In services/crunch-run, (*CollectionWriter).Open(path string) returns an io.WriteCloser, i.e., it can write but not read or seek.

Improvement¶

In sdk/go/arvados, a Collection should have Open, Create, and OpenFile methods that correspond to os.Open, os.Create, and os.OpenFile.

Open returns a read-only file, so it can be implemented by calling CollectionFileReader.

For now, Create and OpenFile can be implemented by refactoring the CollectionWriter bits into the SDK, but the returned object would not be readable or seekable. See #11249.

Optimization ideas¶

Presumably not implemented at the same time, but can be kept in mind so we don't burn too many bridges:

• BlockCache could use AsyncStream, e.g. 10990-keep-web-ranges-pa
• Could cache smaller portions of blocks