Story #7477

[SDKs] Go Keep client enforces a minimum transfer rate for proxy connections

Added by Brett Smith over 5 years ago. Updated over 5 years ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


The gist of this story is "Make the Go Keep client have the same functionality the Python Keep client gets from #7235." Don't set any hard time limit for a block to be transferred to or from a Keep proxy. Instead, simply ensure that the connection keeps making progress, and fails if it stalls.


Make a wrapper for an io.Reader that returns an error if traffic rate falls below a given rate.

The returned reader should
  • Be a private type (type minProgressReader struct {...})
  • Proxy each Read() call to the underlying reader
  • Return an error if the underlying reader goes more than period seconds without returning at least minBytes bytes
  • in its Close() method, call Close() on the underlying reader if it implements io.ReadCloser, otherwise do nothing
  • if given a threshold of 0 bytes or interval of 0 seconds, do not enforce any limit at all, just pass through Read() to the underlying reader.

Should be possible to implement the timer using

Perhaps something like this

func NewMinProgressReader(r io.Reader, minBytes int, period time.Duration) io.ReadCloser {
    return &minProgressReader{
        reader:   r,
        minBytes: minBytes,
        timer:    time.NewTimer(period),

func (r *minProgressReader) Read(buf []byte) (int, error) {
    if r.err != nil {
        return 0, r.err
    var n int
    var err error
    ok := make(chan struct{})
    go func() {
        n, err = r.reader.Read(buf)
        ok <- struct{}{}
    select {
    case <-ok:
        r.haveBytes += n
        if r.haveBytes >= r.minBytes {
            r.haveBytes = 0
        return n, err
    case <-timer.C:
        r.err = ErrTransferTooSlow
        return n, r.err

Wrap the relevant readers in sdk/go/keepclient with NewMinProgressReader().

The caller should be able to adjust the #bytes and time interval by setting public fields on a KeepClient. MakeKeepClient() should initialize with reasonable defaults. The existing network timeout settings should be disabled: this will make them redundant.

The current (non-proxy) timeout default is 20 seconds for an entire block transfer, which would be expressed as minBytes==2<<26 && period==20*time.Second. Something more like minBytes==2<<20 && period==20*time.Second is probably reasonable for both proxy and non-proxy scenarios.

Related issues

Related to Arvados - Bug #7235: [SDKs] Python Keep client whole-transfer timeout should be more lenientResolved09/08/2015


#1 Updated by Brett Smith over 5 years ago

  • Target version set to Arvados Future Sprints

#2 Updated by Tom Clegg over 5 years ago

  • Description updated (diff)

Also available in: Atom PDF