Story #7477

[SDKs] Go Keep client enforces a minimum transfer rate for proxy connections

Added by Brett Smith over 3 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
SDKs
Target version:
Start date:
10/07/2015
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

The gist of this story is "Make the Go Keep client have the same functionality the Python Keep client gets from #7235." Don't set any hard time limit for a block to be transferred to or from a Keep proxy. Instead, simply ensure that the connection keeps making progress, and fails if it stalls.

Implementation

Make a wrapper for an io.Reader that returns an error if traffic rate falls below a given rate.

The returned reader should
  • Be a private type (type minProgressReader struct {...})
  • Proxy each Read() call to the underlying reader
  • Return an error if the underlying reader goes more than period seconds without returning at least minBytes bytes
  • in its Close() method, call Close() on the underlying reader if it implements io.ReadCloser, otherwise do nothing
  • if given a threshold of 0 bytes or interval of 0 seconds, do not enforce any limit at all, just pass through Read() to the underlying reader.

Should be possible to implement the timer using https://golang.org/pkg/time/#NewTimer

Perhaps something like this

func NewMinProgressReader(r io.Reader, minBytes int, period time.Duration) io.ReadCloser {
    return &minProgressReader{
        reader:   r,
        minBytes: minBytes,
        timer:    time.NewTimer(period),
    }
}

func (r *minProgressReader) Read(buf []byte) (int, error) {
    if r.err != nil {
        return 0, r.err
    }
    var n int
    var err error
    ok := make(chan struct{})
    go func() {
        n, err = r.reader.Read(buf)
        ok <- struct{}{}
    }()
    select {
    case <-ok:
        r.haveBytes += n
        if r.haveBytes >= r.minBytes {
            r.timer.Reset()
            r.haveBytes = 0
        }
        return n, err
    case <-timer.C:
        r.Close()
        r.err = ErrTransferTooSlow
        return n, r.err
    }
}

Wrap the relevant readers in sdk/go/keepclient with NewMinProgressReader().

The caller should be able to adjust the #bytes and time interval by setting public fields on a KeepClient. MakeKeepClient() should initialize with reasonable defaults. The existing network timeout settings should be disabled: this will make them redundant.

The current (non-proxy) timeout default is 20 seconds for an entire block transfer, which would be expressed as minBytes==2<<26 && period==20*time.Second. Something more like minBytes==2<<20 && period==20*time.Second is probably reasonable for both proxy and non-proxy scenarios.


Related issues

Related to Arvados - Bug #7235: [SDKs] Python Keep client whole-transfer timeout should be more lenientResolved09/08/2015

History

#1 Updated by Brett Smith over 3 years ago

  • Target version set to Arvados Future Sprints

#2 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

Also available in: Atom PDF