Feature #3145

[Keep] keepstore supports read-only flag for each volume, for cases like full, draining, mounted read-only

Added by Peter Amstutz over 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
04/15/2015
Due date:
% Done:

100%

Estimated time:
(Total: 6.00 h)
Story points:
0.5

Description

When all volumes are flagged read-only, all PUT and DELETE requests should be rejected immediately.

There should also be a convenient command line option to put keepstored into this readonly mode, regardless of the state of its volumes.


Subtasks

Task #5746: Accept readonly flag in -volumes=... argumentResolvedTom Clegg

Task #5747: Set readonly flag from mount opts in FindKeepVolumesResolvedTom Clegg

Task #5750: Add testsResolvedTom Clegg

Task #5751: Review 3145-readonly-volumesResolvedTom Clegg

Task #3844: Agressively close PUT connections when there are no writable volumes.ResolvedTom Clegg

Task #5869: Review 3145-missing-content-length-errorResolvedPeter Amstutz

Task #5849: Review 3145-close-earlyResolvedPeter Amstutz


Related issues

Has duplicate Arvados - Feature #4716: [Keep] Keepstore has read-only mode (deny all PUT or DELETE)Rejected

Associated revisions

Revision 4d078362
Added by Tom Clegg over 6 years ago

Merge branch '3145-readonly-volumes' refs #3145

Revision 624c5c8d
Added by Tom Clegg over 6 years ago

Merge branch '3145-close-early' closes #3145

Revision 7990a777
Added by Peter Amstutz over 6 years ago

Merge branch '3145-missing-content-length-error' closes #3145

History

#1 Updated by Tom Clegg over 7 years ago

  • Tracker changed from Bug to Feature
  • Subject changed from Keep server tries to write to read-only volumes every time. to Keep blob server supports read-only flag for each volume, for cases like full, draining, mounted read-only
  • Story points set to 1.0

#2 Updated by Tom Clegg over 7 years ago

  • Target version set to Arvados Future Sprints

#3 Updated by Tom Clegg over 7 years ago

  • Subject changed from Keep blob server supports read-only flag for each volume, for cases like full, draining, mounted read-only to [Keep] blob server supports read-only flag for each volume, for cases like full, draining, mounted read-only
  • Category set to Keep

#4 Updated by Tom Clegg almost 7 years ago

  • Description updated (diff)

#5 Updated by Ward Vandewege almost 7 years ago

  • Description updated (diff)

#6 Updated by Tom Clegg over 6 years ago

  • Subject changed from [Keep] blob server supports read-only flag for each volume, for cases like full, draining, mounted read-only to [Keep] keepstore supports read-only flag for each volume, for cases like full, draining, mounted read-only

#7 Updated by Tom Clegg over 6 years ago

  • Status changed from New to In Progress
  • Assigned To set to Tom Clegg
  • Target version changed from Arvados Future Sprints to 2015-04-29 sprint

#8 Updated by Tom Clegg over 6 years ago

41caa50 on 3145-readonly-volumes

#9 Updated by Peter Amstutz over 6 years ago

3145-readonly-volumes LGTM

#10 Updated by Peter Amstutz over 6 years ago

3145-close-early LGTM

We should use pycurl.UPLOAD in KeepClient (refs #5562)

http://curl.haxx.se/libcurl/c/CURLOPT_UPLOAD.html

This sets the method to PUT and sets the "Expect: 100-continue" header.

The "100 Continue" flow means the client is not supposed to send the request body until the server sends a "100 Continue" response. This gives the server a chance to check whether that the request can reasonably be expected to succeed, then either tells the client to continue or sends a final error.

From reading the Go net/http code, Go supports "100 Continue" transparently by automatically sending the "100 Continue" response when the handler starts reading the request body (but not before). So your branch requires no additional changes to support "Expect: 100-continue" (awesome!)

The other benefit of 100 continue is to open the possibility of serializing requests from a given host, so if a host with a lot of cores is trying to send a lot of data from a lot of processes, we can use "100 Continue" with cooperative clients to limit upload to 2-3 simultaneous senders and reduce network stack congestion.

#11 Updated by Tom Clegg over 6 years ago

Peter Amstutz wrote:

3145-close-early LGTM

We should use pycurl.UPLOAD in KeepClient (refs #5562)

http://curl.haxx.se/libcurl/c/CURLOPT_UPLOAD.html

This sets the method to PUT and sets the "Expect: 100-continue" header.

More precisely, that page says:
  • uploading means using the PUT request unless you tell libcurl otherwise.
  • Using PUT with HTTP 1.1 implies the use of a "Expect: 100-continue" header.

We're already using PUT with HTTP 1.1, therefore we are already using the Expect header. I haven't tcpdumped this but that's my reading of the docs.

And yes, Go made it really easy to do the server side of this: just don't read the body if you don't want it!

#12 Updated by Tom Clegg over 6 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:624c5c8d13d6c2e21a80379928387944bedae2a3.

#13 Updated by Peter Amstutz over 6 years ago

  • Status changed from Resolved to In Progress
  • Target version changed from 2015-04-29 sprint to 2015-05-20 sprint
  • Story points changed from 1.0 to 0.5

#14 Updated by Peter Amstutz over 6 years ago

  • Status changed from In Progress to Feedback

#15 Updated by Tom Clegg over 6 years ago

e94da792 lgtm, thanks.

Nice work on the HTTP status codes that are already defined to mean exactly what we want to say! Way better than "504 Gateway Timeout" as a "block too big" response. I looked up the >1 year history of this one and my favorite part was my own change, "3220: Fix HTTP status codes":

-       TooLongError    = &KeepError{504, "Too Long"}
+       TooLongError    = &KeepError{504, "Timeout"}

#16 Updated by Peter Amstutz over 6 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 86 to 100

Applied in changeset arvados|commit:7990a7778f70c243b50ea878787ea83689f5b07e.

Also available in: Atom PDF