[keepstore] S3 driver should handle sub-second precision in timestamps
Before deleting a block, in order to detect GC races, keepstore's trash worker compares the timestamp of the stored block to the timestamp provided by keep-balance in the trash list, which in turn comes from keepstore's "index" worker.
The stored block's timestamp comes from the Last-Modified header from an S3 "head object" response, which is RFC1123-formatted and therefore has no better than 1-second precision.
The index worker's timestamp comes from the LastModified XML tag in the S3 "list objects" response, which is RFC3339-formatted and potentially has a non-zero nanosecond part.
AWS and Google implementations of S3 always return zero nanoseconds in the "list objects" response, but Ceph/nautilus returns timestamps with milliseconds, which as a result will never match the RFC1123-formatted timestamp used by the trash worker.
To resolve this, the S3 index handler should truncate the nanosecond part when parsing timestamps.
#2 Updated by Tom Clegg about 1 year ago
Aside: the rationale for truncating the index timestamps (rather than truncating before comparing in the trash worker) is to avoid a situation where keep-balance sees multiple copies with differing timestamps (say, 1234567.123 and 1234567.456) and concludes it's safe to trash one of them, but then keepstore only checks the 1234567 part, and trashes both copies.
16842-s3-timestamp-precision @ f317fc0d8e77ce950b6a650149600b0c8f6c38f3 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2100/
#3 Updated by Ward Vandewege about 1 year ago
Developer tests very unhappy/stuck, I cleared them all out and restarted them at https://ci.arvados.org/view/Developer/job/developer-run-tests/2102/, they passed finally.
#5 Updated by Ward Vandewege about 1 year ago
Tom Clegg wrote:
16842-s3-timestamp-precision @ 7e503a238d67b79a9f87e50086ca217d4873726e -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2103/
#7 Updated by Ward Vandewege about 1 year ago
- File keepstore_2.0.4.dev20200917174905-1_amd64.deb keepstore_2.0.4.dev20200917174905-1_amd64.deb added
Attaching a keepstore Ubuntu 18.04 package built from the 2.0-dev branch, with f317fc0d8e77ce950b6a650149600b0c8f6c38f3 cherry-picked on top.
Functionally, that means version 2.0.4 plus the fix from this ticket.