Bug #16842

[keepstore] S3 driver should handle sub-second precision in timestamps

Added by Tom Clegg about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
09/16/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Before deleting a block, in order to detect GC races, keepstore's trash worker compares the timestamp of the stored block to the timestamp provided by keep-balance in the trash list, which in turn comes from keepstore's "index" worker.

The stored block's timestamp comes from the Last-Modified header from an S3 "head object" response, which is RFC1123-formatted and therefore has no better than 1-second precision.

The index worker's timestamp comes from the LastModified XML tag in the S3 "list objects" response, which is RFC3339-formatted and potentially has a non-zero nanosecond part.

AWS and Google implementations of S3 always return zero nanoseconds in the "list objects" response, but Ceph/nautilus returns timestamps with milliseconds, which as a result will never match the RFC1123-formatted timestamp used by the trash worker.

To resolve this, the S3 index handler should truncate the nanosecond part when parsing timestamps.

keepstore (23.4 MB) keepstore 7e503a238d67b79a9f87e50086ca217d4873726e-dev (go1.14) Tom Clegg, 09/17/2020 01:47 PM
keepstore_2.0.4.dev20200917174905-1_amd64.deb (8.26 MB) keepstore_2.0.4.dev20200917174905-1_amd64.deb Ward Vandewege, 09/17/2020 06:09 PM

Subtasks

Task #16844: Review 16842-s3-timestamp-precisionResolvedWard Vandewege

Associated revisions

Revision 2a3c7865
Added by Tom Clegg about 1 year ago

Merge branch '16842-s3-timestamp-precision'

fixes #16842

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg about 1 year ago

  • Description updated (diff)
  • Subject changed from [keepstore] S3 driver trash worker should accept 1-second-precision timestamps to [keepstore] S3 driver should handle sub-second precision in timestamps

#2 Updated by Tom Clegg about 1 year ago

Aside: the rationale for truncating the index timestamps (rather than truncating before comparing in the trash worker) is to avoid a situation where keep-balance sees multiple copies with differing timestamps (say, 1234567.123 and 1234567.456) and concludes it's safe to trash one of them, but then keepstore only checks the 1234567 part, and trashes both copies.

16842-s3-timestamp-precision @ f317fc0d8e77ce950b6a650149600b0c8f6c38f3 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2100/

#3 Updated by Ward Vandewege about 1 year ago

Developer tests very unhappy/stuck, I cleared them all out and restarted them at https://ci.arvados.org/view/Developer/job/developer-run-tests/2102/, they passed finally.

#6 Updated by Anonymous about 1 year ago

  • Status changed from In Progress to Resolved

#7 Updated by Ward Vandewege about 1 year ago

Attaching a keepstore Ubuntu 18.04 package built from the 2.0-dev branch, with f317fc0d8e77ce950b6a650149600b0c8f6c38f3 cherry-picked on top.

Functionally, that means version 2.0.4 plus the fix from this ticket.

#8 Updated by Tom Clegg about 1 year ago

  • Related to Story #16360: Keep-web supports S3 compatible interface added

#9 Updated by Tom Clegg about 1 year ago

  • Related to deleted (Story #16360: Keep-web supports S3 compatible interface)

Also available in: Atom PDF