Arvados

h1. S3 bucket volume implementation

{{toc}}

h2. Background - unsafe delete

With the current (2016-07-04) S3 volume implementation, an S3 volume can store blocks safely, but cannot do garbage collection safely.

There is a command line flag @-s3-unsafe-delete@ that enables unsafe garbage collection. With this setting, it is possible to lose data: if a client writes a new copy of a previously stored block that is now scheduled for deletion, the newly written copy can be deleted even though the client is told the data is safely stored.

h2. S3 API challenge

S3 offers eventual consistency. Specifically:

* Reading an object doesn't necessarily return the latest version: "write X=foo; write X=bar; read X" might return foo.

* Deleting is not immediate: "write X=foo; delete X; read X" might return foo.

* There is no atomic conditional write operation (e.g., "overwrite X only if X has etag=foo").

This makes it hard to do garbage collection. How can we delete X if we don't know whether X has been written recently?

h2. Implementation draft

Proposed approach: Each block is represented by (up to) three S3 objects.

* Data object, name="{md5}" -- contains the stored data; timestamp and metadata are irrelevant.

* Mtime object, name="recent/{md5}" -- zero data bytes; Last-Modified reflects most recent Put or Touch.

* Trash object, name="trash/{md5}" -- contains a copy of the stored data; Last-Modified reflects most recent Trash.

Put(X):

# Write data to data object, "X"

# Write zero bytes to mtime object, "recent/X"

Touch(X):

# Write zero bytes to mtime object, "recent/X"

# Ensure data object "X" exists. If not:

#* If FixRace(X) returns false, return ErrNotExist

#* Else return nil (OK)

#* (Handlers only use Touch after Get succeeds, so a 404 here is almost certainly an early opportunity to catch and correct a touch-vs.-trash race.)

Mtime(X):

# Ensure data object "X" exists. If not, return ErrNotExist.

# Ensure "recent/X" exists. If not, create it.

# Return LastModified("recent/X")

Trash(X):

# If trashLifetime==0 && !unsafeDelete: "not implemented" error

# Get "recent/X"

# PutCopy("X", "trash/X")

# Del("X")

Get(X):

# If Get("X") succeeds, return content

# Else if Get returned a 404, and FixRace(X) returns true, call Get("X") again and return content

FixRace(X): "check whether X needs to be cleaned up after a race; if so, clean it up. Return true if X was untrashed during cleanup."

# Use HEAD to get LastModified("recent/X").

# If "recent/X" does not exist, an emptyTrash/Trash race went wrong, so we need to fix it by saving the very conservative value "now" as "most recent Put or Touch":

#* Put("recent/X", zeroBytes)

#* Log this event.

# Use HEAD to get LastModified("trash/X"). If "trash/X" does not exist, return false.

# If LastModified("trash/X") < LastModified("recent/X") + permissionTTL, a touch/trash race went wrong, so we need to fix it:

#* Perform HEAD requests for "X" and "trash/X"

#* If "X" exists but etag differs from "trash/X", log a collision warning and do nothing

#* If "X" exists and etag matches, delete "trash/X"

#* If "X" does not exist, PutCopy("trash/X", "X") (but don't delete "trash/X" -- EmptyTrash will get it next time)

#* Log this event.

#* Return true.

# Return false.

Untrash(X):

# PutCopy("trash/X", "X") (but don't delete "trash/X" -- EmptyTrash will get it next time)

EmptyTrash():

# List all trash objects and mtime objects ("trash/X" and "recent/X"), using a merge sort to get them in matching pairs. For each X...

# If LastModified("trash/X") < LastModified("recent/X") + permissionTTL: call FixRace(X)

# Else if now() > LastModified("trash/X") + trashLifetime, it's time to really delete this object:

#* Del("trash/X")

#* Del("recent/X")

Index():

# List all data objects and mtime objects ("X" and "recent/X"), using a merge sort to get them in matching pairs.

# Include X in the index only if "X" *and* "recent/X" exist.

# In the index response row "hash+size timestamp": hash is X, size is Size("X"), timestamp is LastModified("recent/X")

Sometimes, races will cause valuable blocks to be trashed, but they will be recovered automatically by FixRace(), _provided the S3 service achieves eventual consistency faster than trashLifetime._

These "accidentally trashed" blocks will be omitted from the index, which means keep-balance will occasionally report them as "lost". This is undesirable, because never cry wolf.

h2. Unresolved issues

If both X and trash/X exist (e.g., a client has written a copy of a block that was already in trash), a race between Trash and FixRace can delete both X and trash/X (see #8555#note-21).