Project

General

Profile

Actions

Idea #3762

closed

[Keep] Process entries on the current "delete" list.

Added by Tom Clegg over 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
Keep
Target version:
Start date:
07/29/2014
Due date:
Story points:
2.0

Description

Keep should process the file asynchronously and delete each block in it.

This is written out in more detail at the bottom of:
https://arvados.org/projects/orvos-private/wiki/Keep_Design_Doc#Keep-Server

Notes:
  • For now, begin processing the trash list right away. We can defer to a future sprint how Keep should decide when to start recycling blocks.
  • If a block's actual age is less than (some delta) it should not be deleted. Need to define how this threshold is chosen. (Ideally this is defaultTrashLifetime from the discovery doc, but a command line flag would be OK.) "Some delta" is the value of the existing -permission-ttl flag.
  • If a new list is received while the old one is being processed, abandon the existing one, replace it with the new one, and start over. (WorkQueue already takes care of this.)

Subtasks 2 (0 open2 closed)

Task #5537: Review 3762-delete-list-workerResolvedRadhika Chippada07/29/2014Actions
Task #3682: Implement TrashCollectorResolvedRadhika Chippada07/29/2014Actions

Related issues

Blocked by Arvados - Idea #3413: [Keep] Keep Server accepts list of trashable blocksResolvedTim Pierce07/29/2014Actions
Actions #1

Updated by Tom Clegg over 9 years ago

  • Description updated (diff)
  • Category set to Keep
Actions #2

Updated by Tom Clegg about 9 years ago

  • Target version changed from Arvados Future Sprints to 2015-03-11 sprint
Actions #3

Updated by Tom Clegg about 9 years ago

  • Target version changed from 2015-03-11 sprint to Arvados Future Sprints
Actions #4

Updated by Tom Clegg about 9 years ago

  • Target version changed from Arvados Future Sprints to 2015-04-01 sprint
Actions #5

Updated by Radhika Chippada about 9 years ago

  • Assigned To set to Radhika Chippada
Actions #6

Updated by Tom Clegg about 9 years ago

  • Description updated (diff)
Actions #7

Updated by Radhika Chippada about 9 years ago

  • Status changed from New to In Progress
Actions #8

Updated by Peter Amstutz about 9 years ago

Reviewing 3762-delete-list-worker fed6d3b

The cached discovery document should part of the ArvadosClient struct, not the DISCOVERY global variable. You should probably fetch the discovery document in MakeArvadosClient().

If you want to return "any type" in Go, you use interface{}. Then Discovery() does not need to create valueMap just to return as single item.

I just realized that having the trashWorker fetch "defaultTrashLifetime" from the discovery document means that we are re-introducing a dependency on the API server that we had just worked to eliminate with the pullWorker. We should consult tom/ward on their preference.

I noticed this in both pullq and trashq:

    if pullq == nil {
        pullq = NewWorkQueue()
    }

trashq and pullq should never be nil, and if they are nil, allocating a new work queue won't do anything because there won't be a worker goroutine associated with it. These tests should be taken out or converted into a assertions that raise an error.

Actions #9

Updated by Tom Clegg about 9 years ago

  • Description updated (diff)
Actions #10

Updated by Radhika Chippada about 9 years ago

  • Removed all the new code for API discovery and instead using permission_ttl. Per our discussion with Tom, defaultTrashLifetime is the same as permission_ttl, which is already available in keepstore and this should be used.
  • Removed "if pullq == nil ... " from handlers.go and updated the handler_test instead to create the instances of pullq and trashq.
Actions #11

Updated by Peter Amstutz about 9 years ago

LGTM

Actions #12

Updated by Radhika Chippada about 9 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados|commit:0d1971f04c8f0674ff8d94bae5dbb7f6b94e7f5f.

Actions

Also available in: Atom PDF