Story #3762

[Keep] Process entries on the current "delete" list.

Added by Tom Clegg over 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
Keep
Target version:
Start date:
07/29/2014
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
2.0

Description

Keep should process the file asynchronously and delete each block in it.

This is written out in more detail at the bottom of:
https://arvados.org/projects/orvos-private/wiki/Keep_Design_Doc#Keep-Server

Notes:
  • For now, begin processing the trash list right away. We can defer to a future sprint how Keep should decide when to start recycling blocks.
  • If a block's actual age is less than (some delta) it should not be deleted. Need to define how this threshold is chosen. (Ideally this is defaultTrashLifetime from the discovery doc, but a command line flag would be OK.) "Some delta" is the value of the existing -permission-ttl flag.
  • If a new list is received while the old one is being processed, abandon the existing one, replace it with the new one, and start over. (WorkQueue already takes care of this.)

Subtasks

Task #5537: Review 3762-delete-list-workerResolvedRadhika Chippada

Task #3682: Implement TrashCollectorResolvedRadhika Chippada


Related issues

Blocked by Arvados - Story #3413: [Keep] Keep Server accepts list of trashable blocksResolved07/29/2014

Associated revisions

Revision 0d1971f0
Added by Radhika Chippada over 5 years ago

closes #3762
Merge branch '3762-delete-list-worker'

History

#1 Updated by Tom Clegg over 6 years ago

  • Description updated (diff)
  • Category set to Keep

#2 Updated by Tom Clegg almost 6 years ago

  • Target version changed from Arvados Future Sprints to 2015-03-11 sprint

#3 Updated by Tom Clegg almost 6 years ago

  • Target version changed from 2015-03-11 sprint to Arvados Future Sprints

#4 Updated by Tom Clegg over 5 years ago

  • Target version changed from Arvados Future Sprints to 2015-04-01 sprint

#5 Updated by Radhika Chippada over 5 years ago

  • Assigned To set to Radhika Chippada

#6 Updated by Tom Clegg over 5 years ago

  • Description updated (diff)

#7 Updated by Radhika Chippada over 5 years ago

  • Status changed from New to In Progress

#8 Updated by Peter Amstutz over 5 years ago

Reviewing 3762-delete-list-worker fed6d3b

The cached discovery document should part of the ArvadosClient struct, not the DISCOVERY global variable. You should probably fetch the discovery document in MakeArvadosClient().

If you want to return "any type" in Go, you use interface{}. Then Discovery() does not need to create valueMap just to return as single item.

I just realized that having the trashWorker fetch "defaultTrashLifetime" from the discovery document means that we are re-introducing a dependency on the API server that we had just worked to eliminate with the pullWorker. We should consult tom/ward on their preference.

I noticed this in both pullq and trashq:

    if pullq == nil {
        pullq = NewWorkQueue()
    }

trashq and pullq should never be nil, and if they are nil, allocating a new work queue won't do anything because there won't be a worker goroutine associated with it. These tests should be taken out or converted into a assertions that raise an error.

#9 Updated by Tom Clegg over 5 years ago

  • Description updated (diff)

#10 Updated by Radhika Chippada over 5 years ago

  • Removed all the new code for API discovery and instead using permission_ttl. Per our discussion with Tom, defaultTrashLifetime is the same as permission_ttl, which is already available in keepstore and this should be used.
  • Removed "if pullq == nil ... " from handlers.go and updated the handler_test instead to create the instances of pullq and trashq.

#11 Updated by Peter Amstutz over 5 years ago

LGTM

#12 Updated by Radhika Chippada over 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados|commit:0d1971f04c8f0674ff8d94bae5dbb7f6b94e7f5f.

Also available in: Atom PDF