Idea #11016
closedDocument how to choose a suitable blob signature TTL
Related issues
Updated by Tom Morris almost 7 years ago
- Target version set to Arvados Future Sprints
Updated by Nico César about 5 years ago
I am re-using this ticket, since we had a lot of inquiries about this. This is an example of it:
We are trying to delete data from the keepstore. But after we run "arv collection delete --uuid=zzzzz-4zz18-7p9s7j1qa" it trashes it, but sets the final delete date 2 weeks in the future.
I ask because on our test server we have filled up the keepstore and can't delete data to continue testing. It's put a stop to the project.
"delete_at":"2019-07-11T20:45:30.557593000Z", "trash_at":"2019-06-27T20:45:30.557593000Z", "is_trashed":true,Even if we change the trash times in " /etc/arvados/keepstore/keepstore.yml" and reinstall keepstore + restart the service.
\# How often to check for (and delete) trashed blocks whose
# TrashLifetime has expired. TrashCheckInterval: 1h0m0s # Time duration after a block is trashed during which it can be # recovered using an /untrash request. TrashLifetime: 1h0m0s
this was my answer:
Just to make sure we are all in the same page in terms of terminology, there is a good explanation here: https://doc.arvados.org/user/tutorials/tutorial-keep-collection-lifecycle.html
And the method you used is "delete" in collections, from https://doc.arvados.org/v1.4/api/methods/collections.html
----------
deletePut a Collection in the trash. This sets the trash_at field to now and delete_at field to now + token TTL. A trashed collection is invisible to most API calls unless the include_trash parameter is true.
-----------As you can see the "token TTL" expressed there is set by default in 2 weeks, this is the Collections->BlobSigningTTL and Collections->DefaultThrashLifetime parameters in the configuration. Here is description from in https://doc.arvados.org/v1.4/admin/config.html
------- # Lifetime (in seconds) of blob permission signatures generated by # the API server. This determines how long a client can take (after # retrieving a collection record) to retrieve the collection data # from Keep. If the client needs more time than that (assuming the # collection still has the same content and the relevant user/token # still has permission) the client can retrieve the collection again # to get fresh signatures. # # This must be exactly equal to the -blob-signature-ttl flag used by # keepstore servers. Otherwise, reading data blocks and saving # collections will fail with HTTP 403 permission errors. # # Modifying blob_signature_ttl invalidates existing signatures; see # blob_signing_key note above. # # The default is 2 weeks. BlobSigningTTL: 336h # Default lifetime for ephemeral collections: 2 weeks. This must not # be less than blob_signature_ttl. DefaultTrashLifetime: 336h ------This assumes that you have the central configuration in /etc/arvados/config.yml, keep-balance.service up and running.
As you can see we have 3 different places with pieces of the information. And usually our test server we have filled up the keepstore is the reason they need a quick "delete all this" process without having to wait 2 weeks.
Updated by Tom Morris about 5 years ago
One of the main things that's missing from https://doc.arvados.org/user/tutorials/tutorial-keep-collection-lifecycle.html (which is really conceptual documentation, not a tutorial)
is "when do I get my disk space back?" and the associated Keep store pieces of the data lifecycle.
Updated by Tom Morris about 5 years ago
- Target version changed from Arvados Future Sprints to 2019-07-31 Sprint
Updated by Tom Clegg about 5 years ago
- Target version changed from 2019-07-31 Sprint to 2019-08-14 Sprint
Updated by Tom Clegg about 5 years ago
- Status changed from New to In Progress
- A data block must be written to a disk/cloud backend device before a collection can be created/updated with a reference to it.
- Deleting or updating a collection can have the effect of removing the last remaining reference to a data block.
If BlobSigningTTL is too short, long-running processes/containers will fail when they take too long (a) between writing blocks and writing collections that reference them, or (b) between reading collections and reading the referenced blocks.
If BlobSigningTTL is too long, data will still be stored long after the referring collections are deleted, and you will needlessly fill up disks or waste money on cloud storage.
Updated by Tom Clegg about 5 years ago
- Target version changed from 2019-08-14 Sprint to 2019-08-28 Sprint
Updated by Tom Morris almost 5 years ago
- Target version changed from 2019-08-28 Sprint to 2019-09-11 Sprint
Updated by Tom Clegg almost 5 years ago
- Target version changed from 2019-09-11 Sprint to 2019-09-25 Sprint
Updated by Tom Clegg almost 5 years ago
- Target version changed from 2019-09-25 Sprint to 2019-10-09 Sprint
Updated by Tom Clegg almost 5 years ago
11016-doc-signing-ttl @ f902c301c8c40707bb25f8dae01537f8a50be286 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1571/
Updated by Tom Morris almost 5 years ago
- add "(ie Time-To-Live)" after "BlobSigningTTL determines the minimum lifetime" to help explain TTL. I'm not sure it's a familiar term outside of the networking space.
- it seems like there are pieces of the lifecycle for blocks and signed block references missing from https://doc.arvados.org/user/tutorials/tutorial-keep-collection-lifecycle.html unless we have them covered somewhere else (ie the two cases mentioned in the config file description)
Updated by Tom Clegg almost 5 years ago
- Related to Idea #15697: [doc] explain lifecycle of Keep blocks, and how it affects storage backend usage/cost added
Updated by Tom Clegg almost 5 years ago
- Status changed from In Progress to Resolved
Updated by Peter Amstutz over 4 years ago
- Release set to 22
- Status changed from Resolved to Rejected
Updated by Peter Amstutz over 4 years ago
- Status changed from Rejected to Resolved