Story #16516

Run Keepstore on local compute nodes

Added by Peter Amstutz over 1 year ago. Updated 13 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
10/01/2021
Due date:
11/30/2021
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

  • new API endpoint that returns the keep volumes section of config + credentials
    • only available with a special token that combines container uuid + system root token
  • should run in separate process so that problems like OOM kill can be detected
  • specify number of memory buffers to use
    • need to be able to configure the static / dynamic keepstore instances separately
  • need to include memory that will be used by keepstore in node size request

Related issues

Related to Arvados - Feature #16519: [keepstore] optimize md5sum calculationsNew

Related to Arvados - Story #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client libraryResolved11/08/2016

Related to Arvados - Feature #16745: [keep-web] Improve performance of S3 APIs using server-side cacheResolved03/02/2021

Related to Arvados - Feature #16347: crunch-run runs local keepstoreNew

Related to Arvados - Story #17699: Design new keepstore internal architecture that supports efficient streaming & range requestsIn Progress

Related to Arvados - Feature #17749: [Keep] avoid AWS S3 request limits -- add option to use more prefixes on S3In Progress09/21/2021

Blocked by Arvados - Feature #16513: Get reference Keep performance numbers for Keep-on-S3Resolved06/15/2020

Blocked by Arvados - Feature #2960: Keep can stream GET and PUT requestsNew

Blocked by Arvados - Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstoreNew

Blocks Arvados - Bug #18101: [a-d-c] [AWS] add option to spin up (spot) instances in more/all availability zones in the regionNew

History

#1 Updated by Ward Vandewege over 1 year ago

  • Blocked by Feature #16513: Get reference Keep performance numbers for Keep-on-S3 added

#2 Updated by Ward Vandewege over 1 year ago

  • Blocked by Feature #2960: Keep can stream GET and PUT requests added

#3 Updated by Peter Amstutz over 1 year ago

  • Start date set to 06/01/2020
  • Due date set to 09/01/2020

#4 Updated by Ward Vandewege over 1 year ago

  • Blocked by Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstore added

#5 Updated by Ward Vandewege over 1 year ago

  • Related to Feature #16519: [keepstore] optimize md5sum calculations added

#6 Updated by Tom Clegg over 1 year ago

  • Related to Story #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client library added

#7 Updated by Peter Amstutz about 1 year ago

  • Start date changed from 06/01/2020 to 10/01/2020
  • Due date changed from 09/01/2020 to 12/31/2020

#8 Updated by Peter Amstutz about 1 year ago

  • Start date changed from 10/01/2020 to 01/01/2021
  • Due date changed from 12/31/2020 to 03/31/2021

#9 Updated by Peter Amstutz about 1 year ago

  • Start date changed from 01/01/2021 to 02/01/2021

#10 Updated by Peter Amstutz 10 months ago

  • Start date changed from 02/01/2021 to 08/01/2021
  • Due date changed from 03/31/2021 to 10/31/2021

#11 Updated by Peter Amstutz 10 months ago

  • Related to Feature #16745: [keep-web] Improve performance of S3 APIs using server-side cache added

#12 Updated by Peter Amstutz 8 months ago

#13 Updated by Peter Amstutz 8 months ago

  • Subject changed from Keep performance to Run Keepstore on local compute nodes

#14 Updated by Peter Amstutz 6 months ago

  • Start date changed from 08/01/2021 to 07/01/2021

#15 Updated by Peter Amstutz 4 months ago

  • Related to Story #17699: Design new keepstore internal architecture that supports efficient streaming & range requests added

#16 Updated by Ward Vandewege 4 months ago

  • Related to Feature #17749: [Keep] avoid AWS S3 request limits -- add option to use more prefixes on S3 added

#17 Updated by Peter Amstutz 4 months ago

  • Description updated (diff)

#18 Updated by Peter Amstutz 3 months ago

  • Start date changed from 07/01/2021 to 08/01/2021
  • Due date changed from 10/31/2021 to 09/30/2021

#19 Updated by Peter Amstutz about 1 month ago

  • Start date changed from 08/01/2021 to 10/01/2021
  • Due date changed from 09/30/2021 to 12/31/2021

#20 Updated by Peter Amstutz about 1 month ago

  • Start date changed from 10/01/2021 to 09/01/2021
  • Due date changed from 12/31/2021 to 11/30/2021

#21 Updated by Ward Vandewege 18 days ago

  • Blocks Bug #18101: [a-d-c] [AWS] add option to spin up (spot) instances in more/all availability zones in the region added

#22 Updated by Peter Amstutz 13 days ago

  • Start date changed from 09/01/2021 to 10/01/2021

Also available in: Atom PDF