Project

General

Profile

Actions

Story #16516

closed

Run Keepstore on local compute nodes

Added by Peter Amstutz about 2 years ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
10/01/2021
Due date:
11/30/2021
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

  • new API endpoint that returns the keep volumes section of config + credentials
    • only available with a special token that combines container uuid + system root token
  • should run in separate process so that problems like OOM kill can be detected
  • specify number of memory buffers to use
    • need to be able to configure the static / dynamic keepstore instances separately
  • need to include memory that will be used by keepstore in node size request

Related issues

Related to Arvados - Story #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client libraryResolvedWard Vandewege11/08/2016

Actions
Related to Arvados - Feature #16745: [keep-web] Improve performance of S3 APIs using server-side cacheResolvedTom Clegg03/02/2021

Actions
Related to Arvados - Feature #16347: crunch-run runs local keepstoreResolvedTom Clegg10/08/2021

Actions
Related to Arvados - Feature #17749: [Keep] avoid AWS S3 request limits -- add option to use more prefixes on S3ResolvedTom Clegg09/21/2021

Actions
Related to Arvados - Feature #18216: Refactor keepstore to be part of arvados-serverResolvedTom Clegg09/30/2021

Actions
Blocked by Arvados - Feature #16513: Get reference Keep performance numbers for Keep-on-S3ResolvedWard Vandewege06/15/2020

Actions
Actions #1

Updated by Ward Vandewege about 2 years ago

  • Blocked by Feature #16513: Get reference Keep performance numbers for Keep-on-S3 added
Actions #2

Updated by Ward Vandewege about 2 years ago

  • Blocked by Feature #2960: Keep can stream GET and PUT requests added
Actions #3

Updated by Peter Amstutz about 2 years ago

  • Start date set to 06/01/2020
  • Due date set to 09/01/2020
Actions #4

Updated by Ward Vandewege about 2 years ago

  • Blocked by Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstore added
Actions #5

Updated by Ward Vandewege about 2 years ago

  • Related to Feature #16519: [keepstore] optimize md5sum calculations added
Actions #6

Updated by Tom Clegg about 2 years ago

  • Related to Story #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client library added
Actions #7

Updated by Peter Amstutz almost 2 years ago

  • Start date changed from 06/01/2020 to 10/01/2020
  • Due date changed from 09/01/2020 to 12/31/2020
Actions #8

Updated by Peter Amstutz almost 2 years ago

  • Start date changed from 10/01/2020 to 01/01/2021
  • Due date changed from 12/31/2020 to 03/31/2021
Actions #9

Updated by Peter Amstutz almost 2 years ago

  • Start date changed from 01/01/2021 to 02/01/2021
Actions #10

Updated by Peter Amstutz over 1 year ago

  • Start date changed from 02/01/2021 to 08/01/2021
  • Due date changed from 03/31/2021 to 10/31/2021
Actions #11

Updated by Peter Amstutz over 1 year ago

  • Related to Feature #16745: [keep-web] Improve performance of S3 APIs using server-side cache added
Actions #12

Updated by Peter Amstutz over 1 year ago

Actions #13

Updated by Peter Amstutz over 1 year ago

  • Subject changed from Keep performance to Run Keepstore on local compute nodes
Actions #14

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 08/01/2021 to 07/01/2021
Actions #15

Updated by Peter Amstutz about 1 year ago

  • Related to Story #17699: Design new keepstore internal architecture that supports efficient streaming & range requests added
Actions #16

Updated by Ward Vandewege about 1 year ago

  • Related to Feature #17749: [Keep] avoid AWS S3 request limits -- add option to use more prefixes on S3 added
Actions #17

Updated by Peter Amstutz about 1 year ago

  • Description updated (diff)
Actions #18

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 07/01/2021 to 08/01/2021
  • Due date changed from 10/31/2021 to 09/30/2021
Actions #19

Updated by Peter Amstutz 11 months ago

  • Start date changed from 08/01/2021 to 10/01/2021
  • Due date changed from 09/30/2021 to 12/31/2021
Actions #20

Updated by Peter Amstutz 11 months ago

  • Start date changed from 10/01/2021 to 09/01/2021
  • Due date changed from 12/31/2021 to 11/30/2021
Actions #21

Updated by Ward Vandewege 10 months ago

  • Blocks Bug #18101: [a-d-c] [AWS] add option to spin up (spot) instances in more/all availability zones in the region added
Actions #22

Updated by Peter Amstutz 10 months ago

  • Start date changed from 09/01/2021 to 10/01/2021
Actions #23

Updated by Peter Amstutz 9 months ago

  • Related to Feature #18216: Refactor keepstore to be part of arvados-server added
Actions #24

Updated by Peter Amstutz 9 months ago

  • Status changed from New to In Progress
Actions #25

Updated by Peter Amstutz 8 months ago

  • Related to deleted (Story #17699: Design new keepstore internal architecture that supports efficient streaming & range requests)
Actions #26

Updated by Peter Amstutz 7 months ago

  • Related to deleted (Feature #16519: [keepstore] optimize md5sum calculations)
Actions #27

Updated by Peter Amstutz 7 months ago

  • Blocked by deleted (Feature #2960: Keep can stream GET and PUT requests)
Actions #28

Updated by Peter Amstutz 7 months ago

  • Blocked by deleted (Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstore)
Actions #29

Updated by Peter Amstutz 7 months ago

  • Blocks deleted (Bug #18101: [a-d-c] [AWS] add option to spin up (spot) instances in more/all availability zones in the region)
Actions #30

Updated by Peter Amstutz 7 months ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF