Project

General

Profile

Actions
Copy link

Idea #16516

closed

Run Keepstore on local compute nodes

Added by Peter Amstutz over 4 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
10/01/2021
Due date:
11/30/2021
Story points:
-
Release:
Epics
Release relationship:
Auto

Description

  • new API endpoint that returns the keep volumes section of config + credentials
    • only available with a special token that combines container uuid + system root token
  • should run in separate process so that problems like OOM kill can be detected
  • specify number of memory buffers to use
    • need to be able to configure the static / dynamic keepstore instances separately
  • need to include memory that will be used by keepstore in node size request

Related issues

Related to Arvados - Idea #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client libraryResolvedWard Vandewege11/08/2016Actions
Related to Arvados - Feature #16745: [keep-web] Improve performance of S3 APIs using server-side cacheResolvedTom Clegg03/02/2021Actions
Related to Arvados - Feature #16347: crunch-run runs local keepstoreResolvedTom Clegg10/08/2021Actions
Related to Arvados - Feature #17749: [Keep] avoid AWS S3 request limits -- add option to use more prefixes on S3ResolvedTom Clegg09/21/2021Actions
Related to Arvados - Feature #18216: Refactor keepstore to be part of arvados-serverResolvedTom Clegg09/30/2021Actions
Blocked by Arvados - Feature #16513: Get reference Keep performance numbers for Keep-on-S3ResolvedWard Vandewege06/15/2020Actions
Actions
Copy link
 #1

Updated by Ward Vandewege over 4 years ago

  • Blocked by Feature #16513: Get reference Keep performance numbers for Keep-on-S3 added
Actions
Copy link
 #2

Updated by Ward Vandewege over 4 years ago

  • Blocked by Feature #2960: Keepstore can stream GET and PUT requests using keep-gateway API added
Actions
Copy link
 #3

Updated by Peter Amstutz over 4 years ago

  • Start date set to 06/01/2020
  • Due date set to 09/01/2020
Actions
Copy link
 #4

Updated by Ward Vandewege over 4 years ago

  • Blocked by Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstore added
Actions
Copy link
 #5

Updated by Ward Vandewege over 4 years ago

  • Related to Feature #16519: [keepstore] optimize md5sum calculations added
Actions
Copy link
 #6

Updated by Tom Clegg over 4 years ago

  • Related to Idea #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client library added
Actions
Copy link
 #7

Updated by Peter Amstutz over 4 years ago

  • Start date changed from 06/01/2020 to 10/01/2020
  • Due date changed from 09/01/2020 to 12/31/2020
Actions
Copy link
 #8

Updated by Peter Amstutz about 4 years ago

  • Start date changed from 10/01/2020 to 01/01/2021
  • Due date changed from 12/31/2020 to 03/31/2021
Actions
Copy link
 #9

Updated by Peter Amstutz about 4 years ago

  • Start date changed from 01/01/2021 to 02/01/2021
Actions
Copy link
 #10

Updated by Peter Amstutz almost 4 years ago

  • Start date changed from 02/01/2021 to 08/01/2021
  • Due date changed from 03/31/2021 to 10/31/2021
Actions
Copy link
 #11

Updated by Peter Amstutz almost 4 years ago

  • Related to Feature #16745: [keep-web] Improve performance of S3 APIs using server-side cache added
Actions
Copy link
 #12

Updated by Peter Amstutz almost 4 years ago

Actions
Copy link
 #13

Updated by Peter Amstutz almost 4 years ago

  • Subject changed from Keep performance to Run Keepstore on local compute nodes
Actions
Copy link
 #14

Updated by Peter Amstutz over 3 years ago

  • Start date changed from 08/01/2021 to 07/01/2021
Actions
Copy link
 #15

Updated by Peter Amstutz over 3 years ago

  • Related to Idea #17699: Design new keepstore internal architecture that supports efficient streaming & range requests added
Actions
Copy link
 #16

Updated by Ward Vandewege over 3 years ago

  • Related to Feature #17749: [Keep] avoid AWS S3 request limits -- add option to use more prefixes on S3 added
Actions
Copy link
 #17

Updated by Peter Amstutz over 3 years ago

  • Description updated (diff)
Actions
Copy link
 #18

Updated by Peter Amstutz over 3 years ago

  • Start date changed from 07/01/2021 to 08/01/2021
  • Due date changed from 10/31/2021 to 09/30/2021
Actions
Copy link
 #19

Updated by Peter Amstutz over 3 years ago

  • Start date changed from 08/01/2021 to 10/01/2021
  • Due date changed from 09/30/2021 to 12/31/2021
Actions
Copy link
 #20

Updated by Peter Amstutz over 3 years ago

  • Start date changed from 10/01/2021 to 09/01/2021
  • Due date changed from 12/31/2021 to 11/30/2021
Actions
Copy link
 #21

Updated by Ward Vandewege about 3 years ago

  • Blocks Bug #18101: [a-d-c] [AWS] add option to spin up (spot) instances in more/all availability zones in the region added
Actions
Copy link
 #22

Updated by Peter Amstutz about 3 years ago

  • Start date changed from 09/01/2021 to 10/01/2021
Actions
Copy link
 #23

Updated by Peter Amstutz about 3 years ago

  • Related to Feature #18216: Refactor keepstore to be part of arvados-server added
Actions
Copy link
 #24

Updated by Peter Amstutz about 3 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #25

Updated by Peter Amstutz about 3 years ago

  • Related to deleted (Idea #17699: Design new keepstore internal architecture that supports efficient streaming & range requests)
Actions
Copy link
 #26

Updated by Peter Amstutz almost 3 years ago

  • Related to deleted (Feature #16519: [keepstore] optimize md5sum calculations)
Actions
Copy link
 #27

Updated by Peter Amstutz almost 3 years ago

  • Blocked by deleted (Feature #2960: Keepstore can stream GET and PUT requests using keep-gateway API)
Actions
Copy link
 #28

Updated by Peter Amstutz almost 3 years ago

  • Blocked by deleted (Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstore)
Actions
Copy link
 #29

Updated by Peter Amstutz almost 3 years ago

  • Blocks deleted (Bug #18101: [a-d-c] [AWS] add option to spin up (spot) instances in more/all availability zones in the region)
Actions
Copy link
 #30

Updated by Peter Amstutz almost 3 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF