Project

General

Profile

Actions

Feature #17217

closed

[controller] move blob signature calculation from api to controller

Added by Ward Vandewege over 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/30/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Starting a lot of nodes that all read from the same large collection overloads the api server, with tons of ruby processes fighting over cpu. This happened on su92l today. It is possible that the blog signature calculation is the culprit. It would not be hard to move that code to controller.

Should also add a test case that puts different kinds of large manifests (lots of blocks, lots of files, lots of lines) through both controller+railsapi.


Subtasks 1 (0 open1 closed)

Task #18073: Review 17217-collection-signaturesResolvedTom Clegg08/30/2021

Actions

Related issues

Blocked by Arvados - Feature #17531: [controller] Remove ForceLegacyAPI14 config flagResolvedTom Clegg04/14/2021

Actions
Actions #1

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #2

Updated by Tom Clegg over 2 years ago

Suspected example: failed container request su92l-xvhdp-ro44bqknjjbkpdv, log su92l-4zz18-s0iat32namtr51h

2020-12-17T16:02:29.440839471Z /mnt/su92l-4zz18-9ghkf2xpro0ff9q/NA19732.haplotypeCalls.er.raw.vcf.gz phase 1: request failed: https://su92l.arvadosapi.com:443/arvados/v1/collections/su92l-4zz18-9ghkf2xpro0ff9q?select=%5B%22uuid%22%2C%22manifest_text%22%5D: 503 Service Unavailable: request failed: http://localhost:8000/arvados/v1/collections/su92l-4zz18-9ghkf2xpro0ff9q?select=%5B%22uuid%22%2C%22manifest_text%22%5D: 503 Service Unavailable
Actions #3

Updated by Tom Clegg over 2 years ago

  • Target version set to To Be Groomed
  • Subject changed from [controller] move blog signature calculation from api to controller to [controller] move blob signature calculation from api to controller
Actions #4

Updated by Tom Clegg about 2 years ago

  • Blocked by Feature #17531: [controller] Remove ForceLegacyAPI14 config flag added
Actions #5

Updated by Tom Clegg about 2 years ago

  • Assigned To set to Tom Clegg
  • Status changed from New to In Progress
Actions #6

Updated by Tom Clegg about 2 years ago

some work in progress on 17217-collection-signatures

Actions #7

Updated by Tom Clegg about 2 years ago

  • Description updated (diff)
Actions #8

Updated by Peter Amstutz almost 2 years ago

  • Target version deleted (To Be Groomed)
Actions #9

Updated by Tom Clegg over 1 year ago

17217-collection-signatures @ d67a11c2ee42159dd0ecd8f6ef39af38b6380dfd -- developer-run-tests: #2664
  • RailsAPI returns unsigned manifests (signing code still exists only for testing purposes)
  • Controller signs manifests returned by create/update/get/list calls
  • As before, manifests returned by groups#contents are not signed
  • Removed a bunch of unused code from controller's "old code path" (not strictly necessary, but it made it easier to confirm that the old code path wasn't actually used and didn't need to be updated)
Actions #10

Updated by Tom Clegg over 1 year ago

  • Target version set to 2021-09-01 sprint
Actions #11

Updated by Lucas Di Pentima over 1 year ago

It would be interesting to have a performance comparison, but I guess it isn't trivial to make if the issue presents on production-level workloads.

LGTM, thanks!

Actions #12

Updated by Tom Clegg over 1 year ago

  • Status changed from In Progress to Resolved
Actions #13

Updated by Peter Amstutz over 1 year ago

  • Release set to 42
Actions

Also available in: Atom PDF