Feature #17217

[controller] move blob signature calculation from api to controller

Added by Ward Vandewege 9 months ago. Updated 21 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/30/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Starting a lot of nodes that all read from the same large collection overloads the api server, with tons of ruby processes fighting over cpu. This happened on su92l today. It is possible that the blog signature calculation is the culprit. It would not be hard to move that code to controller.

Should also add a test case that puts different kinds of large manifests (lots of blocks, lots of files, lots of lines) through both controller+railsapi.


Subtasks

Task #18073: Review 17217-collection-signaturesResolvedTom Clegg


Related issues

Blocked by Arvados - Feature #17531: [controller] Remove ForceLegacyAPI14 config flagResolved04/14/2021

Associated revisions

Revision d8e3a67d
Added by Tom Clegg 21 days ago

Merge branch '17217-collection-signatures' into main

closes #17217

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Ward Vandewege 9 months ago

  • Description updated (diff)

#2 Updated by Tom Clegg 9 months ago

Suspected example: failed container request su92l-xvhdp-ro44bqknjjbkpdv, log su92l-4zz18-s0iat32namtr51h

2020-12-17T16:02:29.440839471Z /mnt/su92l-4zz18-9ghkf2xpro0ff9q/NA19732.haplotypeCalls.er.raw.vcf.gz phase 1: request failed: https://su92l.arvadosapi.com:443/arvados/v1/collections/su92l-4zz18-9ghkf2xpro0ff9q?select=%5B%22uuid%22%2C%22manifest_text%22%5D: 503 Service Unavailable: request failed: http://localhost:8000/arvados/v1/collections/su92l-4zz18-9ghkf2xpro0ff9q?select=%5B%22uuid%22%2C%22manifest_text%22%5D: 503 Service Unavailable

#3 Updated by Tom Clegg 7 months ago

  • Target version set to To Be Groomed
  • Subject changed from [controller] move blog signature calculation from api to controller to [controller] move blob signature calculation from api to controller

#4 Updated by Tom Clegg 5 months ago

  • Blocked by Feature #17531: [controller] Remove ForceLegacyAPI14 config flag added

#5 Updated by Tom Clegg 5 months ago

  • Assigned To set to Tom Clegg
  • Status changed from New to In Progress

#6 Updated by Tom Clegg 5 months ago

some work in progress on 17217-collection-signatures

#7 Updated by Tom Clegg 5 months ago

  • Description updated (diff)

#8 Updated by Peter Amstutz 3 months ago

  • Target version deleted (To Be Groomed)

#9 Updated by Tom Clegg 21 days ago

17217-collection-signatures @ d67a11c2ee42159dd0ecd8f6ef39af38b6380dfd -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2664/
  • RailsAPI returns unsigned manifests (signing code still exists only for testing purposes)
  • Controller signs manifests returned by create/update/get/list calls
  • As before, manifests returned by groups#contents are not signed
  • Removed a bunch of unused code from controller's "old code path" (not strictly necessary, but it made it easier to confirm that the old code path wasn't actually used and didn't need to be updated)

#10 Updated by Tom Clegg 21 days ago

  • Target version set to 2021-09-01 sprint

#11 Updated by Lucas Di Pentima 21 days ago

It would be interesting to have a performance comparison, but I guess it isn't trivial to make if the issue presents on production-level workloads.

LGTM, thanks!

#12 Updated by Tom Clegg 21 days ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF