Bug #15486

[crunchstat-summary] error fetching container log collection for v1.4+

Added by Tom Morris 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/13/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

When attempting to analyze Arvados v1.4 container requests, crunchstat-summary attempts to load the collection specified by the log PDH in the Container record, but that collection never exists because it is deleted immediately. Also there is no longer a `log` field for the container_request record.

For Arvados v1.4 and up, crunchstat-summary needs to be modified so that:
  • for Containers, it first looks up the associated Container Request and uses the log from there
  • for Container Requests, remove the current logic which looks up the associated Container log and instead load the CR log by UUID from `log_uuid` (there is no `log` field)

Subtasks

Task #15513: Review 15486-crunchstat-summary-v1.4-supportResolvedEric Biagiotti


Related issues

Blocks Arvados - Story #14952: [crunchstat-summary] Add arv-mount FUSE ops timeIn Progress

Associated revisions

Revision 69d3ccc6 (diff)
Added by Tom Morris 3 months ago

Fix so it works with Arvados v1.4+

refs #15486

Arvados v1.4 changed things so that container logs are no longer
accessible via PDH so we need to go to the container request and
access the log via UUID from there.

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

Revision deb7ae71 (diff)
Added by Tom Morris 3 months ago

15486: Fix crunchstat-summary to work with Arvados v1.4+

refs #15486

Arvados v1.4 changed things so that container logs are no longer
accessible via PDH so we need to go to the container request and
access the log via UUID from there.

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

Revision d2393e33 (diff)
Added by Tom Morris 2 months ago

15486: Fix crunchstat-summary to work with Arvados v1.4+

refs #15486

Arvados v1.4 changed things so that container logs are no longer
accessible via PDH so we need to go to the container request and
access the log via UUID from there.

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

Revision 30f8e85a (diff)
Added by Tom Morris 2 months ago

Add missing test file

Refs #15486

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

Revision dce9a9c4 (diff)
Added by Tom Morris 2 months ago

Sanity check that we have a container request for this container

Should never be empty, but just in case ...

Refs #15486

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

Revision 16a5fe8c (diff)
Added by Tom Morris 2 months ago

Add more missing test files

Refs #15486

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

Revision 764bb78e (diff)
Added by Tom Morris 2 months ago

15486: Fix crunchstat-summary to work with Arvados v1.4+

refs #15486

Arvados v1.4 changed things so that container logs are no longer
accessible via PDH so we need to go to the container request and
access the log via UUID from there.

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

Revision 86b79031
Added by Tom Morris 2 months ago

Merge branch '15486-crunchstat-summary-v1.4-support'

Fixes #15486

Arvados-DCO-1.1-Signed-off-by: Tom Morris <>

History

#2 Updated by Tom Morris 3 months ago

  • Description updated (diff)

#3 Updated by Tom Morris 3 months ago

I need to add some v1.4+ tests, but the code on branch 15486-crunchstat-summary-v1.4-support should be usable for those who need an immediate workaround.

#4 Updated by Tom Morris 3 months ago

  • Target version changed from 2019-07-31 Sprint to 2019-08-14 Sprint

#5 Updated by Tom Morris 2 months ago

  • Blocks Story #14952: [crunchstat-summary] Add arv-mount FUSE ops time added

#6 Updated by Eric Biagiotti 2 months ago

Reviewing at d2393e33ce53554ebd271244bf988d92fdd5293e.

  • Might want to check if len(items) > 0 in the following:
    process = arv.container_requests().list(filters=[['container_uuid','=',uuid]],limit=1).execute()['items'][0]
    

Also please do a developer run on jerkins once you're done. Thanks!

#7 Updated by Tom Morris 2 months ago

  • Assigned To changed from Tom Morris to Eric Biagiotti

Both issues fixed at dce9a9c4800a79d357fd6a8400b81c9b7404dbd5

Developer tests running at https://ci.curoverse.com/view/Developer/job/developer-run-tests/1478/

#8 Updated by Eric Biagiotti 2 months ago

  • Assigned To changed from Eric Biagiotti to Tom Morris

#9 Updated by Tom Morris 2 months ago

  • Status changed from In Progress to Resolved

Merged at 86b7903116608ac3a8314437639cb61d774fb510

#10 Updated by Tom Morris 2 months ago

  • Release set to 26

Also available in: Atom PDF