Bug #15946

[crunch-run] [collectionfs] Deadlock while writing output collection

Added by Tom Clegg 9 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
12/23/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Running several similar containers, some succeeded but some got stuck while writing the final output collection. The log indicates all of the expected files were written to collectionfs ("copying ... (... bytes)") but the finished collection was never saved.


Subtasks

Task #15947: Review 15946-collectionfs-deadlockResolvedTom Clegg


Related issues

Related to Arvados - Bug #15928: Deadlock in crunch-run testsResolved12/11/2019

Related to Arvados - Bug #15910: [crunch-run] Crash while writing output filesResolved12/04/2019

Associated revisions

Revision 949abe8b
Added by Tom Clegg 9 months ago

Merge branch '15946-collectionfs-deadlock'

fixes #15946

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg 9 months ago

Found a case where commitBlock would set a "flushing in progress" flag on a segment, but then return early (and never indicate
it was done) because a different segment had another flush in progress. Once that happened, sync operations like MarshalManifest would block forever.

15946-collectionfs-deadlock @ c05caa378debd04205690c6cb96508e4e7fb6c8b -- https://ci.arvados.org/view/Developer/job/developer-run-tests/1700/

#2 Updated by Tom Clegg 9 months ago

  • Related to Bug #15928: Deadlock in crunch-run tests added

#3 Updated by Tom Clegg 9 months ago

  • Related to Bug #15910: [crunch-run] Crash while writing output files added

#4 Updated by Lucas Di Pentima 9 months ago

This LGTM, thanks!

#5 Updated by Anonymous 9 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#6 Updated by Peter Amstutz 9 months ago

  • Target version changed from 2020-01-15 Sprint to 2020-01-02 Sprint

Also available in: Atom PDF