Bug #17853

[keep-web] concurrent map iteration and map write

Added by Tom Clegg 4 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
07/05/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

https://ci.arvados.org/job/developer-run-tests-remainder/2663/consoleFull

We've fixed this before (in #17464 when it started happening) but, evidently, the fix was not complete.

fatal error: concurrent map iteration and map write

goroutine 11372 [running]:
runtime.throw(0xe70ca4, 0x26)
    /usr/src/go/src/runtime/panic.go:1117 +0x72 fp=0xc0042e7598 sp=0xc0042e7568 pc=0x43a5f2
runtime.mapiternext(0xc0042e7658)
    /usr/src/go/src/runtime/map.go:858 +0x54c fp=0xc0042e7618 sp=0xc0042e7598 pc=0x412d8c
runtime.mapiterinit(0xd68960, 0xc005dbced0, 0xc0042e7658)
    /usr/src/go/src/runtime/map.go:848 +0x1c5 fp=0xc0042e7638 sp=0xc0042e7618 pc=0x412745
git.arvados.org/arvados.git/sdk/go/arvados.(*treenode).MemorySize(0xc00558c630, 0x0)
    /tmp/workspace/developer-run-tests-remainder/sdk/go/arvados/fs_base.go:355 +0xf2 fp=0xc0042e76d8 sp=0xc0042e7638 pc=0x8cfdf2
git.arvados.org/arvados.git/sdk/go/arvados.(*fileSystem).MemorySize(0xc0052fd730, 0x0)
    /tmp/workspace/developer-run-tests-remainder/sdk/go/arvados/fs_base.go:650 +0x33 fp=0xc0042e76f8 sp=0xc0042e76d8 pc=0x8d2313
git.arvados.org/arvados.git/services/keep-web.(*cache).pruneSessions(0xc003517d08)
    /tmp/workspace/developer-run-tests-remainder/services/keep-web/cache.go:277 +0x415 fp=0xc0042e77d8 sp=0xc0042e76f8 pc=0xc82475
runtime.goexit()
    /usr/src/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0042e77e0 sp=0xc0042e77d8 pc=0x474241
created by git.arvados.org/arvados.git/services/keep-web.(*cache).GetSession
    /tmp/workspace/developer-run-tests-remainder/services/keep-web/cache.go:249 +0x1b0

Subtasks

Task #17854: Review 17853-fix-write-with-rlockResolvedTom Clegg

Associated revisions

Revision 50e69b50
Added by Tom Clegg 4 months ago

Merge branch '17853-fix-write-with-rlock' into main

fixes #17853

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg 4 months ago

  • Status changed from New to In Progress

#2 Updated by Tom Clegg 4 months ago

  • Description updated (diff)

17853-fix-write-with-rlock @ dd8ae798774d29b35532b99b78cae67a151654be -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2565/

-       if createMode {
-               parent.Lock()
-               defer parent.Unlock()
-       } else {
-               parent.RLock()
-               defer parent.RUnlock()
-       }
+       // We always need to take Lock() here, not just RLock(). Even
+       // if we know we won't be creating a file, parent might be a
+       // lookupnode, which sometimes populates its inodes map during
+       // a Child() call.
+       parent.Lock()
+       defer parent.Unlock()

#3 Updated by Lucas Di Pentima 4 months ago

Although I wasn't able to make the previous version fail, this LGTM.

#4 Updated by Tom Clegg 4 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

Also available in: Atom PDF