Bug #22226
Updated by Brett Smith about 1 month ago
I have the full log if needed but here's the important bits. Worth noting, this is on a test compute node with Docker 27. The same compute image works with Singularity okay.
<pre>Oct 21 14:51:18 crunch-run[1495]: crunch-run 2.8.0~dev20241015141800 (go1.21.10) started
…
Oct 21 14:51:55 crunch-run[1495]: tordo-dz642-84o7gwkioo1nvn9 2024-10-21T14:51:55.242438032Z copying "*"
Oct 21 14:51:55 crunch-run[1495]: tordo-dz642-84o7gwkioo1nvn9 2024-10-21T14:51:55.245246755Z copying ".aws/config" from c80031b757ffc50684d0b8cbe5282587+112/.aws/config
Oct 21 14:51:55 crunch-run[1495]: tordo-dz642-84o7gwkioo1nvn9 2024-10-21T14:51:55.269394051Z copying "download.sh" from c80031b757ffc50684d0b8cbe5282587+112/download.sh
Oct 21 14:51:55 crunch-run[1495]: tordo-dz642-84o7gwkioo1nvn9 2024-10-21T14:51:55.271639670Z Running [arv-mount --unmount-timeout=8 --unmount /tmp/crunch-run.tordo-dz642-84o7gwkioo1nvn9.3147090036/keep1196122805]
Oct 21 14:51:55 crunch-run[1495]: tordo-dz642-84o7gwkioo1nvn9 2024-10-21T14:51:55.981973623Z crunch-run finished
Oct 21 14:51:55 crunch-run[1495]: panic: runtime error: invalid memory address or nil pointer dereference
Oct 21 14:51:55 crunch-run[1495]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9581e9]
Oct 21 14:51:55 crunch-run[1495]: goroutine 1 [running]:
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/sdk/go/arvados.(*dirnode).Child(0xc0000a4900, {0xc0007b8570, 0xb}, 0x0?)
Oct 21 14:51:55 crunch-run[1495]: /arvados/sdk/go/arvados/fs_collection.go:993 +0x29
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/sdk/go/arvados.(*filenode).Splice(0xc0000b2e60, {0x2969988?, 0xc000444e60?})
Oct 21 14:51:55 crunch-run[1495]: /arvados/sdk/go/arvados/fs_collection.go:964 +0x168
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/sdk/go/arvados.(*filehandle).Splice(0x0?, 0x0?)
Oct 21 14:51:55 crunch-run[1495]: /arvados/sdk/go/arvados/fs_filehandle.go:152 +0x2e
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/sdk/go/arvados.Splice({0x2969e28, 0xc0000be840}, {0xc0000ad4ce, 0xc}, 0xc000524d00?)
Oct 21 14:51:55 crunch-run[1495]: /arvados/sdk/go/arvados/fs_base.go:871 +0x1e4
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*copier).copyFromCollection(0xc000712ce8, {0xc0000ad4ce, 0xc}, 0xc000252300?, {0xc0002d8610, 0xb})
Oct 21 14:51:55 crunch-run[1495]: /arvados/lib/crunchrun/copier.go:424 +0x20b
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*copier).walkMount(0xc00076ece8, {0xc0000ad4ce, 0xc}, {0xc0000ad4c0, 0x1a}, 0x0?, 0x0)
Oct 21 14:51:55 crunch-run[1495]: /arvados/lib/crunchrun/copier.go:375 +0x925
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*copier).walkMountsBelow(0xc00076ece8, {0x0, 0x0}, {0xc0002d96d0, 0xe})
Oct 21 14:51:55 crunch-run[1495]: /arvados/lib/crunchrun/copier.go:454 +0x24f
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*copier).walkMount(0xc000712ce8, {0x0, 0x0}, {0xc0002d96d0, 0xe}, 0x3?, 0x1)
Oct 21 14:51:55 crunch-run[1495]: /arvados/lib/crunchrun/copier.go:401 +0xc0d
Oct 21 14:51:55 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*copier).Copy(0xc00076ece8)
Oct 21 14:51:55 crunch-run[1495]: /arvados/lib/crunchrun/copier.go:80 +0x139
Oct 21 14:51:56 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*ContainerRunner).CaptureOutput(0xc00001c300, 0xc0003863c0?)
Oct 21 14:51:56 crunch-run[1495]: /arvados/lib/crunchrun/crunchrun.go:1336 +0x27c
Oct 21 14:51:56 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*ContainerRunner).Run.func2()
Oct 21 14:51:56 crunch-run[1495]: /arvados/lib/crunchrun/crunchrun.go:1679 +0xf9
Oct 21 14:51:56 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.(*ContainerRunner).Run(0xc00001c300)
Oct 21 14:51:56 crunch-run[1495]: /arvados/lib/crunchrun/crunchrun.go:1765 +0x1007
Oct 21 14:51:56 crunch-run[1495]: git.arvados.org/arvados.git/lib/crunchrun.command.RunCommand({}, {0x7ffc7d645dae, 0x43}, {0xc000106190, 0x5, 0x5}, {0x2932aa0?, 0xc000130018}, {0x2932ac0, 0xc000130020}, ...)
Oct 21 14:51:56 crunch-run[1495]: /arvados/lib/crunchrun/crunchrun.go:2101 +0x29ff
Oct 21 14:51:56 crunch-run[1495]: git.arvados.org/arvados.git/lib/cmd.Multi.RunCommand(0x47415a?, {0x7ffc7d645dae, 0x43}, {0xc000106190?, 0x5, 0x5}, {0x2932aa0, 0xc000130018}, {0x2932ac0, 0xc000130020}, ...)
Oct 21 14:51:56 crunch-run[1495]: /arvados/lib/cmd/cmd.go:105 +0x2da
Oct 21 14:51:56 crunch-run[1495]: main.main()
Oct 21 14:51:56 crunch-run[1495]: /arvados/cmd/arvados-server/cmd.go:73 +0x8e
</pre>