Project

General

Profile

Actions

Bug #18293

closed

MemoryError creating SIF

Added by Peter Amstutz over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Description

https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-tzmwfcbx9r415ed#Log

crunch-run.txt:

While performing build: while creating SIF: while creating container: writing data object for SIF file: copying data object file to SIF file: write /tmp/crunch-run.9tee4-dz642-0prt4gvmv3qkmna.1720801655/keep2342085199/by_uuid/9tee4-4zz18-i232kkojjjrtrt0/image.sif: input/output error

arv-mount.txt:

2021-10-25T20:46:57.239059516Z 2021-10-25 20:46:57 arvados.arvados_fuse3532322 ERROR: Unhandled exception during FUSE operation
2021-10-25T20:46:57.239059516Z Traceback (most recent call last):
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados_fuse/__init__.py", line 327, in catch_exceptions_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados_fuse/__init__.py", line 676, in write
2021-10-25T20:46:57.239059516Z w = handle.obj.writeto(off, buf, self.num_retries)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados_fuse/fusefile.py", line 69, in writeto
2021-10-25T20:46:57.239059516Z return self.arvfile.writeto(off, buf, num_retries)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 469, in must_be_writable_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 1153, in writeto
2021-10-25T20:46:57.239059516Z self._current_bblock.repack_writes()
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 436, in repack_writes
2021-10-25T20:46:57.239059516Z new_bb.append(contents[t.segment_offset:t.segment_offset+t.range_size])
2021-10-25T20:46:57.239059516Z MemoryError

Actions #1

Updated by Peter Amstutz over 2 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz over 2 years ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz over 2 years ago

  • Release set to 42
Actions #4

Updated by Ward Vandewege over 2 years ago

Both compute nodes on 9tee4 have

mksquashfs mem = 512M

in their /etc/singularity/singularity.conf file.

Actions #5

Updated by Peter Amstutz over 2 years ago

  • Status changed from In Progress to Resolved

We've lowered the recommended memory limit for mksquashfs, and also confirmed that we can work around it by increasing RequestExtraRAM to the config.

Actions

Also available in: Atom PDF