Project

General

Profile

Actions

Bug #11510

closed

[SDK] Support writes to offsets beyond end of file

Added by Peter Amstutz over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-

Description

Two distinct but closely related issues encountered by a customer:

  1. Using s3.download_fileobj() with an ArvadosFileWriter target: files over some threshold (likely 8MB) are arbitrarily truncated.
  2. Using "aws s3 cp" to download to a writable keep mount: a 15 GB file results in only an 8 MB file appearing it keep(!)

I believe the common theme here is that they both rely on the Python SDK and are using multipart download (request and commit chunks 8 MB at a time) which suggests some issue with the way it writes chunks non-sequentially (possibly relating to seek() or ftruncate()).


Subtasks 1 (0 open1 closed)

Task #11528: Review 11510-sdk-extend-filesResolvedPeter Amstutz04/20/2017Actions

Related issues

Related to Arvados - Bug #10035: [FUSE] Determine why bcl2fastq doesn't work with writable keep mountResolvedPeter Amstutz09/13/2016Actions
Actions

Also available in: Atom PDF