Bug #10243
closed
[PySDK] Enable the Collection file packing feature by explicit flag
Description
Pass an optional flush=False to ArvadosFileWriter.close() to tell the SDK to pack small files, keeping the feature disabled by default to doesn't confuse FUSE.
Updated by Lucas Di Pentima over 7 years ago
- Subject changed from Enable the Collection file packing feature by explicit flag to [PySDK] Enable the Collection file packing feature by explicit flag
Updated by Lucas Di Pentima over 7 years ago
- Status changed from New to In Progress
Updated by Lucas Di Pentima over 7 years ago
e86904aa647a984a04aea2c4ce20c61cae8ee9f5
Added an optional "flush" flag on ArvadosFileWriter.close(), that's True by default so that all clients get the old behaviour implicitly.
Updated some new tests to explictly not flush when closing to test the new packing feature.
Run the tests a couple of times on Jenkins:
https://ci.curoverse.com/job/developer-run-tests/24/
https://ci.curoverse.com/job/developer-run-tests/25/
...with different tests failing, and fuse tests failing on both. Locally all the tests passed, including the fuse ones, which I ran at least 3 times.
Updated by Tom Clegg over 7 years ago
Looks like builds 24 and 25 both failed on an unrelated(?) flaky test #10008.
FUSE tests pass for me as well.
I think it would be good to have a test in sdk/python that ensures close() (without flush=False) causes short blocks to be committed. Currently it seems like we are relying on FUSE integration test tests.test_mount.FuseModifyFileTest to catch that.
With that, e86904a LGTM, thanks.
Updated by Lucas Di Pentima over 7 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:2656053035f46e6097b380f5c0da7d54b04a60c4.