Bug #21205
Updated by Peter Amstutz 6 months ago
User reported "duplicate key value violates unique constraint index_collections_on_owner_and_name". This error occurred from arvados-cwl-runner creating a collection called "Intermediate collection". This case does supply the ensure_unique_name flag. However, it seems that between running 100s of workflows, and this particular step doing a 24-way scatter, that it attempting to create ~10 collections per second (see screenshot), and that, while < 1% chance, it is evidently possible for two or more collections to be created at the same time and for save_with_unique_name! to fail enough times in a row that one of those collection create operations fails with a uniqueness error. From discussion: I propose two tweaks to make this function more robust: Instead of adding # increase max_retries from 2 to 10 # make the "sleep" statement pick a timestamp, we should add ~10 random alphanumeric digits (similar to the logic used sleep time between 2 and 10 milliseconds instead of always sleeping for UUIDs). As demonstrated, timestamps are prone to collision, and using millisecond precision that means we are limited to creating 1000 collections per second. exactly 2 milliseconds