Arvados

A user's container request failed to dispatch because the container created for it had corrupted mounts, so Crunch could not dispatch it. There were ~2,250 mounts of individual files. This container was near the end of a workflow, so all its inputs came from a single collection, the output of a previous step.

The mounts were corrupted in different ways:

• Some had their path changed, so the mount for a given file would be backed by a different file.
• Some became the empty array [].
• Some became the literal object {"for": "to_hash"}.

We haven't yet identified a pattern for which mounts got corrupted which way.

We did check the database record and confirmed the mounts were corrupted in the container's mounts column. This suggests that the problem happened during container creation, and isn't an issue with, e.g., serializing the record to JSON.

See child ticket for supporting documentation.