Arvados

The API server has a configurable limit to the manifest size it will accept. When arv-put is asked to upload a set of files and directories, it just starts uploading until it has all the blocks written, and then calculates the manifest and tries to save it.

When that fails because the manifest is too large, the user has wasted a lot of time and network bandwidth. They will need to make a smaller version of the collection, and then reupload all the blocks. The arv-put block cache will not help them because the collection to be uploaded will be different.

Add a feature to arv-put to calculate the (worst case) manifest size up front, and refuse to upload when it is too large. This should happen before arv-put reads all the data that it needs to upload, it should error out as early as possible.