Arvados

The science team has expressed the need for a command line tool to provide comprehensive editing of collections/manifests. Some possible features include:

• Delete files or streams from a collection
• Filter collection (include/exclude files by regex)
• Rename files or streams (by name or by regex)
• Move files/streams from one collection to another
• Combine manifests to get a new manifest
• Add file(s) from another collection
• Upload files using arvados.command.put module and add to collection
• Optimize manifest by re-packing small files (or files that are split across many more blocks than necessary) into continuous ranges that fills whole blocks (possibly with some logic to avoid starting new files at the very end of a block)

User should be able to specify whether to modify collection in place, create new collection, or operate on manifest text read in from stdin.