Feature #8502

[SDKs] Fetch all results with the CLI `list` command

Added by Joshua Randall about 5 years ago. Updated about 5 years ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


It would be nice if the arv CLI `list` command had a way to handle fetching more records than the API server allows, by doing the batching for you and just giving all of the matching records requested. Implementing this externally is not trivial, so I think it would make sense for the CLI to do it internally.

I'd suggest something like a `--batch` or `-b` option that enabled this behaviour.

If a `-l` limit is also given, that should still be respected, but there should also be a way to signify that you want ALL of the matching records. There are at least a few options for how that might work:
- the default behaviour of `-l` could be changed when `--batch` is in effect (such that not specifying `-l` implies no limit)
- a special value for `-l` could indicate "no limit" (e.g. `--limit -1` or `-l -1` although the readability of the latter leaves something to be desired)
- another new option `--all` that overrides the setting of `-l`

It may also be advantageous to be able to specify the batch size, so perhaps the `--batch` / `-b` options could take an (optional?) numeric argument.

It probably would be a good idea to also document what the expected behaviour would be if data changes between the first and last batch, although I suspect there is functionality missing from the API server that would make it easy to guarantee a consistent set of results (e.g. using cursors).


#1 Updated by Brett Smith about 5 years ago

  • Subject changed from automatically fetching all results with the CLI `list` command to [SDKs] Fetch all results with the CLI `list` command

#2 Updated by Brett Smith about 5 years ago

  • Target version set to Arvados Future Sprints

Also available in: Atom PDF