Bug #17507
closed[keep-web] [s3] Implement NextContinuationToken per ListObjectsV2 API
Description
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html the ListObjectsV2 API should return NextContinuationToken when isTruncated is true, and accept that value as the continuation-token parameter in a subsequent request.
Currently, keep-web only implements ListObjectsV1 paging (i.e., it sends NextMarker and accepts that value as a marker parameter).
To fix this we should look for the list-type=2 parameter, and if it's set, send NextContinuationToken and accept continuation-token, instead of sending NextMarker and accepting marker.
Probably best to use a base64-encoded filename as a token, rather than the plain filename, in case client implementations rely on the assumption (which appears to be true with AWS) that the token won't have special characters.
Related issues
Updated by Peter Amstutz over 3 years ago
- Related to Idea #16360: Keep-web supports S3 compatible interface added
Updated by Peter Amstutz over 3 years ago
- Target version changed from Arvados Future Sprints to 2021-04-28 bughunt sprint
- Assigned To set to Tom Clegg
Updated by Tom Clegg over 3 years ago
17507-listobjectsv2 @ c2d2234ce0da91881fc63459a30c5efcfbe29a26 -- developer-run-tests: #2437
Updated by Nico César over 3 years ago
review @ c2d2234ce0da91881fc63459a30c5efcfbe29a26
Not much to object here, TestS3ListObjectsV2EncodingTypeURL and testS3CollectionListRollup seem to cover good amount of ground
reusing 'marker' for v2 and giving a different semantic and '+' divergence from the standard was not cool but that's not our fault.
Ready to merge
Updated by Tom Clegg over 3 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|40693f6b000a2089038af611eb93dc69d66377e3.