Bug #17507
[keep-web] [s3] Implement NextContinuationToken per ListObjectsV2 API
100%
Description
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html the ListObjectsV2 API should return NextContinuationToken when isTruncated is true, and accept that value as the continuation-token parameter in a subsequent request.
Currently, keep-web only implements ListObjectsV1 paging (i.e., it sends NextMarker and accepts that value as a marker parameter).
To fix this we should look for the list-type=2 parameter, and if it's set, send NextContinuationToken and accept continuation-token, instead of sending NextMarker and accepting marker.
Probably best to use a base64-encoded filename as a token, rather than the plain filename, in case client implementations rely on the assumption (which appears to be true with AWS) that the token won't have special characters.
Subtasks
Related issues
Associated revisions
History
#1
Updated by Peter Amstutz about 1 year ago
- Related to Story #16360: Keep-web supports S3 compatible interface added
#2
Updated by Peter Amstutz about 1 year ago
- Target version changed from Arvados Future Sprints to 2021-04-28 bughunt sprint
- Assigned To set to Tom Clegg
#3
Updated by Tom Clegg about 1 year ago
- Status changed from New to In Progress
#4
Updated by Tom Clegg about 1 year ago
17507-listobjectsv2 @ c2d2234ce0da91881fc63459a30c5efcfbe29a26 -- developer-run-tests: #2437
#5
Updated by Nico César about 1 year ago
review @ c2d2234ce0da91881fc63459a30c5efcfbe29a26
Not much to object here, TestS3ListObjectsV2EncodingTypeURL and testS3CollectionListRollup seem to cover good amount of ground
reusing 'marker' for v2 and giving a different semantic and '+' divergence from the standard was not cool but that's not our fault.
Ready to merge
#6
Updated by Tom Clegg about 1 year ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|40693f6b000a2089038af611eb93dc69d66377e3.
#7
Updated by Peter Amstutz 11 months ago
- Release set to 38
Merge branch '17507-listobjectsv2'
fixes #17507
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>