Bug #17507

[keep-web] [s3] Implement NextContinuationToken per ListObjectsV2 API

Added by Tom Clegg 8 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Start date:
04/23/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html the ListObjectsV2 API should return NextContinuationToken when isTruncated is true, and accept that value as the continuation-token parameter in a subsequent request.

Currently, keep-web only implements ListObjectsV1 paging (i.e., it sends NextMarker and accepts that value as a marker parameter).

To fix this we should look for the list-type=2 parameter, and if it's set, send NextContinuationToken and accept continuation-token, instead of sending NextMarker and accepting marker.

Probably best to use a base64-encoded filename as a token, rather than the plain filename, in case client implementations rely on the assumption (which appears to be true with AWS) that the token won't have special characters.


Subtasks

Task #17569: Review 17507-listobjectsv2ResolvedTom Clegg


Related issues

Related to Arvados Epics - Story #16360: Keep-web supports S3 compatible interfaceResolved07/01/202004/30/2021

Associated revisions

Revision 40693f6b
Added by Tom Clegg 7 months ago

Merge branch '17507-listobjectsv2'

fixes #17507

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz 8 months ago

  • Related to Story #16360: Keep-web supports S3 compatible interface added

#2 Updated by Peter Amstutz 7 months ago

  • Target version changed from Arvados Future Sprints to 2021-04-28 bughunt sprint
  • Assigned To set to Tom Clegg

#3 Updated by Tom Clegg 7 months ago

  • Status changed from New to In Progress

#5 Updated by Nico C├ęsar 7 months ago

review @ c2d2234ce0da91881fc63459a30c5efcfbe29a26

Not much to object here, TestS3ListObjectsV2EncodingTypeURL and testS3CollectionListRollup seem to cover good amount of ground

reusing 'marker' for v2 and giving a different semantic and '+' divergence from the standard was not cool but that's not our fault.

Ready to merge

#6 Updated by Tom Clegg 7 months ago

  • Status changed from In Progress to Resolved

#7 Updated by Peter Amstutz 6 months ago

  • Release set to 38

Also available in: Atom PDF