Bug #17507

[keep-web] [s3] Implement NextContinuationToken per ListObjectsV2 API

Added by Tom Clegg 22 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Keep
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html the ListObjectsV2 API should return NextContinuationToken when isTruncated is true, and accept that value as the continuation-token parameter in a subsequent request.

Currently, keep-web only implements ListObjectsV1 paging (i.e., it sends NextMarker and accepts that value as a marker parameter).

To fix this we should look for the list-type=2 parameter, and if it's set, send NextContinuationToken and accept continuation-token, instead of sending NextMarker and accepting marker.

Probably best to use a base64-encoded filename as a token, rather than the plain filename, in case client implementations rely on the assumption (which appears to be true with AWS) that the token won't have special characters.


Related issues

Related to Arvados Epics - Story #16360: Keep-web supports S3 compatible interfaceIn Progress07/01/202004/30/2021

History

#1 Updated by Peter Amstutz 14 days ago

  • Related to Story #16360: Keep-web supports S3 compatible interface added

Also available in: Atom PDF