Project

General

Profile

Actions

Bug #17507

closed

[keep-web] [s3] Implement NextContinuationToken per ListObjectsV2 API

Added by Tom Clegg over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Story points:
-
Release relationship:
Auto

Description

According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html the ListObjectsV2 API should return NextContinuationToken when isTruncated is true, and accept that value as the continuation-token parameter in a subsequent request.

Currently, keep-web only implements ListObjectsV1 paging (i.e., it sends NextMarker and accepts that value as a marker parameter).

To fix this we should look for the list-type=2 parameter, and if it's set, send NextContinuationToken and accept continuation-token, instead of sending NextMarker and accepting marker.

Probably best to use a base64-encoded filename as a token, rather than the plain filename, in case client implementations rely on the assumption (which appears to be true with AWS) that the token won't have special characters.


Subtasks 1 (0 open1 closed)

Task #17569: Review 17507-listobjectsv2ResolvedTom Clegg04/23/2021Actions

Related issues

Related to Arvados Epics - Idea #16360: Keep-web supports S3 compatible interfaceResolved07/01/202004/30/2021Actions
Actions #1

Updated by Peter Amstutz over 3 years ago

  • Related to Idea #16360: Keep-web supports S3 compatible interface added
Actions #2

Updated by Peter Amstutz over 3 years ago

  • Target version changed from Arvados Future Sprints to 2021-04-28 bughunt sprint
  • Assigned To set to Tom Clegg
Actions #3

Updated by Tom Clegg over 3 years ago

  • Status changed from New to In Progress
Actions #5

Updated by Nico César over 3 years ago

review @ c2d2234ce0da91881fc63459a30c5efcfbe29a26

Not much to object here, TestS3ListObjectsV2EncodingTypeURL and testS3CollectionListRollup seem to cover good amount of ground

reusing 'marker' for v2 and giving a different semantic and '+' divergence from the standard was not cool but that's not our fault.

Ready to merge

Actions #6

Updated by Tom Clegg over 3 years ago

  • Status changed from In Progress to Resolved
Actions #7

Updated by Peter Amstutz over 3 years ago

  • Release set to 38
Actions

Also available in: Atom PDF