Keep real world performance numbers

AWS/S3 numbers

Here are some benchmark numbers against AWS S3, as measured in July 2020. They were generated with the keep-exercise tool in an Arvados installation with one dedicated Keepstore node (c5n.2xlarge) and one dedicated node for running keep-exercise (c5n.2xlarge). The Keepstore node was backed by one S3 bucket, in a VPC with an S3 endpoint installed. Versioning, Server Access logging, Static website hosting, Object-level logging and Default encryption were disabled. Object lock, Transfer acceleration and Requester pays were also disabled. There were no Active notifications. Each test consisted of 4 one minute runs, which were averaged into one number. The tests were repeated 3 times, and of those 3 runs, the highest average speed was selected and included in the table below.

The goamz driver is the default Keep S3 driver as of Arvados 2.0. In Arvados 2.1, a new S3 driver is added based on the aws-sdk-go-v2 driver. It offers better read performance but has not had as much production use. To enable the aws-sdk-go-v2 driver, set the UseAWSS3v2Driver flag to true in config.yml, as described on the Arvados documentation: S3 configuration page.

threads goamz aws-sdk-go-v2 command line
single read 32.53 MiB/s 79.48 MiB/s keep-exercise -repeat 4 -run-time 60s -vary-request -use-index -rthreads 1 -wthreads 1
single write 39.75 MiB/s 41.05 MiB/s keep-exercise -repeat 4 -run-time 60s -vary-request -use-index -rthreads 1 -wthreads 1

When using 20 concurrent read or write threads, the numbers from both drivers are:

threads goamz aws-sdk-go-v2 command line
20 read 585.60 MiB/s 898.93 MiB/s keep-exercise -repeat 4 -run-time 60s -vary-request -use-index -rthreads 20 -wthreads 0
20 write 610.40 MiB/s 688.25 MiB/s keep-exercise -repeat 4 -run-time 60s -vary-request -use-index -rthreads 0 -wthreads 20

When using 40 concurrent read or write threads, the numbers from both drivers are:

threads goamz aws-sdk-go-v2 command line comment
40 read 1005.30 MiB/s 1005.35 MiB/s keep-exercise -repeat 4 -run-time 60s -vary-request -use-index -rthreads 40 -wthreads 0
40 write 653.05 MiB/s 974.925 MiB/s keep-exercise -repeat 4 -run-time 60s -vary-request -use-index -rthreads 0 -wthreads 40 goamz is cpu bound on the keepstore (800%)

Historical AWS/S3 numbers

Name Network performance CPU RAM Keepstore Data Payload Throughput CPU load Date tested Tested by
c5.9xlarge 10 Gigabit 36 cores 72 GiB 0.1.20171212165144.296aa66-1 Saturates RX and TX (1+ GiB/sec) ~4 2018-01-18 Ward
r4.8xlarge 10 Gigabit 32 cores 244 GiB 0.1.20171212165144.296aa66-1 Saturates RX and TX (1+ GiB/sec) ~4 2018-01-17 Ward
r4.large "up to" 10 Gigabit 2 cores 15 GiB 0.1.20171212165144.296aa66-1 ~400 MiB/sec but long periods of cap at ~90 MiB/sec 2018-01-16 Ward