Project

General

Profile

Actions

Idea #10484

closed

[keepstore] Add S3 bucket statistics and current #clients to status.json

Added by Tom Morris about 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
11/17/2016
Due date:
Story points:
1.0

Description

- requests by type - PUT, GET, LIST, DELETE
- bytes on the wire - received, sent
- errors, grouped by status codes (HTTP response code + S3)


Subtasks 3 (0 open3 closed)

Task #10508: Review 10484-keepstore-statsResolvedRadhika Chippada11/18/2016Actions
Task #10580: Review 10484-count-clientsResolvedTom Clegg11/17/2016Actions
Task #10547: refactor as tracking-proxy objectResolvedTom Clegg11/17/2016Actions

Related issues

Related to Arvados - Feature #13025: [keepstore] runtime metrics endpointResolvedTom Clegg02/08/2018Actions
Actions #1

Updated by Tom Morris about 8 years ago

  • Description updated (diff)
Actions #2

Updated by Tom Clegg about 8 years ago

  • Description updated (diff)
  • Subject changed from [keepstore] Log S3 statistics to [keepstore] Add S3 bucket statistics to status.json
Actions #3

Updated by Tom Morris about 8 years ago

  • Assigned To set to Tom Clegg
  • Target version changed from Arvados Future Sprints to 2016-11-23 sprint
  • Story points set to 1.0
Actions #4

Updated by Tom Morris about 8 years ago

Not to dive into implementation details, but introducing a general purpose metrics package may provide benefits going forward.

I've used the Dropwizard metrics in the past and like them. A Go port of that package can be found here: https://github.com/armon/go-metrics

There's also: https://github.com/rcrowley/go-metrics which is described in this blog post: http://blog.dekstroza.io/monitoring-software-written-in-golang/

S3 Error codes are listed here: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

Actions #5

Updated by Tom Clegg about 8 years ago

  • Subject changed from [keepstore] Add S3 bucket statistics to status.json to [keepstore] Add S3 bucket statistics and current #clients to status.json
Actions #6

Updated by Tom Clegg about 8 years ago

  • Status changed from New to In Progress
Actions #7

Updated by Radhika Chippada about 8 years ago

  • I am unable to run tests. Seeing this error when I invoke run-tests with install: (with go version go1.7 linux/amd64)
  ********** Running services/keepstore install **********

package context: unrecognized import path "context" (import path does not begin with hostname)

    ********** !!!!!! services/keepstore install FAILED !!!!!! **********
  • In s3_volume.go -> IndexTo method, why invoke stats.tick twice in a row at lines 459 and 460?
Actions #8

Updated by Tom Clegg about 8 years ago

Radhika Chippada wrote:

  • I am unable to run tests. Seeing this error when I invoke run-tests with install: (with go version go1.7 linux/amd64)

package context: unrecognized import path "context" (import path does not begin with hostname)

Hm, not sure what to make of this. https://blog.golang.org/go1.7 says context was added to stdlib in go1.7. Does it help to reinstall (1.7.3 is current)? Maybe make sure you don't have a GOROOT pointing to an older Go installation?

  • In s3_volume.go -> IndexTo method, why invoke stats.tick twice in a row at lines 459 and 460?

One for dataL.First() and one for recentL.First(), both on the following line...

Actions #9

Updated by Radhika Chippada almost 8 years ago

Tom said:

Maybe make sure you don't have a GOROOT pointing to an older Go installation

Yes, this was the cause. Tests are passing now.

LGTM

Actions #10

Updated by Tom Clegg almost 8 years ago

Actions #11

Updated by Radhika Chippada almost 8 years ago

Branch 10484-count-clients LGTM

Actions #12

Updated by Tom Clegg almost 8 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 67 to 100

Applied in changeset arvados|commit:d13a369065b4e72047c8c267d6ceaa5c84d50c3d.

Actions #13

Updated by Tom Morris almost 7 years ago

  • Related to Feature #13025: [keepstore] runtime metrics endpoint added
Actions

Also available in: Atom PDF