Project

General

Profile

Actions

Idea #12707

closed

[API] Add columns for desired/actual storage classes for each collection

Added by Tom Clegg over 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
1.0
Release relationship:
Auto

Description

Similar to replication_* fields:
  • storage_classes_desired (jsonb, array of strings, default ["default"])
  • storage_classes_confirmed (jsonb, array of strings, default [])
  • storage_classes_confirmed_at (timestamp, default null)

Don't include any of these columns in the fulltext index.


Subtasks 1 (0 open1 closed)

Task #13084: Review 12707-storage-classes-columnsResolvedPeter Amstutz02/22/2018Actions

Related issues

Blocks Arvados - Feature #11184: [Keep] Support multiple storage classesResolvedTom MorrisActions
Blocks Arvados - Idea #12708: [keep-balance] Move blocks to satisfy storage_classes_desiredResolvedTom Clegg04/19/2018Actions
Actions #1

Updated by Tom Clegg over 6 years ago

  • Tracker changed from Bug to Feature
Actions #2

Updated by Tom Clegg over 6 years ago

  • Blocks Feature #11184: [Keep] Support multiple storage classes added
Actions #3

Updated by Tom Clegg over 6 years ago

  • Blocks Idea #12708: [keep-balance] Move blocks to satisfy storage_classes_desired added
Actions #4

Updated by Tom Clegg over 6 years ago

  • Tracker changed from Feature to Task
  • Description updated (diff)
Actions #5

Updated by Tom Clegg over 6 years ago

  • Description updated (diff)
Actions #6

Updated by Tom Clegg over 6 years ago

  • Tracker changed from Task to Idea
  • Story points set to 1.0
Actions #7

Updated by Tom Morris about 6 years ago

  • Target version changed from To Be Groomed to 2018-02-28 Sprint
Actions #8

Updated by Lucas Di Pentima about 6 years ago

  • Assigned To set to Lucas Di Pentima
Actions #9

Updated by Lucas Di Pentima about 6 years ago

  • Status changed from New to In Progress
Actions #11

Updated by Peter Amstutz about 6 years ago

For some reason, your structure.sql added "Tablespace" to every comment, what version of Postgres are you using?

(This creates enough noise in the diff I had trouble finding what actually changed)

- -- Name: api_client_authorizations; Type: TABLE; Schema: public; Owner: -
+ -- Name: api_client_authorizations; Type: TABLE; Schema: public; Owner: -; Tablespace: 

Rest look good.

Actions #12

Updated by Lucas Di Pentima about 6 years ago

Fixed the issue by upgrading to PostgreSQL 9.6 (I was running both 9.4 & 9.6 after upgrading to stretch, as suggested by Peter)
Rebased - b0d7f37097e8b5c56bfbfa79a134a8074789ebb4
Test run: https://ci.curoverse.com/job/developer-run-tests/610/

Actions #13

Updated by Peter Amstutz about 6 years ago

  • Is one of these redundant? If not, why not?
add_column :collections, :storage_classes_desired, :jsonb, :default => ["default"]
add_column :collections, :storage_classes_confirmed, :jsonb, :default => []
  before_validation :default_storage_classes, on: :create

  def default_storage_classes
    if self.storage_classes_desired.nil? || self.storage_classes_desired.empty?
      self.storage_classes_desired = ["default"]
    end
    self.storage_classes_confirmed ||= []
  end
  • I don't think we have used jsonb Array columns before (only Hash columns). Can you investigate which Arvados filters work with jsonb Array columns?
  • We should probably validate that the elements of storage_classes_desired and storage_classes_confirmed are only non-empty strings.
Actions #14

Updated by Lucas Di Pentima about 6 years ago

Updates at 281fe0c40c16aacd82cd1ca122e38aa3a3854f2e
Test run: https://ci.curoverse.com/job/developer-run-tests/619/

  • Removed default value setting on the model code.
  • Added validation on list element types (strings), with relevant tests.
  • Checked that 'exists' filter work on jsonb array by adding some tests.

Pending: Some workbench tests are failing due to storage_classes_desired being empty, investigating why this happens.

Actions #15

Updated by Lucas Di Pentima about 6 years ago

Updates at 05ececf63
Test run: https://ci.curoverse.com/job/developer-run-tests/620/

Added back the default value setting code on Collection's model as some workbench tests were failing. The default values on the schema will be useful for database migrations with already existing collections.

Actions #16

Updated by Peter Amstutz about 6 years ago

Lucas Di Pentima wrote:

Updates at 05ececf63
Test run: https://ci.curoverse.com/job/developer-run-tests/620/

Added back the default value setting code on Collection's model as some workbench tests were failing. The default values on the schema will be useful for database migrations with already existing collections.

This LGTM

Actions #17

Updated by Anonymous about 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100
Actions #18

Updated by Tom Morris over 5 years ago

  • Release set to 17
Actions

Also available in: Atom PDF