Story #12707

[API] Add columns for desired/actual storage classes for each collection

Added by Tom Clegg over 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
02/22/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0
Release relationship:
Auto

Description

Similar to replication_* fields:
  • storage_classes_desired (jsonb, array of strings, default ["default"])
  • storage_classes_confirmed (jsonb, array of strings, default [])
  • storage_classes_confirmed_at (timestamp, default null)

Don't include any of these columns in the fulltext index.


Subtasks

Task #13084: Review 12707-storage-classes-columnsResolvedPeter Amstutz


Related issues

Blocks Arvados - Feature #11184: [Keep] Support multiple storage classesIn Progress

Blocks Arvados - Story #12708: [keep-balance] Move blocks to satisfy storage_classes_desiredResolved04/19/2018

Associated revisions

Revision 704dd826
Added by Lucas Di Pentima over 2 years ago

Merge branch '12707-storage-classes-columns'
Closes #12707

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

History

#1 Updated by Tom Clegg over 2 years ago

  • Tracker changed from Bug to Feature

#2 Updated by Tom Clegg over 2 years ago

  • Blocks Feature #11184: [Keep] Support multiple storage classes added

#3 Updated by Tom Clegg over 2 years ago

  • Blocks Story #12708: [keep-balance] Move blocks to satisfy storage_classes_desired added

#4 Updated by Tom Clegg over 2 years ago

  • Tracker changed from Feature to Task
  • Description updated (diff)

#5 Updated by Tom Clegg over 2 years ago

  • Description updated (diff)

#6 Updated by Tom Clegg over 2 years ago

  • Tracker changed from Task to Story
  • Story points set to 1.0

#7 Updated by Tom Morris over 2 years ago

  • Target version changed from To Be Groomed to 2018-02-28 Sprint

#8 Updated by Lucas Di Pentima over 2 years ago

  • Assigned To set to Lucas Di Pentima

#9 Updated by Lucas Di Pentima over 2 years ago

  • Status changed from New to In Progress

#11 Updated by Peter Amstutz over 2 years ago

For some reason, your structure.sql added "Tablespace" to every comment, what version of Postgres are you using?

(This creates enough noise in the diff I had trouble finding what actually changed)

- -- Name: api_client_authorizations; Type: TABLE; Schema: public; Owner: -
+ -- Name: api_client_authorizations; Type: TABLE; Schema: public; Owner: -; Tablespace: 

Rest look good.

#12 Updated by Lucas Di Pentima over 2 years ago

Fixed the issue by upgrading to PostgreSQL 9.6 (I was running both 9.4 & 9.6 after upgrading to stretch, as suggested by Peter)
Rebased - b0d7f37097e8b5c56bfbfa79a134a8074789ebb4
Test run: https://ci.curoverse.com/job/developer-run-tests/610/

#13 Updated by Peter Amstutz over 2 years ago

  • Is one of these redundant? If not, why not?
add_column :collections, :storage_classes_desired, :jsonb, :default => ["default"]
add_column :collections, :storage_classes_confirmed, :jsonb, :default => []
  before_validation :default_storage_classes, on: :create

  def default_storage_classes
    if self.storage_classes_desired.nil? || self.storage_classes_desired.empty?
      self.storage_classes_desired = ["default"]
    end
    self.storage_classes_confirmed ||= []
  end
  • I don't think we have used jsonb Array columns before (only Hash columns). Can you investigate which Arvados filters work with jsonb Array columns?
  • We should probably validate that the elements of storage_classes_desired and storage_classes_confirmed are only non-empty strings.

#14 Updated by Lucas Di Pentima over 2 years ago

Updates at 281fe0c40c16aacd82cd1ca122e38aa3a3854f2e
Test run: https://ci.curoverse.com/job/developer-run-tests/619/

  • Removed default value setting on the model code.
  • Added validation on list element types (strings), with relevant tests.
  • Checked that 'exists' filter work on jsonb array by adding some tests.

Pending: Some workbench tests are failing due to storage_classes_desired being empty, investigating why this happens.

#15 Updated by Lucas Di Pentima over 2 years ago

Updates at 05ececf63
Test run: https://ci.curoverse.com/job/developer-run-tests/620/

Added back the default value setting code on Collection's model as some workbench tests were failing. The default values on the schema will be useful for database migrations with already existing collections.

#16 Updated by Peter Amstutz over 2 years ago

Lucas Di Pentima wrote:

Updates at 05ececf63
Test run: https://ci.curoverse.com/job/developer-run-tests/620/

Added back the default value setting code on Collection's model as some workbench tests were failing. The default values on the schema will be useful for database migrations with already existing collections.

This LGTM

#17 Updated by Anonymous over 2 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

#18 Updated by Tom Morris about 2 years ago

  • Release set to 17

Also available in: Atom PDF