Arvados

I've been mulling over how best to have tabular metadata attached to collections using properties.

Assumptions / requirements / constraints¶

• A single collection contains multiple items (e.g. samples) to be described, each of those items has several fields/columns of data.
• data items are smallish, like filenames, identifiers from other systems, controlled vocabulary, dates, etc.
• data items are in the hundreds-to-thousands range
• our current generic indexing strategy on properties allows us to do exact search in multi-valued properties using "contains", however we can't do more complex filtering on multi-valued properties

Design¶

• In collection properties:
  • Meaningful "columns" field which has the list of other properties to display, in what order
  • each property listed in "columns" appears in properties with an array value. the array values should be the same length. same index in the array matches the same row across all the lists (this is a columnar data model)
• if workbench sees the "columns" field, it renders a table with properties listed in columns as a table column
  • client side sorting/filtering of table contents
  • can edit table contents but implemented as updating the entire set of properties at once
• server side ability to search for particular key-values across collections by e.g. ["properties.sample_id", "contains", "sample-123"]