Arvados

Changes at arvados-workbench2|1e6d5bc0ce92a7575030b299193b99cbad50e810 branch 16073-process-io-panels
Tests developer-tests-workbench2: #815

• Adds the process IO panels
• Changes integration test config to TrustAllContent: true for inline image tests
• Includes "Empty value" display when params are missing
• Handles files and file arrays of files with secondary files

• For each input/output I want to be able to navigate to the collection in workbench in addition to linking to the file directly
  • in workbench 1, this was done by making the portable data hash part separately clickable from the file, e.g.
    keep:portable_data_hash+123 /dir/file

• When I click on a workflow step, the "Inputs" and "Outputs" are empty. We should put something useful here, or hide them, not just take up space to say "No parameters found"
  • In the future, we could store the CWL input parameters for individual steps on the container request, then Inputs would be able to show something more useful -- but we don't have that yet
  • For now, the "Inputs" could be populated by going through "mounts" and providing supplying links to the ones that are {"kind": "collection"}
  • The "Outputs" should display the contents of the collection in "output_uuid" -- could we embed a read-only version of the file browser component used on the Collection panel? Or at least display a link to navigate to the Collection panel.
  • The "mounts" and "output collection contents" views should be additional tab on Inputs/Outputs -- so they are available in the workflow view as well

• The "Details" panel has "Inputs" and "Outputs" links.
  • Remove the "Inputs" link, I think what it used to do was open the "Inputs" dialog, which we're replacing here.
  • Change the text of the "Outputs" to use the name of the output collection in output_uuid.

• I didn't get a chance to test how image thumbnails work, do you have an example you were using to test with?

• Split links potentially confusing, pdh leads to wb2 collection view, filename links to keepweb preview.
• Rename label to Name (shows the input ID)
• Show label and doc in separate columns
• Add type column
• Add size column
• Add column selector to disable doc / type / size(?) columns
• Lazy load images when scrolling to images
• Collapse image to a small height with expandability - should we add a global collapse/expand all images button?
• Hide empty tabs (Empty input mounts on main process input card, preview/raw on subprocess input and output if empty)
• Add sort by param name
• Reduce space between table header row and dividing line
• Use inbox/outbox icons for io cards
• Rename Input Mounts and Output Collection tabs to Collections and Collection
• Rename Raw tab to JSON
• Rename Preview tab to Parameters
• Add link to collection in output collection tab
• Make tabs symmetrical

Things that are hopefully easy to do:

• Rename label to Name (shows the input ID)
• Collapse image to a small height with expandability - should we add a global collapse/expand all images button?
• Hide empty tabs (Empty input mounts on main process input card, preview/raw on subprocess input and output if empty)
• Reduce space between table header row and dividing line
• Use inbox/outbox icons for io cards
• Rename Input Mounts and Output Collection tabs to Collections and Collection
• Rename Raw tab to JSON
• Rename Preview tab to Parameters
• Add link to collection in output collection tab
• Make tabs symmetrical

This is probably easy to change but I don't know what we want to change it to?

• Split links potentially confusing, pdh leads to wb2 collection view, filename links to keepweb preview.

Things that seem like more effort and we should think about some more:

• Show label and doc in separate columns
• Add type column
• Add size column
• Add column selector to disable doc / type / size(?) columns
• Lazy load images when scrolling to images
• Add sort by param name

Hi all -

Collection Pane:
I have been thinking about this - and I think I have a solution to one of the issues that has been bugging me. I really like the preview and the raw/JSON but I find the collection view pretty unsatisfying since it is just a poor view into a normal collection page. I think for the main workflow you don't need it because you see the inputs and outputs in the preview and if you have the link there you can go to the collections (see below how I would like to do that). The argument is for steps we aren't ready yet to have the preview pane because we don't have the values. Then I wouldn't have the Input pane at all until we are ready to have it properly.

All we are showing is the collection view which is confusing for everyone. When Peter gets the code working to see the step inputs then we view like we do the workflow inputs.

Hyperlink and Columns:

I strongly feel that (1) you need to break up the link (2) You can fix this by making three columns. I would just make that Name, Value and Collection Link/Collection/Keep something like that. Name - would be what is now shown as ID. Value - would be just the tail end of the link that allows you to download it , for images the preview and name of the file, or for single values the existing view. Collection Link - Would be the full URl that links you access it in Keep. That I think would solve the issue and it would seem obvious to me that clicking the file name would download it and clicking the full path would go to the collection. To make it even more obvious -- could you add a hover that tell you that when you click you are either downloading it or going to the collection.

Max and Minimum of Graphic Peek:
I think you best would be just a toggle button near the X that allows you to toggle on and off the image view. If the image view is off - then the image view is off - then it would either just show the name of the file hyperlink and when on then show it. You could probably use an icon like this. https://www.shutterstock.com/search/view-icon or https://thenounproject.com/icon/photo-library-230056/ -- and have the view OFF by default.

Small note:
I probably would center the name of the image under the image instead of left justified but if we have a habit of always left justifying in the entire WB2 then keep it.

Also - for the labels/doc -- I understand that there is more work making another panel but you should plan for (1) an input/output having both a label and doc and which one you will be showing - as a user I would expect by label to show before my doc (2) if only doc is present, then cut it off at a reasonable size otherwise there will be a mess.

A good example to use perhaps is our https://github.com/arvados/arvados-tutorial/blob/main/WGS-processing/cwl/wgs-processing-wf.cwl which has labels but not doc.

Hi all -

For #1 -- it seems like a workaround for us to do the proper thing and offer a way to show or not show columns which is what I would suggest. It would be good for other panels, so I think we should implement it now. I am not happy with connecting them since Descriptions are very large. If you force me to do something I would just do Label - but seriously I think we need an option to select or not select columns instead of doing these workarounds.

For #2 --
For vertical - I am not sure why they can't adjust to size and that what I would want. But if we can't do that - I would adjust them to some reasonable number of inputs / outputs lets say 5 and then do a maximize button with both having a scroll wheel. Ideally for either inputs/outputs the panel would not appear and/or would appear quite small if there are no inputs or outputs. I am not sure what the question is here. But this is how I would like it to behave.

Also having an eye with a cross through it doesn't mean minimize and maximize (or preview images?)-- it looks like "don't want to see" and I would be confused as a user with the difference between a crossed out eye and the X.

Also as a side note -- there is no button to minimize a panel after you maximize it -- which is super annoying and hard to figure out.

From discussion:

For both inputs and outputs:

1. Eliminate "description" column
2. Hide the "label" column if all the values are blank.
3. Set max-height to half height and add the maximize button which makes it full height
4. When the inputs or outputs is entirely empty, the panel should take up minimal space (not a 100px icon that says "nothing here!")
5. The file browser for outputs should also toggle half height / full height instead of always full height.

The behavior of the maximize button will be addressed in ticket #19531, so for this ticket we add a maximize button but it follows the existing "maximize" behavior that the log and subprocess panels uses.

Changes at ce2ea2b32bdbaa8a9156ca80f8bf2eb33308ba89 16073-process-io-panels

The current issue is the extra space in the output panel collection file browser - since the panel isn't very tall at 50% when un-maximized, I've set the scroll area to scroll past the path and search box to give more space when scrolling.

When viewing this process with a few output files ce8i5-xvhdp-es8zwqbfkks11u6 I tried reducing the min-height of wrapper and dataWrapper, but then the autosizer seems to simply use that min-height as the height for the list of files even though the contents are taller, creating a second scroll bar and not auto-expanding to the content.

It seems if I remove any height restrictions, the auto sizer components collapse to 0 height - hiding all of the content and not propping open the parent elements, so I'm not sure how to get the file browser to behave more like a normal div and just be as tall as the content, which would let the user scroll within the panel without extra space at the end.

Also, I added an empty styles override section in process-output-collection-files.ts for adding styles to the file browser that apply only to the process output panel

Changes at arvados-workbench2|80052fec619ff95dda77e5b9b030fd39c8d26057

Tests developer-tests-workbench2: #924

• Removed doc column
• Use materialdesignicons for input/output icons
• Remove icon from "no parameters" message
• Hide tabs when not relevant
• Conditionally hide label column when there are no labels
• Reduce collection file browser padding to save space
• Add maximize to io panels
• Update tests

Input panel -> JSON¶

[
  {
    "type": "File",
    "id": "#main/clinvarvcf",
    "label": "Reference VCF for ClinVar",
    "value": {
      "basename": "clinvar.vcf.gz",
      "class": "File",
      "format": "http://edamontology.org/format_3016",
      "location": "keep:18657d75efb4afd31a14bb204d073239+13611/clinvar.vcf.gz",
      "size": 25161134
    }
  },

This is the wrong format. It should be the plain contents from mounts[/var/lib/cwl/inputs.json], e.g.

{
  "clinvarvcf": {
      "basename": "clinvar.vcf.gz",
      "class": "File",
      "format": "http://edamontology.org/format_3016",
      "location": "keep:18657d75efb4afd31a14bb204d073239+13611/clinvar.vcf.gz",
      "size": 25161134    
  }
...
}

Parameters->Value column¶

If this is a link to a file, I need to be able right-click and select "copy link" or "open link in new tab". The problem here is that it is an <a> tag without href, it uses a onClick to navigate. Either it needs a correct href or we need to provide own context menu on right click.

Parameters->Collection column¶

You can drop the "keep:" prefix to save space.

We might want to set the column to draw an ellipsis when it is going to overflow to the right.

Other Parameters column notes¶

I think Sarah mentioned this, we could reduce the whitespace separating all columns bet 30%-50% so that things fit better by default.

The label column feels really crowded. It should be wider.

Performance¶

I feel like it is taking much longer for the entire process panel to load. It kind of feels like it is loading them one after another instead of spinning out a bunch of asynchronous calls and then handling them as they come in. In particular, the logs panel doesn't seem to load contents until after the outputs panel has loaded contents.

I wish these panels started by rendering a spinner that made it really clear that it was waiting for something to load, not that there wasn't any content. Because outputs say "No parameters" and logs says "no logs" when actually it is waiting to get a response back.

Changes at arvados-workbench2|927f7b8743df91a181130328857123c72e52ebd2
Tests developer-tests-workbench2: #931

• Correctly fetch raw inputs without processing
• Use link element for keep-web links to allow open in new tab
• Remove keep: prefix from io param pdh links
• Reduce process io preview table padding
• Add min-width to label column
• Add loading indicator for preview tab
• Parallelize loading of output requests

I ended up not adding ellipses to the collection column since the column sizing is fluid, so it's hard to have the last column shrink without assigning static sizes to the other columns / keep them fluid.

I'd like to merge this sooner rather than later but another thought I had is when showing a list of files, if the collection is the same for every one (which is especially almost always going to be the case for output parameters) it should just render the link on the first row and leave the subsequent rows blank.

I agree that secondary files should be linked to the primary inputs. I like the idea of having them indented. Also you could do a pull down where you can shut them if they are too annoying or a toggle button to turn on/off showing them - something this -- https://thenounproject.com/icon/linked-154435/

Stephen Smith wrote in #note-57:

Changes at arvados-workbench2|3c6ea6ded33b5740035d059d7d9812ed951ab215
Tests developer-tests-workbench2: #958

• Add display of parameters from container request properties
• Hide preview tab when workflow definition is absent from mounts
• Filter out objects from process properties chips
• Maintain ordering of secondary files below main files
• Indent secondary files
• Hide PDH links on secondary files which have the same pdh as the main file
• Also hides PDH of files in file array when pdh matches the first file in the array

I am extremely motivated to get this merged, so I'm limiting this to the most important things --

1. I really like how the secondary files are listed now, but there is way too much empty space between the lines. Digging into the styling it looks like there's a row height of 48px which I'd consider cutting in half to 24px
2. It seems like if there is no content for "Outputs" (for example, the container was cancelled) it will spin forever. In that case it should actually say "Nothing here"
3. How hard would it be to use the "DefaultCodeSnippet" control to render the input/output JSON tab? It would be handy to have it turn the uuids and pdhs into hyperlinks. However if this is more than 5-10 minutes of work, let me know.

Changes at arvados-workbench2|e4db3c32b3bc4590ce43357739d95b6d2784f9f9
Tests developer-tests-workbench2: #968

• Fix infinite loading on empty i/o
• Fix showing no params when definitions are present for "Empty Value"
• Add guards against null files
• Display root collection directories as "/"
• Use default code snippet for auto linking json view
• Shrink secondary value row height
• Reduce padding between card title and tabs

arvados-workbench2|e4db3c32b3bc4590ce43357739d95b6d2784f9f9

• you need to merge main

• Containers which are queued/cancelled before running seem to still have a forever spinner on the outputs panel. Example: ce8i5-xvhdp-iyoj9wffcpitf4b
  If the output_uuid field is null I'm not sure we should show the outputs panel at all.

• Here, the contents of the inputs panel displayed briefly and then disappeared. Strangely, if I hit "refresh" then it is displayed. After refreshing, the inline image preview doesn't seem to be working here:

ce8i5-xvhdp-54ttl3sfnma366y

Changes at arvados-workbench2|868a6881bba9107eaf25e2b055dec55bf1fda42d
Tests developer-tests-workbench2: #972

• Merged main
• Fixed incorrectly setting empty inputs to prevent infinite spinner
• Avoid applying empty inputs when inputs already loaded when mounts temporarily disappear while process executes
• Cache output definition and only allow loading updated definitions to avoid missing mounts causing definitions to disappear (should show a preview of output parameters with "No value" while process is still executing)

Stephen Smith wrote in #note-62:

Changes at arvados-workbench2|868a6881bba9107eaf25e2b055dec55bf1fda42d
Tests developer-tests-workbench2: #972

• Merged main
• Fixed incorrectly setting empty inputs to prevent infinite spinner
• Avoid applying empty inputs when inputs already loaded when mounts temporarily disappear while process executes
• Cache output definition and only allow loading updated definitions to avoid missing mounts causing definitions to disappear (should show a preview of output parameters with "No value" while process is still executing)

This LGTM!

changes at arvados-workbench2|c8d7659860af3b12d87ef29478bc1b825a9b3d2f
Tests developer-tests-workbench2: #977

• Refactored useeffects into actions and reducers to use the store instead of component state - this avoids infinite loop issues when checking current state for updates (now done in the reducer) and simplifies clearing state between page changes (added a reset process panel action that gets called at the beginning of each loadProessPanel)

Merged in arvados-workbench2|576cd73ae073187557df3239ca301c16eaca7ae9