Feature #12522

[Composer] Browse Arvados git repos

Added by Peter Amstutz almost 2 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
12/05/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

The "My Projects" side bar should be populated with a list of the Arvados git repositories visible to the user.

This requires accessing the Arvados API. Currently, there isn't a formal Javascript SDK, however it can be accessed with typical AJAX requests.

API server request syntax is described here:

http://doc.arvados.org/api/requests.html

The repositories resource is described here:

http://doc.arvados.org/api/methods/repositories.html

When the user clicks on a repository in the projects panel, it should load the contents of the git repository over HTTP and display the tree of files from the master branch.

When the user clicks on a file in the projects panel, it should open the file in a new tab in the content pane.

At least for the initial release, we are using these simplifying assumptions:

  • all CWL workflows are stored in Arvados-hosted git repositories
  • CWL workflows can have references to external tools/workflows in the same git repository+version.
  • arvados git web service is available

The Arvados git web service can authenticate HTTP basic authentication (username "nobdoy", password is api token) as supported by js-git.


Subtasks

Task #12535: ReviewResolvedPeter Amstutz

Task #12828: Fix savingResolvedPeter Amstutz

Task #12829: Load on demandResolvedPeter Amstutz

Task #12830: SpinnerResolvedPeter Amstutz

Task #12831: Refresh buttonResolvedPeter Amstutz

Task #12968: Review 12522-git-refreshResolvedPeter Amstutz

Task #13024: Testing & feedbackClosedBryan Cosca

History

#1 Updated by Peter Amstutz almost 2 years ago

  • Subject changed from [Composer] Browse git repos to [Composer] Browse Arvados git repos
  • Description updated (diff)
  • Assigned To set to Redzo Dzakmic

#2 Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)

#3 Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)

#4 Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2017-11-08 Sprint to 2017-11-22 Sprint

#5 Updated by Peter Amstutz almost 2 years ago

  • Status changed from New to In Progress
  • Target version changed from 2017-11-22 Sprint to 2017-12-06 Sprint

#6 Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2017-12-06 Sprint to 2017-12-20 Sprint

#7 Updated by Peter Amstutz over 1 year ago

Current issues with repository browsing:

  • Repositories are displayed under the heading "Local Files" which is wrong. Change the label or move "repositories" up one level in the tree
  • It currently clones the repositories immediately when the "repositories" folder is expanded. This needs to change so that it only clones a repository when the user clicks on it
  • When cloning a repository, the web app essentially freezes. If possible it should do more of the work as background tasks. It should also display a spinner to indicate something is happening.

#8 Updated by Peter Amstutz over 1 year ago

Also, there needs to be a "refresh" button to re-read the contents of a repository.

#9 Updated by Peter Amstutz over 1 year ago

  • Assigned To changed from Redzo Dzakmic to Peter Amstutz

#10 Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2017-12-20 Sprint to 2018-01-17 Sprint

#11 Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2018-01-17 Sprint to 2018-01-31 Sprint

#12 Updated by Lucas Di Pentima over 1 year ago

  • Indentation is off all over the place, maybe it’s just my editor that is configured incorrectly? Should I set it up in some way or maybe we should correct the indentation as it’s sometimes difficult to follow the code.
    • Further checking various modified files’ logs, it seems that the bad indentation commits started on early 2018, maybe it’s a good data point to find the issue. Do you see the indentation correctly on your machine?
  • File composer/src/app/core/global/arvados-global.service.ts
    • Imports from lines 6, 7, 8, 9, 11, & 12 seem to be unnecessary
  • When opening an app, the status bar shows a “Checking for app updates…” that’s not removed (don’t know if it does something) until closing the cwl file. It happened once to me that this message was still there after closing all files, couldn’t reproduce it again
  • App name accepts something like ‘somename/other.cwl’, auto creating the ‘somename’ dir if it doesn’t exist. Don’t know if this is intended.

Repo refreshing worked OK on my tests.

#13 Updated by Peter Amstutz over 1 year ago

Lucas Di Pentima wrote:

  • Indentation is off all over the place, maybe it’s just my editor that is configured incorrectly? Should I set it up in some way or maybe we should correct the indentation as it’s sometimes difficult to follow the code.
    • Further checking various modified files’ logs, it seems that the bad indentation commits started on early 2018, maybe it’s a good data point to find the issue. Do you see the indentation correctly on your machine?

Sorry about that, my editor wasn't auto-converting tabs to spaces. I re-indented and re-saved all the affected files.

  • File composer/src/app/core/global/arvados-global.service.ts
    • Imports from lines 6, 7, 8, 9, 11, & 12 seem to be unnecessary

Fixed.

  • When opening an app, the status bar shows a “Checking for app updates…” that’s not removed (don’t know if it does something) until closing the cwl file. It happened once to me that this message was still there after closing all files, couldn’t reproduce it again

Thanks, I commented that out.

  • App name accepts something like ‘somename/other.cwl’, auto creating the ‘somename’ dir if it doesn’t exist. Don’t know if this is intended.

Not intended, but if it is doing something reasonable I'm okay with it. Although now I wonder what happens if you try to create "." and ".." files, which would probably cause problems.

Repo refreshing worked OK on my tests.

I think this is feature complete now. Intended behavior of "synchronize" button (tested manually, you should too):

  • Expanded folders remain expanded
  • Double clicking on an updated (file, tool, workflow) displays the content from the most recent sync
  • Open tabs displaying a (file, tool, workflow) which are not modified by the user are refreshed from latest sync
  • Open tabs displaying a (file, tool, workflow) which have been modified by the user are left alone

The save button interacts with sync this way:

  • When saving, commit and push. If this goes through, we have the latest version.
  • If the upstream rejects the push, do a sync, then create a new commit on the new head (effectively a rebase) and try to push again.
  • The sync behavior described above applies

#14 Updated by Lucas Di Pentima over 1 year ago

Tried creating a new file with name '../../../hello.txt' and it created it and showed as '..' are valid subdir names. When pulling the repo on my computer I got a message like this:

$ git pull
WARNING: 'lucas/lucas4xphq' is an alias for '4xphq-s0uqq-1cqslhxvx2496fq'
remote: Counting objects: 15, done.
remote: Compressing objects: 100% (7/7), done.
Unpacking objects: 100% (15/15), done.
remote: Total 15 (delta 3), reused 0 (delta 0)
From git.4xphq.arvadosapi.com:lucas/lucas4xphq
   1808652..8c8141f  master     -> origin/master
Updating 1808652..8c8141f
error: Invalid path '../../../hello.txt'
Fast-forward
 ../../../hello.txt | 1 +
 test3.txt          | 4 ++++
 2 files changed, 5 insertions(+)
 create mode 100644 ../../../hello.txt
 create mode 100644 test3.txt

The result is that the file wasn't really created on my machine (not even on ../../../ path) and a delete operation is staged for commit. Maybe this should be filed as a separate bug.

More related to the sync feature, I found a strange behavior that may need some fixing:
1. Open composer and expand a repository with files
2. On the computer's clone, remove a file, commit & push
3. On composer double-click on the removed file that still exist on the sidebar: the sidebar will get updated but the file will be opened on a tab with its contents.
4. If the save button is pressed, it'll get re-committed.

#15 Updated by Peter Amstutz over 1 year ago

Feedback from sprint review:

  • If possible, notify to the user when saving over a remotely modified document
  • When refreshing, could have dialog offering whether to save or revert document
  • Refresh automatically by polling?

#16 Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2018-01-31 Sprint to 2018-02-14 Sprint

#17 Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2018-02-14 Sprint to 2018-02-28 Sprint

#18 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-02-28 Sprint to 2018-03-14 Sprint

#19 Updated by Tom Morris over 1 year ago

  • Status changed from In Progress to Closed

#20 Updated by Tom Morris about 1 year ago

  • Release set to 17

Also available in: Atom PDF