Project

General

Profile

Actions

Feature #12320

closed

Access Arvados projects under the "by_id" directory in keep mount

Added by Abram Connelly over 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
0.5
Release:
Release relationship:
Auto

Description

One can access collections either by their portable data hash or UUID from the "by_id" directory in the keep mount. Trying to do the same for an Arvados project will fail with a "no such file or directory".

It would be nice to be able to access Arvados projects by their UUID under the keep "by_id" directory.

Implementation details:

  1. The code is in arvados/services/fuse
  2. The mount point configuration is initialized in arvados_fuse/command.py
  3. The "by_id" directory is implemented by the MagicDirectory class in arvados_fuse/fusedir.py
  4. The MagicDirectory is special, subdirectories only appear in the listing after they are first accessed
  5. The MagicDirectory.__contains__ method is called to determine if a subdirectory exists (named by portable data hash or uuid) and creates on demand it if necessary
  6. Currently it creates CollectionDirectory object for collections.
  7. The goal of this task is to add support to MagicDirectory for accessing projects (ProjectDirectory) by uuid.

Tests are in arvados/services/fuse/tests

The simplest way to run them is using arvbox on Linux (note, won't work on Windows/MacOS so in that case you will need to use a VM).

$ arvbox start test --only services/fuse

Alternately you can set up your workstation or a VM following these instructions: https://dev.arvados.org/projects/arvados/wiki/Hacking_prerequisites (but personally I use arvbox for all my development).

The test framework starts up the API server in testing mode with a known database, so some tests rely on the API server returning known values. These test values are defined in arvados/services/api/test/fixtures. You can probably use an existing record, such as "aproject" (uuid zzzzz-j7d0g-v955i6s2oi1cbso) defined groups.yml.

As a starting point, try looking at FuseMagicTest in arvados/services/fuse/tests/test_mount.py

To run a single test:

$ arvbox start test --only services/fuse services/fuse_test=--test-suite=tests.test_mount.FuseMagicTest

Subtasks 1 (0 open1 closed)

Task #13226: Review 12320-access-projects-from-by_idResolvedFuad Muhic09/25/2017Actions
Actions #1

Updated by Tom Morris over 6 years ago

  • Target version set to Arvados Future Sprints
Actions #2

Updated by Tom Morris over 6 years ago

  • Story points set to 0.5
Actions #3

Updated by Tom Morris about 6 years ago

  • Assigned To set to Fuad Muhic
  • Target version changed from Arvados Future Sprints to 2018-03-28 Sprint
Actions #4

Updated by Peter Amstutz about 6 years ago

  • Description updated (diff)
Actions #5

Updated by Tom Morris about 6 years ago

  • Status changed from New to In Progress
  • Target version changed from 2018-03-28 Sprint to 2018-04-11 Sprint
Actions #6

Updated by Lucas Di Pentima about 6 years ago

Reviewing branch 12320-access-projects-from-by_id at 1f15c5b6060b0d82c9b17fb724b7128374f65747

Just one question:

Should all types of groups be accessible on fuse mount? Maybe we’re only looking for the ones with [‘group_class’, ‘=‘, ‘project’] (just in case, double check with Tom or Peter)

Actions #7

Updated by Fuad Muhic about 6 years ago

Lucas Di Pentima wrote:

Reviewing branch 12320-access-projects-from-by_id at 1f15c5b6060b0d82c9b17fb724b7128374f65747

Just one question:

Should all types of groups be accessible on fuse mount? Maybe we’re only looking for the ones with [‘group_class’, ‘=‘, ‘project’] (just in case, double check with Tom or Peter)

You are right, I fixed it.

Actions #8

Updated by Lucas Di Pentima about 6 years ago

Reviewing fcdea9d131e4ba823ac8b69224d90b8eb5f4ae2e

Do you think wouldn't be better to add a filter to the api call so that it isn't retrieved in the first place?

Actions #9

Updated by Fuad Muhic about 6 years ago

Lucas Di Pentima wrote:

Reviewing fcdea9d131e4ba823ac8b69224d90b8eb5f4ae2e

Do you think wouldn't be better to add a filter to the api call so that it isn't retrieved in the first place?

Do you mean to do a api group list with filter [‘group_class’, ‘=‘, ‘project’] and [‘uuid’, ‘=‘, k]?

Actions #10

Updated by Tom Morris about 6 years ago

  • Target version changed from 2018-04-11 Sprint to 2018-04-25 Sprint
Actions #11

Updated by Lucas Di Pentima about 6 years ago

Fuad Muhic wrote:

Do you think wouldn't be better to add a filter to the api call so that it isn't retrieved in the first place?

Do you mean to do a api group list with filter [‘group_class’, ‘=‘, ‘project’] and [‘uuid’, ‘=‘, k]?

Yes, I believe it would be better in terms of bandwidth usage, for example if some user tries to access lots of group uuids.

Actions #13

Updated by Fuad Muhic about 6 years ago

  • Status changed from In Progress to Resolved
Actions #14

Updated by Tom Morris almost 6 years ago

  • Release set to 13
Actions

Also available in: Atom PDF