Story #2803

Python SDK and FUSE driver are separate packages

Added by Brett Smith over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
-
Target version:
2014-05-28 Pipeline Factory
Start date:
05/09/2014
Due date:
% Done:

100%

Estimated time:
(Total: 0.50 h)
Story points:
1.0

Description

These are conceptually two different packages, and our efforts to share build infrastructure are requiring weirder and wilder hacks to keep it going. Properly separating them will keep our infrastructure clean and avoid bugs caused by pip tripping over all our unorthodox choices.

Subtasks

Task #2804: Review 2803-python-package-split-wipResolvedTom Clegg

Associated revisions

Revision a2d52e56 (diff)
Added by Brett Smith over 7 years ago

2803: Split arvados and arvados_fuse Python modules.

Prior commits tried to make separate SDK and FUSE packages from the
same sdk/python source tree. However, this didn't work as intended
once the packages were installed. Python modules don't work like Ruby
namespaces. If you ask Python to import arvados.fuse, it finds the
arvados module, then looks for the fuse submodule under it. So you
can't have arvados.fuse installed somewhere completely differently.

In addition, Python packages assume the installation script is called
setup.py. So they wouldn't find setup_fuse.py.

We still want to have the FUSE module separate, so in discussion on
IRC we decided that the least worst option was to rename the module to
arvados_fuse. This commit implements that. If accepted, the new
build procedure will be:

$ python setup.py egg_info -b \
    ".$(git log --format=format:%ct.%h -n1 .)" sdist upload

Refs #2803.

Revision 1217dbbe (diff)
Added by Brett Smith over 7 years ago

2803: Relax Python library requirements.

Refs #2803. We're generally undemanding on the libraries we pull in,
and this can help us use more OS-packaged libraries.

Revision 60c604d8 (diff)
Added by Brett Smith over 7 years ago

2803: Split arvados and arvados_fuse Python modules.

Prior commits tried to make separate SDK and FUSE packages from the
same sdk/python source tree. However, this didn't work as intended
once the packages were installed. Python modules don't work like Ruby
namespaces. If you ask Python to import arvados.fuse, it finds the
arvados module, then looks for the fuse submodule under it. So you
can't have arvados.fuse installed somewhere completely differently.

In addition, Python packages assume the installation script is called
setup.py. So they wouldn't find setup_fuse.py.

We still want to have the FUSE module separate, so in discussion on
IRC we decided that the least worst option was to rename the module to
arvados_fuse. This commit implements that. If accepted, the new
build procedure will be:

$ python setup.py egg_info -b \
    ".$(git log --format=format:%ct.%h -n1 .)" sdist upload

Refs #2803.

Revision b20333af (diff)
Added by Brett Smith over 7 years ago

2803: Relax Python library requirements.

Refs #2803. We're generally undemanding on the libraries we pull in,
and this can help us use more OS-packaged libraries.

Revision 949e27e0
Added by Brett Smith over 7 years ago

Merge branch '2803-python-package-split-wip'

Closes #2803, #2804.

History

#1 Updated by Brett Smith over 7 years ago

  • Description updated (diff)
  • Assigned To set to Brett Smith

#2 Updated by Tom Clegg over 7 years ago

Notes
  • Assigning the git-rev suffix in the "build+upload" command line makes much more sense than hacking it into setup.py. Thanks for that.
  • Unless we change /COPYING, moving arv-mount from /sdk/ to /services/ changes its license from Apache2 to GPL. Unless we want this for some reason, I think we should either update /COPYING or (for a simpler license↔tree map) put arv-mount in /sdk/fuse (or...?).
  • Perhaps change package name to just "arvados-fuse"? (Whatever we do, it's nice to have symmetry between package name, directory name, and module name -- except that I wouldn't necessarily want lots of directories in arvados called arvados-something.)

Test procedure (not including Keep server), starting with no arvados package installed in system dirs:

sudo apt-get remove python-yaml
sudo apt-get install libyaml-dev

export PYTHONBASE=/tmp/pythonpath
mkdir -p $PYTHONBASE
export PYTHONPATH=$PYTHONBASE/lib/python2.7/site-packages

cd ~/arvados/sdk/python
pip install --install-option="--prefix=$PYTHONBASE" -r requirements.txt
python -m unittest discover
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist
pip install --install-option="--prefix=$PYTHONBASE" dist/arvados-python-client-0.1.*.tar.gz

cd ~/arvados/services/fuse
pip install --install-option="--prefix=$PYTHONBASE" -r <(grep -v arvados requirements.txt)
python -m unittest discover
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist
pip install --install-option="--prefix=$PYTHONBASE" dist/arvados_fuse-0.1.*.tar.gz

And it all worked!

#3 Updated by Brett Smith over 7 years ago

[I believe all of this is per discussion in IRC.]

Tom Clegg wrote:

  • Unless we change /COPYING, moving arv-mount from /sdk/ to /services/ changes its license from Apache2 to GPL. Unless we want this for some reason, I think we should either update /COPYING or (for a simpler license↔tree map) put arv-mount in /sdk/fuse (or...?).

A license change seems OK. Updated setup.py to match.

  • Perhaps change package name to just "arvados-fuse"? (Whatever we do, it's nice to have symmetry between package name, directory name, and module name -- except that I wouldn't necessarily want lots of directories in arvados called arvados-something.)

Filename is now /services/fuse, and both the package and module are named arvados_fuse.

#4 Updated by Brett Smith over 7 years ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:949e27e0dd0a97cf78f0b90636df387fa0d65be3.

Also available in: Atom PDF