Story #2803
Python SDK and FUSE driver are separate packages
100%
Description
These are conceptually two different packages, and our efforts to share build infrastructure are requiring weirder and wilder hacks to keep it going. Properly separating them will keep our infrastructure clean and avoid bugs caused by pip tripping over all our unorthodox choices.
Subtasks
Associated revisions
2803: Relax Python library requirements.
Refs #2803. We're generally undemanding on the libraries we pull in,
and this can help us use more OS-packaged libraries.
2803: Split arvados and arvados_fuse Python modules.
Prior commits tried to make separate SDK and FUSE packages from the
same sdk/python source tree. However, this didn't work as intended
once the packages were installed. Python modules don't work like Ruby
namespaces. If you ask Python to import arvados.fuse, it finds the
arvados module, then looks for the fuse submodule under it. So you
can't have arvados.fuse installed somewhere completely differently.
In addition, Python packages assume the installation script is called
setup.py. So they wouldn't find setup_fuse.py.
We still want to have the FUSE module separate, so in discussion on
IRC we decided that the least worst option was to rename the module to
arvados_fuse. This commit implements that. If accepted, the new
build procedure will be:
$ python setup.py egg_info -b \
".$(git log --format=format:%ct.%h -n1 .)" sdist upload
Refs #2803.
2803: Relax Python library requirements.
Refs #2803. We're generally undemanding on the libraries we pull in,
and this can help us use more OS-packaged libraries.
History
#1
Updated by Brett Smith about 8 years ago
- Description updated (diff)
- Assigned To set to Brett Smith
#2
Updated by Tom Clegg about 8 years ago
- Assigning the git-rev suffix in the "build+upload" command line makes much more sense than hacking it into setup.py. Thanks for that.
- Unless we change /COPYING, moving arv-mount from /sdk/ to /services/ changes its license from Apache2 to GPL. Unless we want this for some reason, I think we should either update /COPYING or (for a simpler license↔tree map) put arv-mount in /sdk/fuse (or...?).
- Perhaps change package name to just "arvados-fuse"? (Whatever we do, it's nice to have symmetry between package name, directory name, and module name -- except that I wouldn't necessarily want lots of directories in arvados called arvados-something.)
Test procedure (not including Keep server), starting with no arvados package installed in system dirs:
sudo apt-get remove python-yaml sudo apt-get install libyaml-dev export PYTHONBASE=/tmp/pythonpath mkdir -p $PYTHONBASE export PYTHONPATH=$PYTHONBASE/lib/python2.7/site-packages cd ~/arvados/sdk/python pip install --install-option="--prefix=$PYTHONBASE" -r requirements.txt python -m unittest discover python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist pip install --install-option="--prefix=$PYTHONBASE" dist/arvados-python-client-0.1.*.tar.gz cd ~/arvados/services/fuse pip install --install-option="--prefix=$PYTHONBASE" -r <(grep -v arvados requirements.txt) python -m unittest discover python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist pip install --install-option="--prefix=$PYTHONBASE" dist/arvados_fuse-0.1.*.tar.gz
And it all worked!
#3
Updated by Brett Smith about 8 years ago
[I believe all of this is per discussion in IRC.]
Tom Clegg wrote:
- Unless we change /COPYING, moving arv-mount from /sdk/ to /services/ changes its license from Apache2 to GPL. Unless we want this for some reason, I think we should either update /COPYING or (for a simpler license↔tree map) put arv-mount in /sdk/fuse (or...?).
A license change seems OK. Updated setup.py
to match.
- Perhaps change package name to just "arvados-fuse"? (Whatever we do, it's nice to have symmetry between package name, directory name, and module name -- except that I wouldn't necessarily want lots of directories in arvados called arvados-something.)
Filename is now /services/fuse
, and both the package and module are named arvados_fuse
.
#4
Updated by Brett Smith about 8 years ago
- Status changed from New to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:949e27e0dd0a97cf78f0b90636df387fa0d65be3.
2803: Split arvados and arvados_fuse Python modules.
Prior commits tried to make separate SDK and FUSE packages from the
same sdk/python source tree. However, this didn't work as intended
once the packages were installed. Python modules don't work like Ruby
namespaces. If you ask Python to import arvados.fuse, it finds the
arvados module, then looks for the fuse submodule under it. So you
can't have arvados.fuse installed somewhere completely differently.
In addition, Python packages assume the installation script is called
setup.py. So they wouldn't find setup_fuse.py.
We still want to have the FUSE module separate, so in discussion on
IRC we decided that the least worst option was to rename the module to
arvados_fuse. This commit implements that. If accepted, the new
build procedure will be:
Refs #2803.