Hacking Python SDK » History » Version 10

« Previous - Version 10/24 (diff) - Next » - Current version
Brett Smith, 08/13/2014 01:38 PM
Add logging section

Hacking Python SDK


The FUSE driver requires associated libraries to build:

sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse
sudo adduser "$USER" fuse
sudo chmod g+rw /dev/fuse
sudo chown root:fuse /dev/fuse

After installing fuse and adding yourself to the fuse group, you need to start a new login session. Make sure the groups command reports that you're in the fuse group.

Get the source code

git clone https://github.com/curoverse/arvados.git


virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications. The recommended way to deploy is to build a virtualenv for Arvados development.

To build the virtualenv, run:

$ virtualenv --setuptools VENVDIR

(VENVDIR can be a directory anywhere you like, although best practice is to keep it outside your source directory.)

To set up the shell to use the isolated virtualenv environment, run:

$ source VENVDIR/bin/activate

To learn more about using and configuring virtualenv, read the virtualenv usage documentation.

Run tests

  1. Set up the environment to use a dedicated virtualenv
  2. Run the client library test suite
  3. Build a client library package and install it to the virtualenv
  4. Run the FUSE driver test suite
  5. Build a FUSE driver package and install it to the virtualenv
Note: The test suite brings up a Keep server and an API server to run tests against. For best results:
  • Try Hacking Keep and Hacking API Server to make sure you have all the right dependencies for running the Keep and API servers.
  • Make sure you have a blob_signing_key in services/api/config/application.yml
  • Build Keep binary. cd .../arvados/services/keep; ./go.sh install keep

Script (make sure to edit the first line to refer to your virtualenv):

source VENVDIR/bin/activate

cd ~/arvados/sdk/python
python setup.py test
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
pip install dist/arvados-python-client-0.1.*.tar.gz

cd ~/arvados/services/fuse
python setup.py test
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
pip install dist/arvados_fuse-0.1.*.tar.gz


The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages. The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the ARVADOS_DEBUG setting). Other SDK modules and command-line tools should import arvados and then send messages to a logger under "arvados" to ensure consistent log handling. Typical setup looks like this:

import arvados
import logging

logger = logging.getLogger('arvados.YOURTHING')

Once you've set this up, you can send messages to the logger using methods like logger.debug() and logger.error(). See the Logger class documentation for full details.

Command-line scripts may reconfigure the arvados.logger object based on additional configuration like command-line switches. services/fuse/bin/arv-mount demonstrates adjusting the level and setting a custom log handler.