Hacking Python SDK¶

• Table of contents
• Hacking Python SDK
  • Prerequisites
  • Get the source code
  • virtualenv
  • Run tests
    • Run a single test or test class
  • Logging
  • Python buffer protocol

Prerequisites¶

The FUSE driver requires associated libraries to build:

sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse
sudo adduser "$USER" fuse
sudo chmod g+rw /dev/fuse
sudo chown root:fuse /dev/fuse

After installing fuse and adding yourself to the fuse group, you need to start a new login session. Make sure the groups command reports that you're in the fuse group.

Get the source code¶

cd
git clone https://github.com/curoverse/arvados.git

virtualenv¶

virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications. The recommended way to deploy is to build a virtualenv for Arvados development.

To build the virtualenv, run:

$ virtualenv --setuptools VENVDIR

(VENVDIR can be a directory anywhere you like, although best practice is to keep it outside your source directory.)

To set up the shell to use the isolated virtualenv environment, run:

$ source VENVDIR/bin/activate

To learn more about using and configuring virtualenv, read the virtualenv usage documentation.

Run tests¶

Strategy:

1. Set up the environment to use a dedicated virtualenv
2. Run the client library test suite
3. Build a client library package and install it to the virtualenv
4. Run the FUSE driver test suite
5. Build a FUSE driver package and install it to the virtualenv

Note: The test suite brings up a Keep server and an API server to run tests against. For best results:

• Try Hacking Keep and Hacking API Server to make sure you have all the right dependencies for running the Keep and API servers.
• Make sure you have a blob_signing_key in services/api/config/application.yml
• Install the keepstore binary.
  • Make sure your GOPATH points somewhere, e.g.: export GOPATH=~/gocode; mkdir -p $GOPATH
  • Install keepstore: go get git.curoverse.com/arvados.git/services/keepstore
  • (if you don't do anything special, this fetches "master" from git.curoverse.com -- if you want to build a version of keepstore with local modifications, see Hacking Keep)

Script (make sure to edit the first line to refer to your virtualenv):

source VENVDIR/bin/activate

cd ~/arvados/sdk/python
python setup.py test
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
pip install dist/arvados-python-client-0.1.*.tar.gz

cd ~/arvados/services/fuse
python setup.py test
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
pip install dist/arvados_fuse-0.1.*.tar.gz

Run a single test or test class¶

source VENVDIR/bin/activate
cd ~/arvados/sdk/python

# One test module
python setup.py test --test-suite tests.test_keep_locator

# One test class
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest

# One test case
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators

Logging¶

The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages. The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the ARVADOS_DEBUG setting). Other SDK modules and command-line tools should import arvados and then send messages to a logger under "arvados" to ensure consistent log handling. Typical setup looks like this:

import arvados
import logging

logger = logging.getLogger('arvados.YOURTHING')

Once you've set this up, you can send messages to the logger using methods like logger.debug() and logger.error(). See the Logger class documentation for full details.

Command-line scripts may reconfigure the arvados.logger object based on additional configuration like command-line switches. services/fuse/bin/arv-mount demonstrates adjusting the level and setting a custom log handler.

Python buffer protocol¶

Notes on managing buffers efficiently in Python, we don't use this in the python sdk as of this writing (but we might).

http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/

Example using bytearray() to allocate a buffer, memoryview() to create a writable slice, and readinto() to write directly to the buffer slice:

>>> b = bytearray(20)
>>> c = memoryview(b)
>>> f = open("python.txt", "r")
>>> f.readinto(c[5:10])
5
>>> b
bytearray(b"\x00\x00\x00\x00\x00I\'ve \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")