Arvados

h1. Hacking Python SDK

{{toc}}

h2. Prerequisites

The FUSE driver requires associated libraries to build:

<pre>

sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse

sudo adduser "$USER" fuse

sudo chmod g+rw /dev/fuse

sudo chown root:fuse /dev/fuse

</pre>

After installing @fuse@ and adding yourself to the @fuse@ group, you need to start a new login session. Make sure the @groups@ command reports that you're in the @fuse@ group.

h2. Get the source code

<pre>

cd

git clone https://github.com/curoverse/arvados.git

</pre>

h2. virtualenv

virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications.  The recommended way to deploy is to build a virtualenv for Arvados development.

To build the virtualenv, run:

<pre>

$ virtualenv --setuptools VENVDIR

</pre>

(@VENVDIR@ can be a directory anywhere you like, although best practice is to keep it outside your source directory.)

To set up the shell to use the isolated virtualenv environment, run:

<pre>

$ source VENVDIR/bin/activate

</pre>

To learn more about using and configuring virtualenv, read the "virtualenv usage documentation":https://virtualenv.pypa.io/en/latest/virtualenv.html#usage.

h2. Run tests

Strategy:

# Set up the environment to use a dedicated virtualenv

# Run the client library test suite

# Build a client library package and install it to the virtualenv

# Run the FUSE driver test suite

# Build a FUSE driver package and install it to the virtualenv

Note: The test suite brings up a Keep server and an API server to run tests against. For best results:

* Try [[Hacking Keep]] and [[Hacking API Server]] to make sure you have all the right dependencies for running the Keep and API servers.

* Make sure you have a blob_signing_key in services/api/config/application.yml

* Install the keepstore binary.

** Make sure your GOPATH points somewhere, e.g.: @export GOPATH=~/gocode; mkdir -p $GOPATH@

** Install keepstore: @go get git.curoverse.com/arvados.git/services/keepstore@

** (if you don't do anything special, this fetches "master" from git.curoverse.com -- if you want to build a version of keepstore with local modifications, see [[Hacking Keep]])

Script (make sure to edit the first line to refer to your virtualenv):

<pre>

source VENVDIR/bin/activate

cd ~/arvados/sdk/python

python setup.py test

python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz

pip install dist/arvados-python-client-0.1.*.tar.gz

cd ~/arvados/services/fuse

python setup.py test

python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz

pip install dist/arvados_fuse-0.1.*.tar.gz

</pre>

h3. Run a single test or test class

<pre>

source VENVDIR/bin/activate

cd ~/arvados/sdk/python

# One test module

python setup.py test --test-suite tests.test_keep_locator

# One test class

python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest

# One test case

python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators

</pre>

h2. Logging

The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages.  The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the @ARVADOS_DEBUG@ setting).  Other SDK modules and command-line tools should @import arvados@ and then send messages to a logger under "arvados" to ensure consistent log handling.  Typical setup looks like this:

<pre><code class="python">

import arvados

import logging

logger = logging.getLogger('arvados.YOURTHING')

</code></pre>

Once you've set this up, you can send messages to the logger using methods like @logger.debug()@ and @logger.error()@.  See the "Logger class documentation":https://docs.python.org/2/library/logging.html#logger-objects for full details.

Command-line scripts may reconfigure the @arvados.logger@ object based on additional configuration like command-line switches.  @services/fuse/bin/arv-mount@ demonstrates adjusting the level and setting a custom log handler.

h2. Python buffer protocol

Notes on managing buffers efficiently in Python, we don't use this in the python sdk as of this writing (but we might).

http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/

Example using bytearray() to allocate a buffer, memoryview() to create a writable slice, and readinto() to write directly to the buffer slice:

<pre>

>>> b = bytearray(20)

>>> c = memoryview(b)

>>> f = open("python.txt", "r")

>>> f.readinto(c[5:10])

5

>>> b

bytearray(b"\x00\x00\x00\x00\x00I\'ve \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")

</pre>