Project

General

Profile

Hacking Python SDK » History » Version 13

Tom Clegg, 08/26/2014 09:43 AM

1 1 Tom Clegg
h1. Hacking Python SDK
2
3
{{toc}}
4
5
h2. Prerequisites
6
7 6 Brett Smith
The FUSE driver requires associated libraries to build:
8 4 Tom Clegg
9
<pre>
10 5 Tom Clegg
sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse
11
sudo adduser "$USER" fuse
12
sudo chmod g+rw /dev/fuse
13
sudo chown root:fuse /dev/fuse
14 1 Tom Clegg
</pre>
15 5 Tom Clegg
16
After installing @fuse@ and adding yourself to the @fuse@ group, you need to start a new login session. Make sure the @groups@ command reports that you're in the @fuse@ group.
17 4 Tom Clegg
18 1 Tom Clegg
h2. Get the source code
19
20
<pre>
21
cd
22
git clone https://github.com/curoverse/arvados.git
23
</pre>
24
25 6 Brett Smith
h2. virtualenv
26
27
virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications.  The recommended way to deploy is to build a virtualenv for Arvados development.
28
29
To build the virtualenv, run:
30
31
<pre>
32
$ virtualenv --setuptools VENVDIR
33
</pre>
34
35
(@VENVDIR@ can be a directory anywhere you like, although best practice is to keep it outside your source directory.)
36
37
To set up the shell to use the isolated virtualenv environment, run:
38
39
<pre>
40
$ source VENVDIR/bin/activate
41
</pre>
42
43
To learn more about using and configuring virtualenv, read the "virtualenv usage documentation":https://virtualenv.pypa.io/en/latest/virtualenv.html#usage.
44
45 1 Tom Clegg
h2. Run tests
46
47
Strategy:
48 6 Brett Smith
# Set up the environment to use a dedicated virtualenv
49 1 Tom Clegg
# Run the client library test suite
50
# Build a client library package and install it to the virtualenv
51
# Run the FUSE driver test suite
52
# Build a FUSE driver package and install it to the virtualenv
53
54
Note: The test suite brings up a Keep server and an API server to run tests against. For best results:
55
* Try [[Hacking Keep]] and [[Hacking API Server]] to make sure you have all the right dependencies for running the Keep and API servers.
56
* Make sure you have a blob_signing_key in services/api/config/application.yml
57 11 Tom Clegg
* Install the keepstore binary.
58
** Make sure your GOPATH points somewhere, e.g.: @export GOPATH=~/gocode; mkdir -p $GOPATH@
59
** Install keepstore: @go get git.curoverse.com/arvados.git/services/keepstore@
60
** (if you don't do anything special, this fetches "master" from git.curoverse.com -- if you want to build a version of keepstore with local modifications, see [[Hacking Keep]])
61 4 Tom Clegg
62 6 Brett Smith
Script (make sure to edit the first line to refer to your virtualenv):
63 1 Tom Clegg
64
<pre>
65 6 Brett Smith
source VENVDIR/bin/activate
66 1 Tom Clegg
67
cd ~/arvados/sdk/python
68 9 Tom Clegg
python setup.py test
69 7 Brett Smith
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
70 6 Brett Smith
pip install dist/arvados-python-client-0.1.*.tar.gz
71 1 Tom Clegg
72
cd ~/arvados/services/fuse
73 9 Tom Clegg
python setup.py test
74 7 Brett Smith
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
75 6 Brett Smith
pip install dist/arvados_fuse-0.1.*.tar.gz
76 1 Tom Clegg
</pre>
77 10 Brett Smith
78 12 Tom Clegg
h3. Run a single test or test class
79
80
<pre>
81
source VENVDIR/bin/activate
82
cd ~/arvados/sdk/python
83
84 13 Tom Clegg
# One test module
85 12 Tom Clegg
python setup.py test --test-suite tests.test_keep_locator
86 1 Tom Clegg
87 13 Tom Clegg
# One test class
88
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest
89
90
# One test case
91 12 Tom Clegg
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators
92
</pre>
93
94 10 Brett Smith
h2. Logging
95
96
The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages.  The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the @ARVADOS_DEBUG@ setting).  Other SDK modules and command-line tools should @import arvados@ and then send messages to a logger under "arvados" to ensure consistent log handling.  Typical setup looks like this:
97
98
<pre><code class="python">
99
import arvados
100
import logging
101
102
logger = logging.getLogger('arvados.YOURTHING')
103
</code></pre>
104
105
Once you've set this up, you can send messages to the logger using methods like @logger.debug()@ and @logger.error()@.  See the "Logger class documentation":https://docs.python.org/2/library/logging.html#logger-objects for full details.
106
107
Command-line scripts may reconfigure the @arvados.logger@ object based on additional configuration like command-line switches.  @services/fuse/bin/arv-mount@ demonstrates adjusting the level and setting a custom log handler.