Hacking Python SDK » History » Version 14

Peter Amstutz, 08/29/2014 09:10 PM

1 1 Tom Clegg
h1. Hacking Python SDK
2 1 Tom Clegg
3 1 Tom Clegg
{{toc}}
4 1 Tom Clegg
5 1 Tom Clegg
h2. Prerequisites
6 1 Tom Clegg
7 6 Brett Smith
The FUSE driver requires associated libraries to build:
8 4 Tom Clegg
9 4 Tom Clegg
<pre>
10 5 Tom Clegg
sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse
11 5 Tom Clegg
sudo adduser "$USER" fuse
12 5 Tom Clegg
sudo chmod g+rw /dev/fuse
13 5 Tom Clegg
sudo chown root:fuse /dev/fuse
14 1 Tom Clegg
</pre>
15 5 Tom Clegg
16 5 Tom Clegg
After installing @fuse@ and adding yourself to the @fuse@ group, you need to start a new login session. Make sure the @groups@ command reports that you're in the @fuse@ group.
17 4 Tom Clegg
18 1 Tom Clegg
h2. Get the source code
19 1 Tom Clegg
20 1 Tom Clegg
<pre>
21 1 Tom Clegg
cd
22 1 Tom Clegg
git clone https://github.com/curoverse/arvados.git
23 1 Tom Clegg
</pre>
24 1 Tom Clegg
25 6 Brett Smith
h2. virtualenv
26 6 Brett Smith
27 6 Brett Smith
virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications.  The recommended way to deploy is to build a virtualenv for Arvados development.
28 6 Brett Smith
29 6 Brett Smith
To build the virtualenv, run:
30 6 Brett Smith
31 6 Brett Smith
<pre>
32 6 Brett Smith
$ virtualenv --setuptools VENVDIR
33 6 Brett Smith
</pre>
34 6 Brett Smith
35 6 Brett Smith
(@VENVDIR@ can be a directory anywhere you like, although best practice is to keep it outside your source directory.)
36 6 Brett Smith
37 6 Brett Smith
To set up the shell to use the isolated virtualenv environment, run:
38 6 Brett Smith
39 6 Brett Smith
<pre>
40 6 Brett Smith
$ source VENVDIR/bin/activate
41 6 Brett Smith
</pre>
42 6 Brett Smith
43 6 Brett Smith
To learn more about using and configuring virtualenv, read the "virtualenv usage documentation":https://virtualenv.pypa.io/en/latest/virtualenv.html#usage.
44 6 Brett Smith
45 1 Tom Clegg
h2. Run tests
46 1 Tom Clegg
47 1 Tom Clegg
Strategy:
48 6 Brett Smith
# Set up the environment to use a dedicated virtualenv
49 1 Tom Clegg
# Run the client library test suite
50 1 Tom Clegg
# Build a client library package and install it to the virtualenv
51 1 Tom Clegg
# Run the FUSE driver test suite
52 1 Tom Clegg
# Build a FUSE driver package and install it to the virtualenv
53 1 Tom Clegg
54 1 Tom Clegg
Note: The test suite brings up a Keep server and an API server to run tests against. For best results:
55 1 Tom Clegg
* Try [[Hacking Keep]] and [[Hacking API Server]] to make sure you have all the right dependencies for running the Keep and API servers.
56 1 Tom Clegg
* Make sure you have a blob_signing_key in services/api/config/application.yml
57 11 Tom Clegg
* Install the keepstore binary.
58 11 Tom Clegg
** Make sure your GOPATH points somewhere, e.g.: @export GOPATH=~/gocode; mkdir -p $GOPATH@
59 11 Tom Clegg
** Install keepstore: @go get git.curoverse.com/arvados.git/services/keepstore@
60 11 Tom Clegg
** (if you don't do anything special, this fetches "master" from git.curoverse.com -- if you want to build a version of keepstore with local modifications, see [[Hacking Keep]])
61 4 Tom Clegg
62 6 Brett Smith
Script (make sure to edit the first line to refer to your virtualenv):
63 1 Tom Clegg
64 1 Tom Clegg
<pre>
65 6 Brett Smith
source VENVDIR/bin/activate
66 1 Tom Clegg
67 1 Tom Clegg
cd ~/arvados/sdk/python
68 9 Tom Clegg
python setup.py test
69 7 Brett Smith
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
70 6 Brett Smith
pip install dist/arvados-python-client-0.1.*.tar.gz
71 1 Tom Clegg
72 1 Tom Clegg
cd ~/arvados/services/fuse
73 9 Tom Clegg
python setup.py test
74 7 Brett Smith
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
75 6 Brett Smith
pip install dist/arvados_fuse-0.1.*.tar.gz
76 1 Tom Clegg
</pre>
77 10 Brett Smith
78 12 Tom Clegg
h3. Run a single test or test class
79 12 Tom Clegg
80 12 Tom Clegg
<pre>
81 12 Tom Clegg
source VENVDIR/bin/activate
82 12 Tom Clegg
cd ~/arvados/sdk/python
83 12 Tom Clegg
84 13 Tom Clegg
# One test module
85 12 Tom Clegg
python setup.py test --test-suite tests.test_keep_locator
86 1 Tom Clegg
87 13 Tom Clegg
# One test class
88 13 Tom Clegg
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest
89 13 Tom Clegg
90 13 Tom Clegg
# One test case
91 12 Tom Clegg
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators
92 12 Tom Clegg
</pre>
93 12 Tom Clegg
94 10 Brett Smith
h2. Logging
95 10 Brett Smith
96 10 Brett Smith
The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages.  The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the @ARVADOS_DEBUG@ setting).  Other SDK modules and command-line tools should @import arvados@ and then send messages to a logger under "arvados" to ensure consistent log handling.  Typical setup looks like this:
97 10 Brett Smith
98 10 Brett Smith
<pre><code class="python">
99 10 Brett Smith
import arvados
100 10 Brett Smith
import logging
101 10 Brett Smith
102 10 Brett Smith
logger = logging.getLogger('arvados.YOURTHING')
103 10 Brett Smith
</code></pre>
104 10 Brett Smith
105 10 Brett Smith
Once you've set this up, you can send messages to the logger using methods like @logger.debug()@ and @logger.error()@.  See the "Logger class documentation":https://docs.python.org/2/library/logging.html#logger-objects for full details.
106 10 Brett Smith
107 10 Brett Smith
Command-line scripts may reconfigure the @arvados.logger@ object based on additional configuration like command-line switches.  @services/fuse/bin/arv-mount@ demonstrates adjusting the level and setting a custom log handler.
108 14 Peter Amstutz
109 14 Peter Amstutz
h2. Python buffer protocol
110 14 Peter Amstutz
111 14 Peter Amstutz
Notes on managing buffers efficiently in Python, we don't use this in the python sdk as of this writing (but we might).
112 14 Peter Amstutz
113 14 Peter Amstutz
http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/
114 14 Peter Amstutz
115 14 Peter Amstutz
Example using bytearray() to allocate a buffer, memoryview() to create a writable slice, and readinto() to write directly to the buffer slice:
116 14 Peter Amstutz
117 14 Peter Amstutz
<pre>
118 14 Peter Amstutz
>>> b = bytearray(20)
119 14 Peter Amstutz
>>> c = memoryview(b)
120 14 Peter Amstutz
>>> f = open("python.txt", "r")
121 14 Peter Amstutz
>>> f.readinto(c[5:10])
122 14 Peter Amstutz
5
123 14 Peter Amstutz
>>> b
124 14 Peter Amstutz
bytearray(b"\x00\x00\x00\x00\x00I\'ve \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")
125 14 Peter Amstutz
</pre>