Project

General

Profile

Hacking Python SDK » History » Version 18

Brett Smith, 09/29/2014 04:53 PM
setup.py now takes care of version tagging

1 1 Tom Clegg
h1. Hacking Python SDK
2
3
{{toc}}
4
5
h2. Prerequisites
6
7 6 Brett Smith
The FUSE driver requires associated libraries to build:
8 4 Tom Clegg
9
<pre>
10 5 Tom Clegg
sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse
11
sudo adduser "$USER" fuse
12
sudo chmod g+rw /dev/fuse
13
sudo chown root:fuse /dev/fuse
14 1 Tom Clegg
</pre>
15 5 Tom Clegg
16
After installing @fuse@ and adding yourself to the @fuse@ group, you need to start a new login session. Make sure the @groups@ command reports that you're in the @fuse@ group.
17 4 Tom Clegg
18 1 Tom Clegg
h2. Get the source code
19
20
<pre>
21
cd
22
git clone https://github.com/curoverse/arvados.git
23
</pre>
24
25 6 Brett Smith
h2. virtualenv
26
27
virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications.  The recommended way to deploy is to build a virtualenv for Arvados development.
28
29
To build the virtualenv, run:
30
31
<pre>
32
$ virtualenv --setuptools VENVDIR
33
</pre>
34
35
(@VENVDIR@ can be a directory anywhere you like, although best practice is to keep it outside your source directory.)
36
37
To set up the shell to use the isolated virtualenv environment, run:
38
39
<pre>
40
$ source VENVDIR/bin/activate
41
</pre>
42
43
To learn more about using and configuring virtualenv, read the "virtualenv usage documentation":https://virtualenv.pypa.io/en/latest/virtualenv.html#usage.
44
45 1 Tom Clegg
h2. Run tests
46
47
Strategy:
48 6 Brett Smith
# Set up the environment to use a dedicated virtualenv
49 1 Tom Clegg
# Run the client library test suite
50
# Build a client library package and install it to the virtualenv
51
# Run the FUSE driver test suite
52
# Build a FUSE driver package and install it to the virtualenv
53
54
Note: The test suite brings up a Keep server and an API server to run tests against. For best results:
55
* Try [[Hacking Keep]] and [[Hacking API Server]] to make sure you have all the right dependencies for running the Keep and API servers.
56
* Make sure you have a blob_signing_key in services/api/config/application.yml
57 11 Tom Clegg
* Install the keepstore binary.
58
** Make sure your GOPATH points somewhere, e.g.: @export GOPATH=~/gocode; mkdir -p $GOPATH@
59
** Install keepstore: @go get git.curoverse.com/arvados.git/services/keepstore@
60
** (if you don't do anything special, this fetches "master" from git.curoverse.com -- if you want to build a version of keepstore with local modifications, see [[Hacking Keep]])
61 4 Tom Clegg
62 6 Brett Smith
Script (make sure to edit the first line to refer to your virtualenv):
63 1 Tom Clegg
64
<pre>
65 6 Brett Smith
source VENVDIR/bin/activate
66 1 Tom Clegg
67
cd ~/arvados/sdk/python
68 9 Tom Clegg
python setup.py test
69 18 Brett Smith
python setup.py install
70 1 Tom Clegg
71
cd ~/arvados/services/fuse
72 7 Brett Smith
python setup.py test
73 18 Brett Smith
python setup.py install
74 1 Tom Clegg
</pre>
75 10 Brett Smith
76 12 Tom Clegg
h3. Run a single test or test class
77
78
<pre>
79
source VENVDIR/bin/activate
80
cd ~/arvados/sdk/python
81
82 13 Tom Clegg
# One test module
83 12 Tom Clegg
python setup.py test --test-suite tests.test_keep_locator
84 1 Tom Clegg
85 13 Tom Clegg
# One test class
86
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest
87
88
# One test case
89 12 Tom Clegg
python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators
90 1 Tom Clegg
</pre>
91 12 Tom Clegg
92
h2. Builds and versioning
93 15 Brett Smith
94 18 Brett Smith
When we build each Python package, the version number is generated from the most recent commit that affected the package.  The format is @0.1.[commit's timestamp formatted as %Y%m%d%H%M%S].[commit's short hash]@.
95 15 Brett Smith
96 18 Brett Smith
If you want to make changes in one package and refer to it somewhere else (e.g., have the FUSE package depend on a specific SDK version), you should commit your changes to the original package before anything else.  Then you can consistently refer to the metadata from that dedicated commit in subsequent changes.
97 15 Brett Smith
98 10 Brett Smith
h2. Logging
99
100
The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages.  The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the @ARVADOS_DEBUG@ setting).  Other SDK modules and command-line tools should @import arvados@ and then send messages to a logger under "arvados" to ensure consistent log handling.  Typical setup looks like this:
101
102
<pre><code class="python">
103
import arvados
104
import logging
105
106
logger = logging.getLogger('arvados.YOURTHING')
107
</code></pre>
108
109
Once you've set this up, you can send messages to the logger using methods like @logger.debug()@ and @logger.error()@.  See the "Logger class documentation":https://docs.python.org/2/library/logging.html#logger-objects for full details.
110
111
Command-line scripts may reconfigure the @arvados.logger@ object based on additional configuration like command-line switches.  @services/fuse/bin/arv-mount@ demonstrates adjusting the level and setting a custom log handler.
112 14 Peter Amstutz
113
h2. Python buffer protocol
114
115
Notes on managing buffers efficiently in Python, we don't use this in the python sdk as of this writing (but we might).
116
117
http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/
118
119
Example using bytearray() to allocate a buffer, memoryview() to create a writable slice, and readinto() to write directly to the buffer slice:
120
121
<pre>
122
>>> b = bytearray(20)
123
>>> c = memoryview(b)
124
>>> f = open("python.txt", "r")
125
>>> f.readinto(c[5:10])
126
5
127
>>> b
128
bytearray(b"\x00\x00\x00\x00\x00I\'ve \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")
129
</pre>