Hacking Python SDK » History » Version 10

Brett Smith, 08/13/2014 01:38 PM
Add logging section

1 1 Tom Clegg
h1. Hacking Python SDK
2 1 Tom Clegg
3 1 Tom Clegg
{{toc}}
4 1 Tom Clegg
5 1 Tom Clegg
h2. Prerequisites
6 1 Tom Clegg
7 6 Brett Smith
The FUSE driver requires associated libraries to build:
8 4 Tom Clegg
9 4 Tom Clegg
<pre>
10 5 Tom Clegg
sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse
11 5 Tom Clegg
sudo adduser "$USER" fuse
12 5 Tom Clegg
sudo chmod g+rw /dev/fuse
13 5 Tom Clegg
sudo chown root:fuse /dev/fuse
14 1 Tom Clegg
</pre>
15 5 Tom Clegg
16 5 Tom Clegg
After installing @fuse@ and adding yourself to the @fuse@ group, you need to start a new login session. Make sure the @groups@ command reports that you're in the @fuse@ group.
17 4 Tom Clegg
18 1 Tom Clegg
h2. Get the source code
19 1 Tom Clegg
20 1 Tom Clegg
<pre>
21 1 Tom Clegg
cd
22 1 Tom Clegg
git clone https://github.com/curoverse/arvados.git
23 1 Tom Clegg
</pre>
24 1 Tom Clegg
25 6 Brett Smith
h2. virtualenv
26 6 Brett Smith
27 6 Brett Smith
virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications.  The recommended way to deploy is to build a virtualenv for Arvados development.
28 6 Brett Smith
29 6 Brett Smith
To build the virtualenv, run:
30 6 Brett Smith
31 6 Brett Smith
<pre>
32 6 Brett Smith
$ virtualenv --setuptools VENVDIR
33 6 Brett Smith
</pre>
34 6 Brett Smith
35 6 Brett Smith
(@VENVDIR@ can be a directory anywhere you like, although best practice is to keep it outside your source directory.)
36 6 Brett Smith
37 6 Brett Smith
To set up the shell to use the isolated virtualenv environment, run:
38 6 Brett Smith
39 6 Brett Smith
<pre>
40 6 Brett Smith
$ source VENVDIR/bin/activate
41 6 Brett Smith
</pre>
42 6 Brett Smith
43 6 Brett Smith
To learn more about using and configuring virtualenv, read the "virtualenv usage documentation":https://virtualenv.pypa.io/en/latest/virtualenv.html#usage.
44 6 Brett Smith
45 1 Tom Clegg
h2. Run tests
46 1 Tom Clegg
47 1 Tom Clegg
Strategy:
48 6 Brett Smith
# Set up the environment to use a dedicated virtualenv
49 1 Tom Clegg
# Run the client library test suite
50 1 Tom Clegg
# Build a client library package and install it to the virtualenv
51 1 Tom Clegg
# Run the FUSE driver test suite
52 1 Tom Clegg
# Build a FUSE driver package and install it to the virtualenv
53 1 Tom Clegg
54 1 Tom Clegg
Note: The test suite brings up a Keep server and an API server to run tests against. For best results:
55 1 Tom Clegg
* Try [[Hacking Keep]] and [[Hacking API Server]] to make sure you have all the right dependencies for running the Keep and API servers.
56 1 Tom Clegg
* Make sure you have a blob_signing_key in services/api/config/application.yml
57 9 Tom Clegg
* Build Keep binary. @cd .../arvados/services/keep; ./go.sh install keep@
58 4 Tom Clegg
59 6 Brett Smith
Script (make sure to edit the first line to refer to your virtualenv):
60 1 Tom Clegg
61 1 Tom Clegg
<pre>
62 6 Brett Smith
source VENVDIR/bin/activate
63 1 Tom Clegg
64 1 Tom Clegg
cd ~/arvados/sdk/python
65 9 Tom Clegg
python setup.py test
66 7 Brett Smith
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
67 6 Brett Smith
pip install dist/arvados-python-client-0.1.*.tar.gz
68 1 Tom Clegg
69 1 Tom Clegg
cd ~/arvados/services/fuse
70 9 Tom Clegg
python setup.py test
71 7 Brett Smith
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
72 6 Brett Smith
pip install dist/arvados_fuse-0.1.*.tar.gz
73 1 Tom Clegg
</pre>
74 10 Brett Smith
75 10 Brett Smith
h2. Logging
76 10 Brett Smith
77 10 Brett Smith
The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages.  The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the @ARVADOS_DEBUG@ setting).  Other SDK modules and command-line tools should @import arvados@ and then send messages to a logger under "arvados" to ensure consistent log handling.  Typical setup looks like this:
78 10 Brett Smith
79 10 Brett Smith
<pre><code class="python">
80 10 Brett Smith
import arvados
81 10 Brett Smith
import logging
82 10 Brett Smith
83 10 Brett Smith
logger = logging.getLogger('arvados.YOURTHING')
84 10 Brett Smith
</code></pre>
85 10 Brett Smith
86 10 Brett Smith
Once you've set this up, you can send messages to the logger using methods like @logger.debug()@ and @logger.error()@.  See the "Logger class documentation":https://docs.python.org/2/library/logging.html#logger-objects for full details.
87 10 Brett Smith
88 10 Brett Smith
Command-line scripts may reconfigure the @arvados.logger@ object based on additional configuration like command-line switches.  @services/fuse/bin/arv-mount@ demonstrates adjusting the level and setting a custom log handler.