Project

General

Profile

Hacking Python SDK » History » Revision 17

Revision 16 (Brett Smith, 09/19/2014 02:32 PM) → Revision 17/25 (Brett Smith, 09/19/2014 02:57 PM)

h1. Hacking Python SDK 

 {{toc}} 

 h2. Prerequisites 

 The FUSE driver requires associated libraries to build: 

 <pre> 
 sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse 
 sudo adduser "$USER" fuse 
 sudo chmod g+rw /dev/fuse 
 sudo chown root:fuse /dev/fuse 
 </pre> 

 After installing @fuse@ and adding yourself to the @fuse@ group, you need to start a new login session. Make sure the @groups@ command reports that you're in the @fuse@ group. 

 h2. Get the source code 

 <pre> 
 cd 
 git clone https://github.com/curoverse/arvados.git 
 </pre> 

 h2. virtualenv 

 virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications.    The recommended way to deploy is to build a virtualenv for Arvados development. 

 To build the virtualenv, run: 

 <pre> 
 $ virtualenv --setuptools VENVDIR 
 </pre> 

 (@VENVDIR@ can be a directory anywhere you like, although best practice is to keep it outside your source directory.) 

 To set up the shell to use the isolated virtualenv environment, run: 

 <pre> 
 $ source VENVDIR/bin/activate 
 </pre> 

 To learn more about using and configuring virtualenv, read the "virtualenv usage documentation":https://virtualenv.pypa.io/en/latest/virtualenv.html#usage. 

 h2. Run tests 

 Strategy: 
 # Set up the environment to use a dedicated virtualenv 
 # Run the client library test suite 
 # Build a client library package and install it to the virtualenv 
 # Run the FUSE driver test suite 
 # Build a FUSE driver package and install it to the virtualenv 

 Note: The test suite brings up a Keep server and an API server to run tests against. For best results: 
 * Try [[Hacking Keep]] and [[Hacking API Server]] to make sure you have all the right dependencies for running the Keep and API servers. 
 * Make sure you have a blob_signing_key in services/api/config/application.yml 
 * Install the keepstore binary. 
 ** Make sure your GOPATH points somewhere, e.g.: @export GOPATH=~/gocode; mkdir -p $GOPATH@ 
 ** Install keepstore: @go get git.curoverse.com/arvados.git/services/keepstore@ 
 ** (if you don't do anything special, this fetches "master" from git.curoverse.com -- if you want to build a version of keepstore with local modifications, see [[Hacking Keep]]) 

 Script (make sure to edit the first line to refer to your virtualenv): 

 <pre> 
 source VENVDIR/bin/activate 

 cd ~/arvados/sdk/python 
 python setup.py test 
 python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz 
 pip install dist/arvados-python-client-0.1.*.tar.gz 

 cd ~/arvados/services/fuse 
 python setup.py test 
 python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz 
 pip install dist/arvados_fuse-0.1.*.tar.gz 
 </pre> 

 h3. Run a single test or test class 

 <pre> 
 source VENVDIR/bin/activate 
 cd ~/arvados/sdk/python 

 # One test module 
 python setup.py test --test-suite tests.test_keep_locator 

 # One test class 
 python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest 

 # One test case 
 python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators 
 </pre> 

 h2. Builds and versioning 

 When we build each Python package, the version number is @0.1.$(git log --format=format:%ct.%h -n1 .)@, where the current directory is the package directory.    In plain English, it looks at the most recent git commit that affected the package, and then the version number is @0.1.[commit's timestamp in seconds since the Epoch].[commit's short Epoch].[short commit hash]@. 

 This means that if you want to make changes in one package and refer to it somewhere else (e.g., have the FUSE package depend on a specific SDK version), you should commit your changes to the original package before anything else.    Then you can consistently refer to the metadata from that dedicated commit in subsequent changes. 

 h2. Logging 

 The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages.    The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the @ARVADOS_DEBUG@ setting).    Other SDK modules and command-line tools should @import arvados@ and then send messages to a logger under "arvados" to ensure consistent log handling.    Typical setup looks like this: 

 <pre><code class="python"> 
 import arvados 
 import logging 

 logger = logging.getLogger('arvados.YOURTHING') 
 </code></pre> 

 Once you've set this up, you can send messages to the logger using methods like @logger.debug()@ and @logger.error()@.    See the "Logger class documentation":https://docs.python.org/2/library/logging.html#logger-objects for full details. 

 Command-line scripts may reconfigure the @arvados.logger@ object based on additional configuration like command-line switches.    @services/fuse/bin/arv-mount@ demonstrates adjusting the level and setting a custom log handler. 

 h2. Python buffer protocol 

 Notes on managing buffers efficiently in Python, we don't use this in the python sdk as of this writing (but we might). 

 http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/ 

 Example using bytearray() to allocate a buffer, memoryview() to create a writable slice, and readinto() to write directly to the buffer slice: 

 <pre> 
 >>> b = bytearray(20) 
 >>> c = memoryview(b) 
 >>> f = open("python.txt", "r") 
 >>> f.readinto(c[5:10]) 
 5 
 >>> b 
 bytearray(b"\x00\x00\x00\x00\x00I\'ve \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00") 
 </pre>