Hacking Python SDK¶
- Table of contents
- Hacking Python SDK
Prerequisites¶
The FUSE driver requires associated libraries to build:
sudo apt-get install libattr1-dev libfuse-dev pkg-config fuse sudo adduser "$USER" fuse sudo chmod g+rw /dev/fuse sudo chown root:fuse /dev/fuse
After installing fuse
and adding yourself to the fuse
group, you need to start a new login session. Make sure the groups
command reports that you're in the fuse
group.
Get the source code¶
cd git clone https://github.com/arvados/arvados.git
virtualenv¶
virtualenv helps you isolate the dependencies for a specific package or environment, much like Bundler does for our Rails applications. The recommended way to deploy is to build a virtualenv for Arvados development.
To build the virtualenv, run:
$ sudo apt-get install python-virtualenv $ virtualenv --setuptools VENVDIR
(VENVDIR
can be a directory anywhere you like, although best practice is to keep it outside your source directory.)
To set up the shell to use the isolated virtualenv environment, run:
$ source VENVDIR/bin/activate
To learn more about using and configuring virtualenv, read the virtualenv usage documentation.
Run tests¶
Strategy:- Set up the environment to use a dedicated virtualenv
- Run the client library test suite
- Build a client library package and install it to the virtualenv
- Run the FUSE driver test suite
- Build a FUSE driver package and install it to the virtualenv
- Try Hacking Keep and Hacking API Server to make sure you have all the right dependencies for running the Keep and API servers.
- Make sure you have a blob_signing_key in services/api/config/application.yml
- Install the keepstore binary.
- Make sure your GOPATH points somewhere, e.g.:
export GOPATH=~/gocode; mkdir -p $GOPATH
- Install keepstore:
go get git.arvados.org/arvados.git/services/keepstore
- (if you don't do anything special, this fetches "master" from git.arvados.org -- if you want to build a version of keepstore with local modifications, see Hacking Keep)
- Make sure your GOPATH points somewhere, e.g.:
Script (make sure to edit the first line to refer to your virtualenv):
apt-get install libcurl4-openssl-dev python-dev libssl-dev source VENVDIR/bin/activate cd ~/arvados/sdk/python python setup.py test python setup.py install cd ~/arvados/services/fuse python setup.py test python setup.py install
Run a single test or test class¶
source VENVDIR/bin/activate cd ~/arvados/sdk/python # One test module python setup.py test --test-suite tests.test_keep_locator # One test class python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest # One test case python setup.py test --test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators
Builds and versioning¶
When we build each Python package, the version number is generated from the most recent commit that affected the package. The format is 0.1.[commit's timestamp formatted as %Y%m%d%H%M%S].[commit's short hash]
.
If you want to make changes in one package and refer to it somewhere else (e.g., have the FUSE package depend on a specific SDK version), you should commit your changes to the original package before anything else. Then you can consistently refer to the metadata from that dedicated commit in subsequent changes.
Logging¶
The Python SDK uses Python's built-in logging module to log errors, warnings, and debug messages. The arvados module sets up logging for messages under "arvados" based on local configuration (e.g., the ARVADOS_DEBUG
setting). Other SDK modules and command-line tools should import arvados
and then send messages to a logger under "arvados" to ensure consistent log handling. Typical setup looks like this:
import arvados
import logging
logger = logging.getLogger('arvados.YOURTHING')
Once you've set this up, you can send messages to the logger using methods like logger.debug()
and logger.error()
. See the Logger class documentation for full details.
Command-line scripts may reconfigure the arvados.logger
object based on additional configuration like command-line switches. services/fuse/bin/arv-mount
demonstrates adjusting the level and setting a custom log handler.
Documenting your code¶
See Coding_Standards.
Python buffer protocol¶
Notes on managing buffers efficiently in Python, we don't use this in the python sdk as of this writing (but we might).
Example using bytearray() to allocate a buffer, memoryview() to create a writable slice, and readinto() to write directly to the buffer slice:
>>> b = bytearray(20) >>> c = memoryview(b) >>> f = open("python.txt", "r") >>> f.readinto(c[5:10]) 5 >>> b bytearray(b"\x00\x00\x00\x00\x00I\'ve \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")
Updated by Lucas Di Pentima 2 months ago · 26 revisions