Arvados

Added os.name check to the import in d71e2f17

Unit Tests: https://ci.curoverse.com/view/Developer/job/developer-run-tests/1075/
Conformance and Integration Tests: https://ci.curoverse.com/view/CWL/job/arvados-cwl-conformance-tests/13/

Unfortunately, packaging fails for centos7, debian8, and ubuntu1404. The setup fails because of the extra_requires. Apparently its a setuptools issue.

Example centos7 output. Similar for other environments where its not working:

Building rpm package for arvados-python-client from sdk/python
Creating virtualenv...
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-XZSO8B/
Error, unable to run
build/usr/share/python2.7/dist/python-arvados-python-client/bin/pip install -q arvados-python-client-1.3.0.20190215162809.tar.gz
ERROR: build packages on arvados/build:centos7 failed with exit status 1

Centos7 example doesn't include the following from 1404:

error in arvados-python-client setup command: Invalid environment marker: os.name=="posix" and python_version<"3"
Error, unable to run python setup.py sdist for arvados-python-client

I've pushed 28209cbb68aaa3bcf2832ca739d6c1ed89f4d3fd on this branch to fix the package build on centos7, debian8 and ubuntu1404.

eric@ubuntu:~/projects/arvados$ WORKSPACE=~/projects/arvados ./build/run-build-test-packages-one-target.sh --target ubuntu1404

• Start build packages ****

~/projects/arvados/build/package-build-dockerfiles ~/projects/arvados
wget cqO common-generated/go1.10.1.linux-amd64.tar.gz http://storage.googleapis.com/golang/go1.10.1.linux-amd64.tar.gz
wget -cqO common-generated/node-v6.11.2-linux-x64.tar.xz https://nodejs.org/dist/v6.11.2/node-v6.11.2-linux-x64.tar.xz
wget -cqO common-generated/mpapis.asc https://rvm.io/mpapis.asc
wget -cqO common-generated/pkuczynski.asc https://rvm.io/pkuczynski.asc
test -d ubuntu1404/generated || mkdir ubuntu1404/generated
cp -rlt ubuntu1404/generated common-generated/*
ubuntu1404
Sending build context to Docker daemon 129.2MB
Step 1/16 : FROM ubuntu:trusty
--> 5dbc3f318ea5
Step 2/16 : MAINTAINER Ward Vandewege <ward@curoverse.com>
---> Using cache
---> 2199e783855a
Step 3/16 : ENV DEBIAN_FRONTEND noninteractive
---> Using cache
---> b77c5546f4b2
Step 4/16 : RUN /usr/bin/apt-get update && /usr/bin/apt-get install q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip python3.4-venv python3.4-dev
--> Using cache
---> 9045cede5f56
Step 5/16 : RUN /usr/bin/pip install virtualenv
---> Using cache
---> 48355e95ab96
Step 6/16 : ADD generated/mpapis.asc /tmp/
---> Using cache
---> e8036669d81c
Step 7/16 : ADD generated/pkuczynski.asc /tmp/
---> Using cache
---> fcd8f7158b38
Step 8/16 : RUN gpg --import --no-tty /tmp/mpapis.asc && gpg --import --no-tty /tmp/pkuczynski.asc && curl L https://get.rvm.io | bash -s stable && /usr/local/rvm/bin/rvm install 2.3 && /usr/local/rvm/bin/rvm alias create default ruby-2.3 && /usr/local/rvm/bin/rvm-exec default gem install bundler && /usr/local/rvm/bin/rvm-exec default gem install fpm --version 1.10.2
--> Using cache
---> fcc872ba7bb5
Step 9/16 : ADD generated/go1.10.1.linux-amd64.tar.gz /usr/local/
---> Using cache
---> 034c372b08cf
Step 10/16 : RUN ln s /usr/local/go/bin/go /usr/local/bin/
--> Using cache
---> 301fdc5911ac
Step 11/16 : ADD generated/node-v6.11.2-linux-x64.tar.xz /usr/local/
---> Using cache
---> 447ece390499
Step 12/16 : RUN ln s /usr/local/node-v6.11.2-linux-x64/bin/* /usr/local/bin/
--> Using cache
---> 88fdd57fe41e
Step 13/16 : RUN pip install --upgrade setuptools
---> Using cache
---> d0cd509ef0bb
Step 14/16 : RUN git clone --depth 1 git://git.curoverse.com/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle && rm rf /tmp/arvados
--> Using cache
---> f69add8ad9f8
Step 15/16 : ENV WORKSPACE /arvados
---> Using cache
---> bf09a0dda95a
Step 16/16 : CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "ubuntu1404"]
---> Using cache
---> 1e2e1fdd8b51
Successfully built 1e2e1fdd8b51
Successfully tagged arvados/build:ubuntu1404

real 0m0.993s
user 0m0.136s
sys 0m0.167s
~/projects/arvados

START: build packages on arvados/build:ubuntu1404
Package libarvados-perl_1.3.0.20170804184251-1_amd64.deb not found, building
Warning: prerequisite JSON 0 not found.
Warning: prerequisite LWP 0 not found.
Warning: prerequisite Net::SSL 0 not found.
Package arvados-src_1.3.0.20190219154724-1_all.deb not found, building
Package arvados-client_1.3.0.20190212191710-1_amd64.deb not found, building
Package arvados-server_1.3.0.20190212191710-1_amd64.deb not found, building
Package arvados-controller_1.3.0.20190212191710-1_amd64.deb not found, building
Package arvados-dispatch-cloud_1.3.0.20190212191710-1_amd64.deb not found, building
Package crunchrunner_1.3.0.20190212191710-1_amd64.deb not found, building
Package arvados-git-httpd_1.3.0.20190212191710-1_amd64.deb not found, building
Package crunch-dispatch-local_1.3.0.20190212191710-1_amd64.deb not found, building
Package crunch-dispatch-slurm_1.3.0.20190212191710-1_amd64.deb not found, building
Package crunch-run_1.3.0.20190212191710-1_amd64.deb not found, building
Package crunchstat_1.3.0.20190212191710-1_amd64.deb not found, building
Package arvados-health_1.3.0.20190212191710-1_amd64.deb not found, building
Package keep-balance_1.3.0.20190212191710-1_amd64.deb not found, building
Package keepproxy_1.3.0.20190212191710-1_amd64.deb not found, building
Package keepstore_1.3.0.20190212191710-1_amd64.deb not found, building
Package keep-web_1.3.0.20190212191710-1_amd64.deb not found, building
Package arvados-ws_1.3.0.20190212191710-1_amd64.deb not found, building
Package arvados-sync-groups_1.3.0.20190212191710-1_amd64.deb not found, building
Package keep-block-check_1.3.0.20190212191710-1_amd64.deb not found, building
Package keep-rsync_1.3.0.20190212191710-1_amd64.deb not found, building
Package keep-exercise_1.3.0.20190212191710-1_amd64.deb not found, building
Package python-arvados-python-client_1.3.0.20190215162809-1_amd64.deb not found, building
Building deb package for arvados-python-client from sdk/python
Creating virtualenv...
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:354: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:354: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:354: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:354: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning
/arvados/sdk/python/dist/build/usr/share/python2.7/dist/python-arvados-python-client/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning
Creating package...
Loading fpm overrides from /arvados/sdk/python/fpm-info.sh
python-arvados-python-client_1.3.0.20190215162809-1_amd64.deb

error in arvados-python-client setup command: Invalid environment marker: os.name=="posix" and python_version<"3"
Error, unable to run python setup.py sdist for arvados-python-client
ERROR: build packages on arvados/build:ubuntu1404 failed with exit status 1

• !!!!!! build packages FAILED !!!!!! ****
  • End of build packages (140s) ****
    • Start test packages ****

Skipping package upload, there were errors building the packages

• End of test packages (0s) ****

Pass: test packages (0s)
Failures (1):
Fail: build packages (140s)

I've pushed a fix in c9f9b4cf4659ef801b059a006ce46e3849926f71, have a look?