Feature #15888
closed
Update run-tests.sh to use python 3
Added by Eric Biagiotti over 4 years ago. Updated over 3 years ago.
Description
run-tests.sh uses python in other places besides running the sdk/python tests (e.g. starting services, etc). Update these instances to use python 3. Specifically, change VENVDIR/bin/activate to VENV3DIR/bin/activate.
If py2 isn't installed, run-tests.sh should succeed (by skipping the py2 tests). If py2 is installed, run-tests.sh should not skip the py2 tests.
sdk/python/tests/run_test_server.py is being called with `python` from sdk/go/arvadostest/run_servers.go. Does run_test_server.py even support python 3? We should probably replace it with arvados-boot instead of spending more time on it.
Related issues
Updated by Eric Biagiotti over 4 years ago
- Related to Idea #14532: [Epic] Port to Python 3 to for Python 2 sunset in December 2019 added
Updated by Peter Amstutz over 3 years ago
- Target version changed from Arvados Future Sprints to 2020-08-26 Sprint
- Assigned To set to Nico César
Updated by Nico César over 3 years ago
During standup we agreed (Peter/Tom/Lucas/me) that this ticket has not aged well since we don't support py2 anymore. Migrating all python tooling to py3 is the best idea we have.
Updated by Nico César over 3 years ago
apt-get remove $(COLUMNS=160 dpkg -l | grep -E 'python[^3]'| awk '{ print $2 }' | grep -v 'dh-python' )
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
cython libpython-dev libpython-stdlib libpython2.7 libpython2.7-dev libpython2.7-minimal
libpython2.7-stdlib linkchecker python python-chardet python-epydoc python-minimal
python-pip python-pip-whl python-pkg-resources python-pycurl python-requests
python-setuptools python-six python-urllib3 python-virtualenv python-yaml python2.7
python2.7-minimal python3-pip python3-virtualenv virtualenv
0 upgraded, 0 newly installed, 27 to remove and 1 not upgraded.
After this operation, 79.7 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 38304 files and directories currently installed.)
Removing cython (0.25.2-1) ...
Removing libpython-dev:amd64 (2.7.13-2) ...
Removing python-yaml (3.12-1) ...
Removing linkchecker (9.3-4) ...
Removing python-requests (2.12.4-1) ...
Removing python-urllib3 (1.19.1-1) ...
Removing python-six (1.10.0-3) ...
Removing python-virtualenv (15.1.0+ds-1) ...
Removing python-setuptools (33.1.1-1) ...
Removing libpython2.7-dev:amd64 (2.7.13-2+deb9u3) ...
Removing libpython2.7:amd64 (2.7.13-2+deb9u3) ...
Removing python-chardet (2.3.0-2) ...
Removing python-epydoc (3.0.1+dfsg-14) ...
Removing python-pip (9.0.1-2+deb9u1) ...
Removing virtualenv (15.1.0+ds-1) ...
Removing python3-virtualenv (15.1.0+ds-1) ...
Removing python-pkg-resources (33.1.1-1) ...
Removing python-pycurl (7.43.0-2) ...
Removing python3-pip (9.0.1-2+deb9u1) ...
Removing python (2.7.13-2) ...
Removing libpython-stdlib:amd64 (2.7.13-2) ...
Removing python2.7 (2.7.13-2+deb9u3) ...
Removing libpython2.7-stdlib:amd64 (2.7.13-2+deb9u3) ...
Removing python-minimal (2.7.13-2) ...
Removing python2.7-minimal (2.7.13-2+deb9u3) ...
Removing libpython2.7-minimal:amd64 (2.7.13-2+deb9u3) ...
Removing python-pip-whl (9.0.1-2+deb9u1) ...
Processing triggers for mime-support (3.60) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
root@4e513ebf867f:/usr/src/arvados# apt-get install python3-pip python3-virtualenv
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
dbus python-pip-whl python3-cffi-backend python3-crypto python3-cryptography python3-dbus
python3-gi python3-idna python3-keyring python3-keyrings.alt python3-pyasn1
python3-secretstorage python3-six python3-wheel python3-xdg
Suggested packages:
default-dbus-session-bus | dbus-session-bus python3-crypto-dbg python-crypto-doc
python-cryptography-doc python3-cryptography-vectors python-dbus-doc python3-dbus-dbg
gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0 python3-pykde4 doc-base
python-secretstorage-doc
The following NEW packages will be installed:
dbus python-pip-whl python3-cffi-backend python3-crypto python3-cryptography python3-dbus
python3-gi python3-idna python3-keyring python3-keyrings.alt python3-pip python3-pyasn1
python3-secretstorage python3-six python3-virtualenv python3-wheel python3-xdg
0 upgraded, 17 newly installed, 0 to remove and 1 not upgraded.
Need to get 3,245 kB of archives.
After this operation, 8,668 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian stretch/main amd64 dbus amd64 1.10.32-0+deb9u1 [213 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 python-pip-whl all 9.0.1-2+deb9u1 [1,399 kB]
Get:3 http://deb.debian.org/debian stretch/main amd64 python3-cffi-backend amd64 1.9.1-2 [70.1 kB]
Get:4 http://deb.debian.org/debian stretch/main amd64 python3-crypto amd64 2.6.1-7 [259 kB]
Get:5 http://deb.debian.org/debian stretch/main amd64 python3-idna all 2.2-1 [32.7 kB]
Get:6 http://deb.debian.org/debian stretch/main amd64 python3-pyasn1 all 0.1.9-2 [34.5 kB]
Get:7 http://deb.debian.org/debian stretch/main amd64 python3-six all 1.10.0-3 [14.4 kB]
Get:8 http://deb.debian.org/debian stretch/main amd64 python3-cryptography amd64 1.7.1-3+deb9u2 [211 kB]
Get:9 http://deb.debian.org/debian stretch/main amd64 python3-dbus amd64 1.2.4-1+b1 [184 kB]
Get:10 http://deb.debian.org/debian stretch/main amd64 python3-gi amd64 3.22.0-2 [473 kB]
Get:11 http://deb.debian.org/debian stretch/main amd64 python3-secretstorage all 2.3.1-2 [14.2 kB]
Get:12 http://deb.debian.org/debian stretch/main amd64 python3-keyring all 10.1-1 [36.8 kB]
Get:13 http://deb.debian.org/debian stretch/main amd64 python3-keyrings.alt all 1.3-1 [16.2 kB]
Get:14 http://deb.debian.org/debian stretch/main amd64 python3-pip all 9.0.1-2+deb9u1 [142 kB]
Get:15 http://deb.debian.org/debian stretch/main amd64 python3-virtualenv all 15.1.0+ds-1 [57.7 kB]
Get:16 http://deb.debian.org/debian stretch/main amd64 python3-wheel all 0.29.0-2 [51.8 kB]
Get:17 http://deb.debian.org/debian stretch/main amd64 python3-xdg all 0.25-4 [35.7 kB]
Fetched 3,245 kB in 1s (1,949 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package dbus.
(Reading database ... 36214 files and directories currently installed.)
Preparing to unpack .../00-dbus_1.10.32-0+deb9u1_amd64.deb ...
Unpacking dbus (1.10.32-0+deb9u1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../01-python-pip-whl_9.0.1-2+deb9u1_all.deb ...
Unpacking python-pip-whl (9.0.1-2+deb9u1) ...
Selecting previously unselected package python3-cffi-backend.
Preparing to unpack .../02-python3-cffi-backend_1.9.1-2_amd64.deb ...
Unpacking python3-cffi-backend (1.9.1-2) ...
Selecting previously unselected package python3-crypto.
Preparing to unpack .../03-python3-crypto_2.6.1-7_amd64.deb ...
Unpacking python3-crypto (2.6.1-7) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../04-python3-idna_2.2-1_all.deb ...
Unpacking python3-idna (2.2-1) ...
Selecting previously unselected package python3-pyasn1.
Preparing to unpack .../05-python3-pyasn1_0.1.9-2_all.deb ...
Unpacking python3-pyasn1 (0.1.9-2) ...
Selecting previously unselected package python3-six.
Preparing to unpack .../06-python3-six_1.10.0-3_all.deb ...
Unpacking python3-six (1.10.0-3) ...
Selecting previously unselected package python3-cryptography.
Preparing to unpack .../07-python3-cryptography_1.7.1-3+deb9u2_amd64.deb ...
Unpacking python3-cryptography (1.7.1-3+deb9u2) ...
Selecting previously unselected package python3-dbus.
Preparing to unpack .../08-python3-dbus_1.2.4-1+b1_amd64.deb ...
Unpacking python3-dbus (1.2.4-1+b1) ...
Selecting previously unselected package python3-gi.
Preparing to unpack .../09-python3-gi_3.22.0-2_amd64.deb ...
Unpacking python3-gi (3.22.0-2) ...
Selecting previously unselected package python3-secretstorage.
Preparing to unpack .../10-python3-secretstorage_2.3.1-2_all.deb ...
Unpacking python3-secretstorage (2.3.1-2) ...
Selecting previously unselected package python3-keyring.
Preparing to unpack .../11-python3-keyring_10.1-1_all.deb ...
Unpacking python3-keyring (10.1-1) ...
Selecting previously unselected package python3-keyrings.alt.
Preparing to unpack .../12-python3-keyrings.alt_1.3-1_all.deb ...
Unpacking python3-keyrings.alt (1.3-1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../13-python3-pip_9.0.1-2+deb9u1_all.deb ...
Unpacking python3-pip (9.0.1-2+deb9u1) ...
Selecting previously unselected package python3-virtualenv.
Preparing to unpack .../14-python3-virtualenv_15.1.0+ds-1_all.deb ...
Unpacking python3-virtualenv (15.1.0+ds-1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../15-python3-wheel_0.29.0-2_all.deb ...
Unpacking python3-wheel (0.29.0-2) ...
Selecting previously unselected package python3-xdg.
Preparing to unpack .../16-python3-xdg_0.25-4_all.deb ...
Unpacking python3-xdg (0.25-4) ...
Setting up python-pip-whl (9.0.1-2+deb9u1) ...
Setting up python3-dbus (1.2.4-1+b1) ...
Setting up python3-cffi-backend (1.9.1-2) ...
Setting up python3-crypto (2.6.1-7) ...
Setting up python3-idna (2.2-1) ...
Setting up python3-xdg (0.25-4) ...
Setting up python3-keyrings.alt (1.3-1) ...
Setting up python3-six (1.10.0-3) ...
Setting up python3-wheel (0.29.0-2) ...
Setting up python3-gi (3.22.0-2) ...
Setting up python3-pyasn1 (0.1.9-2) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up python3-virtualenv (15.1.0+ds-1) ...
Setting up python3-pip (9.0.1-2+deb9u1) ...
Setting up dbus (1.10.32-0+deb9u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up python3-cryptography (1.7.1-3+deb9u2) ...
Setting up python3-secretstorage (2.3.1-2) ...
Setting up python3-keyring (10.1-1) ...
Updated by Nico César over 3 years ago
Intalled python-venv
this will allow to do
python3 -m venv /DIR
Updated by Nico César over 3 years ago
python3-httplib2 is needed
++ python3 sdk/python/tests/run_test_server.py setup_config
Traceback (most recent call last):
File "sdk/python/tests/run_test_server.py", line 13, in <module>
import httplib2
ImportError: No module named 'httplib2'
+ eval
Updated by Nico César over 3 years ago
- Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint
Updated by Nico César over 3 years ago
62b3435fc45adff01541c508be22e10be83427ce
There are still some issues with importing arvados in the py3 environment
TestArvKeepGet#test_help = Traceback (most recent call last):
File "./bin/arv-put", line 6, in <module>
from arvados.commands.put import main
ImportError: No module named 'arvados'
Traceback (most recent call last):
File "./bin/arv-put", line 6, in <module>
from arvados.commands.put import main
ImportError: No module named 'arvados'
Traceback (most recent call last):
File "./bin/arv-put", line 6, in <module>
from arvados.commands.put import main
ImportError: No module named 'arvados'
0.11 s = F
Updated by Nico César over 3 years ago
- Target version deleted (
2020-09-09 Sprint) - Assigned To deleted (
Nico César)
cd /usr/src/arvados/build ; bash ./run-tests.sh WORKSPACE=/usr/src/arvados ARVADOS_CONFIG=/etc/arvados/config.yml --temp ~/.cache/arvados-build --leave-temp
Updated by Nico César over 3 years ago
- Target version set to 2020-09-09 Sprint
- Assigned To set to Nico César
Updated by Ward Vandewege over 3 years ago
- Related to Bug #16795: [a-d-c] flaky test added
Updated by Ward Vandewege over 3 years ago
Before merging into master, will need to remove the pycurl pin added here: https://dev.arvados.org/issues/16795#note-3
Updated by Nico César over 3 years ago
Almost there, but not 100%:
Updated by Nico César over 3 years ago
If we use the python2.7 "own python" from package this is the behaviour:
nico@shell:~$ arv-get --version /usr/bin/arv-get 2.0.3 nico@shell:~$ head --lines=1 /usr/bin/arv-get #!/usr/share/python2.7/dist/python-arvados-python-client/bin/python nico@shell:~$ arv-get --version 2> /dev/null nico@shell:~$
1) Failure: TestArvKeepGet#test_get_version [/tmp/workspace/developer-run-tests-remainder/sdk/cli/test/test_arv-keep-get.rb:31]: STDOUT not expected: './bin/arv-get 2.1.0.dev20200826203407 '. Expected "./bin/arv-get 2.1.0.dev20200826203407\n" to be empty.
this is the test from sdk/cli/test/test_arv-keep-get.rb
def test_get_version
out, err = capture_subprocess_io do
assert_arv_get '--version'
end
assert_empty(out, "STDOUT not expected: '#{out}'")
assert_match(/[0-9]+\.[0-9]+\.[0-9]+/, err, "Version information incorrect: '#{err}'")
end
Updated by Nico César over 3 years ago
nico@shell:~$ python3 /usr/bin/arv-get --version
Traceback (most recent call last):
File "/usr/bin/arv-get", line 8, in <module>
from arvados.commands.get import main
ModuleNotFoundError: No module named 'arvados'
because of the lack of that library installed:
nico@shell:~$ dpkg -l | grep python3-arvados-python-client nico@shell:~$
So here we have to make sure that when we change our packages to have the right dependency when deploying
Updated by Nico César over 3 years ago
after installing:
nico@shell:~$ dpkg -l | grep python3-arvados-python-client ii python3-arvados-python-client 2.0.4-1 amd64 Arvados client library nico@shell:~$ head /usr/bin/arv-get #!/usr/share/python3/dist/python3-arvados-python-client/bin/python nico@shell:~$ /usr/share/python3/dist/python3-arvados-python-client/bin/python /usr/bin/arv-get --version 2> /dev/null /usr/bin/arv-get 2.0.4
Updated by Nico César over 3 years ago
developer-run-tests: #2071 successfull run!
Updated by Ward Vandewege over 3 years ago
A few comments:
- shouldn't commit the go.sum and go.mod changes
noted
- trailing space added in run-tests.sh line 657
- run-tests.sh:
@@ -558,7 +555,7 @@ setup_ruby_environment() {
fi
done
"$bundle" version | tee /dev/stderr | grep -q 'version 2'
- ) || fatal 'install bundler'
+ ) #|| fatal 'install bundler' ## This is needed for now.
fi
}
This seems like it should not be part of the patch?
- The patch is adding a few "FIXME" lines, do we need additional changes there?
- Any reason you're removing the '-x' from the bash invocation (2 places)?
Updated by Nico César over 3 years ago
Did most of the changes.
about $pythonstuff...
install_all tries to do a pip install of the packages in $pythonstuff. as of 5aca41295
all of them have the suffix ":py3"
pythonstuff=(
sdk/python:py3
sdk/cwl:py3
services/dockercleaner:py3
services/fuse:py3
tools/crunchstat-summary:py3
)
and I've seen some code that tries to remove the ones ending in ":py3"[1] by doing:
for p in "${pythonstuff[@]}"
do
dir=${p%:py3}
so I think we should remove this part.
My proposal is 4cfab58ae and running developer-run-tests: #2073 now
[1] man bash
${parameter%word}
${parameter%%word}
Remove matching suffix pattern. The word is expanded to produce a pattern just as in pathname expansion. If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the ''%'' case) or the longest matching pattern (the ''%%'' case) deleted. If parameter is @ or *, the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with @ or *, the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list.
Updated by Ward Vandewege over 3 years ago
Nico César wrote:
Did most of the changes.
about $pythonstuff...
install_all tries to do a pip install of the packages in $pythonstuff. as of 5aca41295
all of them have the suffix ":py3"
[...]
and I've seen some code that tries to remove the ones ending in ":py3"[1] by doing:
[...]so I think we should remove this part.
My proposal is 4cfab58ae and running developer-run-tests: #2073
now
[...]
Tests passed, LGTM. Just make sure after you merge master into your branch that you then change this line in sdk/python/setup.py:
'pycurl >=7.19.5.1, <7.43.0.4', # 7.43.0.4 removes support for python2to remove the '<7.43.0.4' part, which should no longer be needed after you purged python2. It was introduced in master after you forked.
Could also do that as a separate commit, of course.
Updated by Nico César over 3 years ago
developer-run-tests: #2073 passed and ward said LGTM.
I'll go ahead and merge it
Updated by Anonymous over 3 years ago
- % Done changed from 0 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|bdeca9098f356a4b2b088a3cf4c9276bb6cd5bdb.
Updated by Ward Vandewege over 3 years ago
Another remnant in arados-server, cf. 94a7479527652cd513a35f75e1da1798d9fdc213 on branch 15888-remove-more-python2-remnants.
Updated by Nico César over 3 years ago
94a7479527652cd513a35f75e1da1798d9fdc213 LGTM,
I noticed developer-run-tests: #2083
with commit 94a7479527652cd513a35f75e1da1798d9fdc213, but it seems to be a flaky test.
Updated by Ward Vandewege over 3 years ago
Nico César wrote:
94a7479527652cd513a35f75e1da1798d9fdc213 LGTM,
I noticed developer-run-tests: #2083
with commit 94a7479527652cd513a35f75e1da1798d9fdc213, but it seems to be a flaky test.
Thanks, merged it. It's a flaky test indeed.
Updated by Ward Vandewege over 2 years ago
- Related to Bug #17985: [a-c-r] add cwltool to the list of binstubs in our packages added