Project

General

Profile

Actions

Feature #15888

closed

Update run-tests.sh to use python 3

Added by Eric Biagiotti over 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Description

run-tests.sh uses python in other places besides running the sdk/python tests (e.g. starting services, etc). Update these instances to use python 3. Specifically, change VENVDIR/bin/activate to VENV3DIR/bin/activate.

If py2 isn't installed, run-tests.sh should succeed (by skipping the py2 tests). If py2 is installed, run-tests.sh should not skip the py2 tests.

sdk/python/tests/run_test_server.py is being called with `python` from sdk/go/arvadostest/run_servers.go. Does run_test_server.py even support python 3? We should probably replace it with arvados-boot instead of spending more time on it.


Subtasks 1 (1 open0 closed)

Task #16768: ReviewNewPeter AmstutzActions

Related issues

Related to Arvados Epics - Idea #14532: [Epic] Port to Python 3 to for Python 2 sunset in December 2019Resolved01/01/202009/16/2020Actions
Related to Arvados - Bug #16795: [a-d-c] flaky testResolvedTom Clegg09/02/2020Actions
Related to Arvados - Bug #17985: [a-c-r] add cwltool to the list of binstubs in our packagesResolvedWard VandewegeActions
Actions #1

Updated by Eric Biagiotti over 4 years ago

  • Description updated (diff)
Actions #2

Updated by Eric Biagiotti over 4 years ago

  • Parent task deleted (#14532)
Actions #3

Updated by Eric Biagiotti over 4 years ago

  • Related to Idea #14532: [Epic] Port to Python 3 to for Python 2 sunset in December 2019 added
Actions #4

Updated by Peter Amstutz almost 4 years ago

  • Target version changed from Arvados Future Sprints to 2020-08-26 Sprint
  • Assigned To set to Nico César
Actions #5

Updated by Peter Amstutz almost 4 years ago

  • Description updated (diff)
Actions #6

Updated by Ward Vandewege almost 4 years ago

  • Tracker changed from Task to Feature
Actions #7

Updated by Nico César almost 4 years ago

  • Description updated (diff)
Actions #8

Updated by Nico César almost 4 years ago

During standup we agreed (Peter/Tom/Lucas/me) that this ticket has not aged well since we don't support py2 anymore. Migrating all python tooling to py3 is the best idea we have.

Actions #9

Updated by Nico César almost 4 years ago

apt-get  remove $(COLUMNS=160 dpkg -l | grep -E 'python[^3]'| awk '{ print $2 }' | grep -v 'dh-python' )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  cython libpython-dev libpython-stdlib libpython2.7 libpython2.7-dev libpython2.7-minimal
  libpython2.7-stdlib linkchecker python python-chardet python-epydoc python-minimal
  python-pip python-pip-whl python-pkg-resources python-pycurl python-requests
  python-setuptools python-six python-urllib3 python-virtualenv python-yaml python2.7
  python2.7-minimal python3-pip python3-virtualenv virtualenv
0 upgraded, 0 newly installed, 27 to remove and 1 not upgraded.
After this operation, 79.7 MB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 38304 files and directories currently installed.)
Removing cython (0.25.2-1) ...
Removing libpython-dev:amd64 (2.7.13-2) ...
Removing python-yaml (3.12-1) ...
Removing linkchecker (9.3-4) ...
Removing python-requests (2.12.4-1) ...
Removing python-urllib3 (1.19.1-1) ...
Removing python-six (1.10.0-3) ...
Removing python-virtualenv (15.1.0+ds-1) ...
Removing python-setuptools (33.1.1-1) ...
Removing libpython2.7-dev:amd64 (2.7.13-2+deb9u3) ...
Removing libpython2.7:amd64 (2.7.13-2+deb9u3) ...
Removing python-chardet (2.3.0-2) ...
Removing python-epydoc (3.0.1+dfsg-14) ...
Removing python-pip (9.0.1-2+deb9u1) ...
Removing virtualenv (15.1.0+ds-1) ...
Removing python3-virtualenv (15.1.0+ds-1) ...
Removing python-pkg-resources (33.1.1-1) ...
Removing python-pycurl (7.43.0-2) ...
Removing python3-pip (9.0.1-2+deb9u1) ...
Removing python (2.7.13-2) ...
Removing libpython-stdlib:amd64 (2.7.13-2) ...
Removing python2.7 (2.7.13-2+deb9u3) ...
Removing libpython2.7-stdlib:amd64 (2.7.13-2+deb9u3) ...
Removing python-minimal (2.7.13-2) ...
Removing python2.7-minimal (2.7.13-2+deb9u3) ...
Removing libpython2.7-minimal:amd64 (2.7.13-2+deb9u3) ...
Removing python-pip-whl (9.0.1-2+deb9u1) ...
Processing triggers for mime-support (3.60) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
root@4e513ebf867f:/usr/src/arvados# apt-get install python3-pip python3-virtualenv
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  dbus python-pip-whl python3-cffi-backend python3-crypto python3-cryptography python3-dbus
  python3-gi python3-idna python3-keyring python3-keyrings.alt python3-pyasn1
  python3-secretstorage python3-six python3-wheel python3-xdg
Suggested packages:
  default-dbus-session-bus | dbus-session-bus python3-crypto-dbg python-crypto-doc
  python-cryptography-doc python3-cryptography-vectors python-dbus-doc python3-dbus-dbg
  gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0 python3-pykde4 doc-base
  python-secretstorage-doc
The following NEW packages will be installed:
  dbus python-pip-whl python3-cffi-backend python3-crypto python3-cryptography python3-dbus
  python3-gi python3-idna python3-keyring python3-keyrings.alt python3-pip python3-pyasn1
  python3-secretstorage python3-six python3-virtualenv python3-wheel python3-xdg
0 upgraded, 17 newly installed, 0 to remove and 1 not upgraded.
Need to get 3,245 kB of archives.
After this operation, 8,668 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian stretch/main amd64 dbus amd64 1.10.32-0+deb9u1 [213 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 python-pip-whl all 9.0.1-2+deb9u1 [1,399 kB]
Get:3 http://deb.debian.org/debian stretch/main amd64 python3-cffi-backend amd64 1.9.1-2 [70.1 kB]
Get:4 http://deb.debian.org/debian stretch/main amd64 python3-crypto amd64 2.6.1-7 [259 kB]
Get:5 http://deb.debian.org/debian stretch/main amd64 python3-idna all 2.2-1 [32.7 kB]
Get:6 http://deb.debian.org/debian stretch/main amd64 python3-pyasn1 all 0.1.9-2 [34.5 kB]
Get:7 http://deb.debian.org/debian stretch/main amd64 python3-six all 1.10.0-3 [14.4 kB]
Get:8 http://deb.debian.org/debian stretch/main amd64 python3-cryptography amd64 1.7.1-3+deb9u2 [211 kB]
Get:9 http://deb.debian.org/debian stretch/main amd64 python3-dbus amd64 1.2.4-1+b1 [184 kB]
Get:10 http://deb.debian.org/debian stretch/main amd64 python3-gi amd64 3.22.0-2 [473 kB]
Get:11 http://deb.debian.org/debian stretch/main amd64 python3-secretstorage all 2.3.1-2 [14.2 kB]
Get:12 http://deb.debian.org/debian stretch/main amd64 python3-keyring all 10.1-1 [36.8 kB]
Get:13 http://deb.debian.org/debian stretch/main amd64 python3-keyrings.alt all 1.3-1 [16.2 kB]
Get:14 http://deb.debian.org/debian stretch/main amd64 python3-pip all 9.0.1-2+deb9u1 [142 kB]
Get:15 http://deb.debian.org/debian stretch/main amd64 python3-virtualenv all 15.1.0+ds-1 [57.7 kB]
Get:16 http://deb.debian.org/debian stretch/main amd64 python3-wheel all 0.29.0-2 [51.8 kB]
Get:17 http://deb.debian.org/debian stretch/main amd64 python3-xdg all 0.25-4 [35.7 kB]
Fetched 3,245 kB in 1s (1,949 kB/s)     
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package dbus.
(Reading database ... 36214 files and directories currently installed.)
Preparing to unpack .../00-dbus_1.10.32-0+deb9u1_amd64.deb ...
Unpacking dbus (1.10.32-0+deb9u1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../01-python-pip-whl_9.0.1-2+deb9u1_all.deb ...
Unpacking python-pip-whl (9.0.1-2+deb9u1) ...
Selecting previously unselected package python3-cffi-backend.
Preparing to unpack .../02-python3-cffi-backend_1.9.1-2_amd64.deb ...
Unpacking python3-cffi-backend (1.9.1-2) ...
Selecting previously unselected package python3-crypto.
Preparing to unpack .../03-python3-crypto_2.6.1-7_amd64.deb ...
Unpacking python3-crypto (2.6.1-7) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../04-python3-idna_2.2-1_all.deb ...
Unpacking python3-idna (2.2-1) ...
Selecting previously unselected package python3-pyasn1.
Preparing to unpack .../05-python3-pyasn1_0.1.9-2_all.deb ...
Unpacking python3-pyasn1 (0.1.9-2) ...
Selecting previously unselected package python3-six.
Preparing to unpack .../06-python3-six_1.10.0-3_all.deb ...
Unpacking python3-six (1.10.0-3) ...
Selecting previously unselected package python3-cryptography.
Preparing to unpack .../07-python3-cryptography_1.7.1-3+deb9u2_amd64.deb ...
Unpacking python3-cryptography (1.7.1-3+deb9u2) ...
Selecting previously unselected package python3-dbus.
Preparing to unpack .../08-python3-dbus_1.2.4-1+b1_amd64.deb ...
Unpacking python3-dbus (1.2.4-1+b1) ...
Selecting previously unselected package python3-gi.
Preparing to unpack .../09-python3-gi_3.22.0-2_amd64.deb ...
Unpacking python3-gi (3.22.0-2) ...
Selecting previously unselected package python3-secretstorage.
Preparing to unpack .../10-python3-secretstorage_2.3.1-2_all.deb ...
Unpacking python3-secretstorage (2.3.1-2) ...
Selecting previously unselected package python3-keyring.
Preparing to unpack .../11-python3-keyring_10.1-1_all.deb ...
Unpacking python3-keyring (10.1-1) ...
Selecting previously unselected package python3-keyrings.alt.
Preparing to unpack .../12-python3-keyrings.alt_1.3-1_all.deb ...
Unpacking python3-keyrings.alt (1.3-1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../13-python3-pip_9.0.1-2+deb9u1_all.deb ...
Unpacking python3-pip (9.0.1-2+deb9u1) ...
Selecting previously unselected package python3-virtualenv.
Preparing to unpack .../14-python3-virtualenv_15.1.0+ds-1_all.deb ...
Unpacking python3-virtualenv (15.1.0+ds-1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../15-python3-wheel_0.29.0-2_all.deb ...
Unpacking python3-wheel (0.29.0-2) ...
Selecting previously unselected package python3-xdg.
Preparing to unpack .../16-python3-xdg_0.25-4_all.deb ...
Unpacking python3-xdg (0.25-4) ...
Setting up python-pip-whl (9.0.1-2+deb9u1) ...
Setting up python3-dbus (1.2.4-1+b1) ...
Setting up python3-cffi-backend (1.9.1-2) ...
Setting up python3-crypto (2.6.1-7) ...
Setting up python3-idna (2.2-1) ...
Setting up python3-xdg (0.25-4) ...
Setting up python3-keyrings.alt (1.3-1) ...
Setting up python3-six (1.10.0-3) ...
Setting up python3-wheel (0.29.0-2) ...
Setting up python3-gi (3.22.0-2) ...
Setting up python3-pyasn1 (0.1.9-2) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up python3-virtualenv (15.1.0+ds-1) ...
Setting up python3-pip (9.0.1-2+deb9u1) ...
Setting up dbus (1.10.32-0+deb9u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up python3-cryptography (1.7.1-3+deb9u2) ...
Setting up python3-secretstorage (2.3.1-2) ...
Setting up python3-keyring (10.1-1) ...

Actions #10

Updated by Nico César almost 4 years ago

Intalled python-venv

this will allow to do

python3 -m venv /DIR

Actions #11

Updated by Nico César almost 4 years ago

python3-httplib2 is needed

++ python3 sdk/python/tests/run_test_server.py setup_config
Traceback (most recent call last):
  File "sdk/python/tests/run_test_server.py", line 13, in <module>
    import httplib2
ImportError: No module named 'httplib2'
+ eval
Actions #12

Updated by Nico César almost 4 years ago

  • Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint
Actions #13

Updated by Nico César almost 4 years ago

62b3435fc45adff01541c508be22e10be83427ce

There are still some issues with importing arvados in the py3 environment

TestArvKeepGet#test_help = Traceback (most recent call last):
  File "./bin/arv-put", line 6, in <module>
    from arvados.commands.put import main
ImportError: No module named 'arvados'
Traceback (most recent call last):
  File "./bin/arv-put", line 6, in <module>
    from arvados.commands.put import main
ImportError: No module named 'arvados'
Traceback (most recent call last):
  File "./bin/arv-put", line 6, in <module>
    from arvados.commands.put import main
ImportError: No module named 'arvados'
0.11 s = F

Actions #14

Updated by Nico César almost 4 years ago

  • Target version deleted (2020-09-09 Sprint)
  • Assigned To deleted (Nico César)

cd /usr/src/arvados/build ; bash ./run-tests.sh WORKSPACE=/usr/src/arvados ARVADOS_CONFIG=/etc/arvados/config.yml --temp ~/.cache/arvados-build --leave-temp

Actions #15

Updated by Nico César almost 4 years ago

  • Target version set to 2020-09-09 Sprint
  • Assigned To set to Nico César
Actions #16

Updated by Ward Vandewege almost 4 years ago

Actions #17

Updated by Ward Vandewege almost 4 years ago

Before merging into master, will need to remove the pycurl pin added here: https://dev.arvados.org/issues/16795#note-3

Actions #18

Updated by Nico César almost 4 years ago

  • Status changed from New to In Progress
Actions #19

Updated by Nico César almost 4 years ago

Almost there, but not 100%:

developer-run-tests: #2066

Actions #20

Updated by Nico César almost 4 years ago

If we use the python2.7 "own python" from package this is the behaviour:

nico@shell:~$ arv-get --version
/usr/bin/arv-get 2.0.3
nico@shell:~$ head --lines=1 /usr/bin/arv-get
#!/usr/share/python2.7/dist/python-arvados-python-client/bin/python
nico@shell:~$ arv-get --version 2> /dev/null
nico@shell:~$ 
1) Failure:
TestArvKeepGet#test_get_version [/tmp/workspace/developer-run-tests-remainder/sdk/cli/test/test_arv-keep-get.rb:31]:
STDOUT not expected: './bin/arv-get 2.1.0.dev20200826203407
'.
Expected "./bin/arv-get 2.1.0.dev20200826203407\n" to be empty.

this is the test from sdk/cli/test/test_arv-keep-get.rb

  def test_get_version
    out, err = capture_subprocess_io do
      assert_arv_get '--version'
    end
    assert_empty(out, "STDOUT not expected: '#{out}'")
    assert_match(/[0-9]+\.[0-9]+\.[0-9]+/, err, "Version information incorrect: '#{err}'")
  end
Actions #21

Updated by Nico César almost 4 years ago

nico@shell:~$ python3 /usr/bin/arv-get --version
Traceback (most recent call last):
  File "/usr/bin/arv-get", line 8, in <module>
    from arvados.commands.get import main
ModuleNotFoundError: No module named 'arvados'

because of the lack of that library installed:

nico@shell:~$ dpkg -l | grep python3-arvados-python-client 
nico@shell:~$ 

So here we have to make sure that when we change our packages to have the right dependency when deploying

Actions #22

Updated by Nico César almost 4 years ago

after installing:

nico@shell:~$ dpkg -l | grep python3-arvados-python-client 
ii  python3-arvados-python-client    2.0.4-1                      amd64        Arvados client library
nico@shell:~$ head /usr/bin/arv-get
#!/usr/share/python3/dist/python3-arvados-python-client/bin/python
nico@shell:~$ /usr/share/python3/dist/python3-arvados-python-client/bin/python /usr/bin/arv-get --version 2> /dev/null
/usr/bin/arv-get 2.0.4

Actions #24

Updated by Nico César almost 4 years ago

Actions #25

Updated by Ward Vandewege almost 4 years ago

A few comments:

  • shouldn't commit the go.sum and go.mod changes

noted

  • trailing space added in run-tests.sh line 657
  • run-tests.sh:
@@ -558,7 +555,7 @@ setup_ruby_environment() {
                 fi
             done
             "$bundle" version | tee /dev/stderr | grep -q 'version 2'
-        ) || fatal 'install bundler'
+        ) #|| fatal 'install bundler' ## This is needed for now.
     fi
 }

This seems like it should not be part of the patch?

  • The patch is adding a few "FIXME" lines, do we need additional changes there?
  • Any reason you're removing the '-x' from the bash invocation (2 places)?
Actions #26

Updated by Nico César almost 4 years ago

Did most of the changes.

about $pythonstuff...

install_all tries to do a pip install of the packages in $pythonstuff. as of 5aca41295

all of them have the suffix ":py3"

pythonstuff=(
    sdk/python:py3
    sdk/cwl:py3
    services/dockercleaner:py3
    services/fuse:py3
    tools/crunchstat-summary:py3
)

and I've seen some code that tries to remove the ones ending in ":py3"[1] by doing:

    for p in "${pythonstuff[@]}" 
    do
            dir=${p%:py3}

so I think we should remove this part.

My proposal is 4cfab58ae and running developer-run-tests: #2073 now

[1] man bash
${parameter%word}
${parameter%%word}
    Remove matching suffix pattern. The word is expanded to produce a pattern just as in pathname expansion. If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the ''%'' case) or the longest matching pattern (the ''%%'' case) deleted. If parameter is @ or *, the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with @ or *, the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list. 
Actions #27

Updated by Ward Vandewege almost 4 years ago

Nico César wrote:

Did most of the changes.

about $pythonstuff...

install_all tries to do a pip install of the packages in $pythonstuff. as of 5aca41295

all of them have the suffix ":py3"
[...]

and I've seen some code that tries to remove the ones ending in ":py3"[1] by doing:
[...]

so I think we should remove this part.

My proposal is 4cfab58ae and running developer-run-tests: #2073 now

[...]

Tests passed, LGTM. Just make sure after you merge master into your branch that you then change this line in sdk/python/setup.py:

'pycurl >=7.19.5.1, &lt;7.43.0.4&#039;, # 7.43.0.4 removes support for python2

to remove the '<7.43.0.4' part, which should no longer be needed after you purged python2. It was introduced in master after you forked.

Could also do that as a separate commit, of course.

Actions #28

Updated by Nico César almost 4 years ago

developer-run-tests: #2073 passed and ward said LGTM.

I'll go ahead and merge it

Actions #29

Updated by Anonymous almost 4 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #32

Updated by Ward Vandewege almost 4 years ago

Another remnant in arados-server, cf. 94a7479527652cd513a35f75e1da1798d9fdc213 on branch 15888-remove-more-python2-remnants.

Actions #33

Updated by Nico César almost 4 years ago

94a7479527652cd513a35f75e1da1798d9fdc213 LGTM,

I noticed developer-run-tests: #2083
with commit 94a7479527652cd513a35f75e1da1798d9fdc213, but it seems to be a flaky test.

Actions #34

Updated by Ward Vandewege almost 4 years ago

Nico César wrote:

94a7479527652cd513a35f75e1da1798d9fdc213 LGTM,

I noticed developer-run-tests: #2083
with commit 94a7479527652cd513a35f75e1da1798d9fdc213, but it seems to be a flaky test.

Thanks, merged it. It's a flaky test indeed.

Actions #35

Updated by Peter Amstutz almost 4 years ago

  • Release set to 25
Actions #36

Updated by Ward Vandewege almost 3 years ago

  • Related to Bug #17985: [a-c-r] add cwltool to the list of binstubs in our packages added
Actions

Also available in: Atom PDF