Arvados

Testing on shell.4xphq

$ /usr/share/python2.7/dist/python-arvados-python-client/bin/python -c 'import future_builtins'
ImportError: No module named future_builtins

$ python -c 'import future_builtins'
# works great

$ find /usr -name "future_builtins*" 
/usr/share/python2.7/dist/libpam-arvados/local/lib/python2.7/lib-dynload/future_builtins.x86_64-linux-gnu.so
/usr/share/python2.7/dist/libpam-arvados/lib/python2.7/lib-dynload/future_builtins.x86_64-linux-gnu.s
/usr/lib/python2.7/lib-dynload/future_builtins.x86_64-linux-gnu.so

$ /usr/share/python2.7/dist/libpam-arvados/bin/python -c 'import future_builtins'
# works great

$ /usr/share/python2.7/dist/python-arvados-python-client/bin/python -c 'import sys; import pprint ; pprint.pprint(sys.path)'
['',
 '/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7',
 '/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-tk',
 '/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-old',
 '/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/site-packages']

$ python -c 'import sys; import pprint ; pprint.pprint(sys.path)'
['',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages']

So the problem seems to be that future_builtins.x86_64-linux-gnu.so is not installed into the virtualenv's lib-dynload, but it doesn't look at the system lib-dynload either.

Peter Amstutz wrote:

So the problem seems to be that future_builtins.x86_64-linux-gnu.so is not installed into the virtualenv's lib-dynload, but it doesn't look at the system lib-dynload either.

This appears to be a packaging problem.

 apt-cache policy python-arvados-python-client |head -n11
python-arvados-python-client:
  Installed: 1.3.1.20190306194329-1
  Candidate: 1.3.1.20190306194329-1
  Version table:
 *** 1.3.1.20190306194329-1 500
        500 http://apt.arvados.org stretch-dev/main amd64 Packages
        100 /var/lib/dpkg/status
     1.3.1.20190301150258-1 500
        500 http://apt.arvados.org stretch-dev/main amd64 Packages
     1.3.1.20190227212550-1 500
        500 http://apt.arvados.org stretch-dev/main amd64 Packages

The problem we're seeing in this ticket is that the lib-dynload directory on these machines is empty:

# ls -laF /usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload
total 8
drwxr-xr-x 2 root root 4096 Mar  8 21:03 ./
drwxr-xr-x 6 root root 4096 Mar  8 21:03 ../

It's not supposed to be.

Purging the python-arvados-python-client package and reinstalling the latest resolves the problem, because the lib-dynload directory is now a symlink to the systemwide lib-dynload directory which is populated properly:

ls -laFd /usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload
lrwxrwxrwx 1 root root 30 Mar  6 21:18 /usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload -> /usr/lib/python2.7/lib-dynload/

So how did we get here then?

I can reproduce getting into this situation by purging the python-arvados-python-client package, installing version python-arvados-python-client_1.3.1.20190227212550-1_amd64.deb, and then upgrading to the next version, python-arvados-python-client_1.3.1.20190301150258-1_amd64.deb.

There is apparently a difference in the packaging for python-arvados-python-client_1.3.1.20190227212550-1_amd64.deb and python-arvados-python-client_1.3.1.20190301150258-1_amd64.deb:

# dpkg -c python-arvados-python-client_1.3.1.20190227212550-1_amd64.deb |grep /usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynloaddrwxr-xr-x 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/bz2.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/bz2.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/termios.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/termios.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_ctypes_test.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_ctypes_test.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_bsddb.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_bsddb.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_curses.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_curses.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/crypt.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/crypt.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/parser.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/parser.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_codecs_iso2022.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_codecs_iso2022.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_sqlite3.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_sqlite3.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_multiprocessing.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_multiprocessing.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/ossaudiodev.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/ossaudiodev.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/Python-2.7.egg-info -> /usr/lib/python2.7/lib-dynload/Python-2.7.egg-info
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_csv.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_csv.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_multibytecodec.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_multibytecodec.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_elementtree.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_elementtree.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_codecs_kr.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_codecs_kr.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_codecs_hk.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_codecs_hk.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/linuxaudiodev.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/linuxaudiodev.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/future_builtins.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/future_builtins.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_codecs_cn.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_codecs_cn.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/mmap.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/mmap.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/audioop.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/audioop.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/nis.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/nis.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_testcapi.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_testcapi.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/dbm.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/dbm.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_hotshot.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_hotshot.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_codecs_jp.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_codecs_jp.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/fpectl.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/fpectl.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_curses_panel.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_curses_panel.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/resource.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/resource.x86_64-linux-gnu.so
lrwxrwxrwx 0/0               0 2019-02-27 22:27 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload/_codecs_tw.x86_64-linux-gnu.so -> /usr/lib/python2.7/lib-dynload/_codecs_tw.x86_64-linux-gnu.so

# dpkg -c python-arvados-python-client_1.3.1.20190301150258-1_amd64.deb |grep /usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload
lrwxrwxrwx 0/0               0 2019-03-01 18:30 ./usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/lib-dynload -> /usr/lib/python2.7/lib-dynload

Upgrading from the former to the latter screws up the lib-dynload directory in the virtualenv.

But what caused this packaging change?

The last successful deploy to c97qk is https://ci.curoverse.com/job/deploy-to-c97qk/2069/ (Mar 1)

The futures_builtins error appears in https://ci.curoverse.com/job/deploy-to-c97qk/2074/ (Mar 5)

More manifestations of this upstream bug:

Having an empty 'encodings' folder like this:

 v /usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/encodings
total 28
drwxr-xr-x 2 root root 20480 Mar  1 19:04 ./
drwxr-xr-x 6 root root  4096 Mar  8 23:04 ../

The fix is easy:

apt-get remove python-arvados-python-client; apt-get install python-arvados-python-client

The same can happen for the python-arvados-fuse package, and the python-arvados-cwl-runner package.