Arvados

Libcloud dev version includes 4 PRs submitted by us, as described on https://github.com/apache/libcloud/blob/trunk/CHANGES.rst

• https://github.com/apache/libcloud/pull/1120
• https://github.com/apache/libcloud/pull/1134
• https://github.com/apache/libcloud/pull/1137
• https://github.com/apache/libcloud/pull/1139

The last of these PRs was merged Nov. 9, 2017 so any release dated later than that should be OK.

The latest release currently is still Libcloud 2.2.1, which was released September 21, 2017.

Update at 318d42f0d - branch 12268-libcloud-2-3-0
Test run: https://ci.curoverse.com/job/developer-run-tests/563/

• Made a new branch on our libcloud fork with 2.3.0 RC version (https://github.com/curoverse/libcloud/tree/apache-libcloud-2.2.2.dev4)
• Updated nodemanager dependency to this version so we can do some testing.
• Ran local nodemanager integration tests: passed OK, now running all the tests on Jenkins before merging.

LGTM

Latest release still is 2.2.1 as of Feb 28th.

Libcloud 2.3.0 is released. The final commit wasn't exactly the one that we used as 2.2.2.dev4 but close, here's the diff:

https://github.com/apache/libcloud/compare/e7429e95b95ddd6789ee97ac484111db2e74c597...a119e396814bf12b7d5631f1f2d97a43c63020a5

Updates at 8e2a634e2 - branch 12268-libcloud230-official
Test run: https://ci.curoverse.com/job/developer-run-tests/629/

Updates nodemanager's setup.py to request libcloud>=2.3, also updated run-tests.sh so it gets the package from PyPI when it doesn't exist as a fork.

There's a question pending to be answered about version requirements: should be keep asking >=2.3 or something more restrictive?

Do we still need this line in run-tests.sh? It seems like it prefers installing the curoverse version 2.3.0, and only falls back on pypi if that fails. Shouldn't we just go straight to pypi?

    || pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-$LIBCLOUD_PIN.zip >/dev/null \

rest LGTM, thanks.

Update at 69ae9c5a8d7e2df93ece51afd4eb471c745eb84c
Test run: https://ci.curoverse.com/job/developer-run-tests/630/

Removed libcloud installation attempt from our forked version.

Branch 12268-build-fix fixes the package build issue by adding apache-libcloud to build.list file, and removing the libcloud fork building code.

review at c6e1f628cc07731cd5fdee991fbd49564dab44b7

I see that this got eliminated:

# Forked libcloud
if test_package_presence "$PYTHON2_PKG_PREFIX"-apache-libcloud "$LIBCLOUD_PIN" python 2
then
  LIBCLOUD_DIR=$(mktemp -d)
  (
      cd $LIBCLOUD_DIR
      git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/libcloud.git .
      git checkout $DASHQ_UNLESS_DEBUG apache-libcloud-$LIBCLOUD_PIN
      # libcloud is absurdly noisy without -q, so force -q here
      OLD_DASHQ_UNLESS_DEBUG=$DASHQ_UNLESS_DEBUG
      DASHQ_UNLESS_DEBUG=-q
      handle_python_package
      DASHQ_UNLESS_DEBUG=$OLD_DASHQ_UNLESS_DEBUG
  )
  fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud "" python "" --iteration 2
  rm -rf $LIBCLOUD_DIR
fi

which is good ... but I have the strong feeling that we'll be forking again sometime in the future

can you add the following comment instead:

## if libcloud becomes our own fork see 
## https://dev.arvados.org/issues/12268#note-27

besides that is ready to merge.