Story #12268

Update to standard libcloud 2.x

Added by Tom Morris about 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
09/14/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5
Release relationship:
Auto

Description

We want to stop using a private fork of libcloud, but we need the regression in the current version fixed and release first.

PR: https://github.com/apache/libcloud/pull/1110

Libcloud 2.2.1 was released September 21, 2017.


Subtasks

Task #12367: Review 12268-libcloud-2-3-0ResolvedLucas Di Pentima

Task #13181: Review 12268-libcloud230-officialResolvedLucas Di Pentima

Task #13189: Review 12268-build-fixResolvedWard Vandewege

Associated revisions

Revision 6331a6c3
Added by Lucas Di Pentima over 1 year ago

Merge branch '12268-libcloud-2-3-0'
Refs #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

Revision 07a09069
Added by Lucas Di Pentima over 1 year ago

Merge branch '12268-libcloud230-official'
Closes #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

Revision 1eb2ff90 (diff)
Added by Lucas Di Pentima over 1 year ago

12268: Fix merge conflict.
Refs #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

Revision a58a2636
Added by Lucas Di Pentima over 1 year ago

Merge branch '12268-build-fix'
Refs #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

History

#1 Updated by Tom Morris about 2 years ago

  • Description updated (diff)

#2 Updated by Tom Morris about 2 years ago

  • Tracker changed from Bug to Story
  • Description updated (diff)
  • Story points set to 0.5

#3 Updated by Lucas Di Pentima about 2 years ago

  • Assigned To set to Lucas Di Pentima

#4 Updated by Lucas Di Pentima about 2 years ago

  • Target version changed from 2017-10-11 Sprint to 2017-10-25 Sprint

#5 Updated by Lucas Di Pentima almost 2 years ago

  • Target version changed from 2017-10-25 Sprint to 2017-11-08 Sprint

#6 Updated by Lucas Di Pentima almost 2 years ago

  • Target version changed from 2017-11-08 Sprint to 2017-11-22 Sprint

#7 Updated by Lucas Di Pentima almost 2 years ago

  • Target version changed from 2017-11-22 Sprint to 2017-12-06 Sprint

#8 Updated by Lucas Di Pentima almost 2 years ago

  • Target version changed from 2017-12-06 Sprint to 2017-12-20 Sprint

#9 Updated by Lucas Di Pentima almost 2 years ago

  • Target version changed from 2017-12-20 Sprint to 2018-01-17 Sprint

#10 Updated by Tom Morris almost 2 years ago

  • Target version changed from 2018-01-17 Sprint to 2018-01-31 Sprint

#12 Updated by Tom Morris almost 2 years ago

The last of these PRs was merged Nov. 9, 2017 so any release dated later than that should be OK.

The latest release currently is still Libcloud 2.2.1, which was released September 21, 2017.

#13 Updated by Lucas Di Pentima over 1 year ago

  • Target version changed from 2018-01-31 Sprint to 2018-02-14 Sprint

#14 Updated by Lucas Di Pentima over 1 year ago

  • Status changed from New to In Progress

#15 Updated by Lucas Di Pentima over 1 year ago

Update at 318d42f0d - branch 12268-libcloud-2-3-0
Test run: https://ci.curoverse.com/job/developer-run-tests/563/

#16 Updated by Tom Clegg over 1 year ago

LGTM

#17 Updated by Lucas Di Pentima over 1 year ago

  • Target version changed from 2018-02-14 Sprint to 2018-02-28 Sprint

#18 Updated by Lucas Di Pentima over 1 year ago

Latest release still is 2.2.1 as of Feb 28th.

#19 Updated by Lucas Di Pentima over 1 year ago

  • Target version changed from 2018-02-28 Sprint to 2018-03-14 Sprint

#20 Updated by Lucas Di Pentima over 1 year ago

Libcloud 2.3.0 is released. The final commit wasn't exactly the one that we used as 2.2.2.dev4 but close, here's the diff:

https://github.com/apache/libcloud/compare/e7429e95b95ddd6789ee97ac484111db2e74c597...a119e396814bf12b7d5631f1f2d97a43c63020a5

#21 Updated by Lucas Di Pentima over 1 year ago

Updates at 8e2a634e2 - branch 12268-libcloud230-official
Test run: https://ci.curoverse.com/job/developer-run-tests/629/

Updates nodemanager's setup.py to request libcloud>=2.3, also updated run-tests.sh so it gets the package from PyPI when it doesn't exist as a fork.

There's a question pending to be answered about version requirements: should be keep asking >=2.3 or something more restrictive?

#22 Updated by Tom Clegg over 1 year ago

Do we still need this line in run-tests.sh? It seems like it prefers installing the curoverse version 2.3.0, and only falls back on pypi if that fails. Shouldn't we just go straight to pypi?

    || pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-$LIBCLOUD_PIN.zip >/dev/null \

rest LGTM, thanks.

#23 Updated by Lucas Di Pentima over 1 year ago

Update at 69ae9c5a8d7e2df93ece51afd4eb471c745eb84c
Test run: https://ci.curoverse.com/job/developer-run-tests/630/

Removed libcloud installation attempt from our forked version.

#24 Updated by Lucas Di Pentima over 1 year ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

#25 Updated by Ward Vandewege over 1 year ago

  • Status changed from Resolved to In Progress

Hmm all package building is now broken because the package testing fails, cf. https://ci.curoverse.com/job/build-packages-debian9/303/console for example:

02:04:14 START: arvados-node-manager test on arvados/package-test:debian9
02:04:17 W: The repository 'file:/arvados/packages/debian9  Release' is not signed.
02:04:18 Reading package lists...
02:04:18 Building dependency tree...
02:04:18 Reading state information...
02:04:18 Some packages could not be installed. This may mean that you have
02:04:18 requested an impossible situation or if you are using the unstable
02:04:18 distribution that some required packages have not yet been created
02:04:18 or been moved out of Incoming.
02:04:18 The following information may help to resolve the situation:
02:04:18 
02:04:18 The following packages have unmet dependencies:
02:04:18  arvados-node-manager : Depends: python-apache-libcloud (>= 2.3) but it is not going to be installed
02:04:18 E: Unable to correct problems, you have held broken packages.
02:04:19 ERROR: arvados-node-manager test on arvados/package-test:debian9 failed with exit status 100
02:04:19 

#26 Updated by Lucas Di Pentima over 1 year ago

Branch 12268-build-fix fixes the package build issue by adding apache-libcloud to build.list file, and removing the libcloud fork building code.

#27 Updated by Nico C├ęsar over 1 year ago

review at c6e1f628cc07731cd5fdee991fbd49564dab44b7

I see that this got eliminated:

# Forked libcloud
if test_package_presence "$PYTHON2_PKG_PREFIX"-apache-libcloud "$LIBCLOUD_PIN" python 2
then
  LIBCLOUD_DIR=$(mktemp -d)
  (
      cd $LIBCLOUD_DIR
      git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/libcloud.git .
      git checkout $DASHQ_UNLESS_DEBUG apache-libcloud-$LIBCLOUD_PIN
      # libcloud is absurdly noisy without -q, so force -q here
      OLD_DASHQ_UNLESS_DEBUG=$DASHQ_UNLESS_DEBUG
      DASHQ_UNLESS_DEBUG=-q
      handle_python_package
      DASHQ_UNLESS_DEBUG=$OLD_DASHQ_UNLESS_DEBUG
  )
  fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud "" python "" --iteration 2
  rm -rf $LIBCLOUD_DIR
fi

which is good ... but I have the strong feeling that we'll be forking again sometime in the future

can you add the following comment instead:

## if libcloud becomes our own fork see 
## https://dev.arvados.org/issues/12268#note-27

besides that is ready to merge.

#28 Updated by Lucas Di Pentima over 1 year ago

  • Status changed from In Progress to Resolved

#29 Updated by Tom Morris about 1 year ago

  • Release set to 17

Also available in: Atom PDF