Feature #3219

[Docker] further docker image improvements

Added by Ward Vandewege almost 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
06/10/2014
Due date:
% Done:

100%

Estimated time:
(Total: 2.00 h)
Story points:
1.0

Subtasks

Task #3331: add crunch-dispatchResolvedWard Vandewege

Task #3330: add compute nodesResolvedWard Vandewege

Task #3337: review 3219-further-docker-improvementsResolvedWard Vandewege

Task #3332: add local git treesResolvedWard Vandewege

Task #3253: add shell imageResolvedWard Vandewege


Related issues

Has duplicate Arvados - Story #3020: [Deployment] Docker installation runs crunch-dispatch and worker nodesClosed

Associated revisions

Revision acc4448c (diff)
Added by Ward Vandewege almost 6 years ago

Whitespace fix.

refs #3219

Revision 60c18cbf (diff)
Added by Ward Vandewege almost 6 years ago

Be consistent about the version of ruby that is installed by rvm and
used by the various containers. Be consistent about using rvm-exec and
bundle exec in every command that uses ruby.

Add documentation and warning about --realclean option to build.sh

refs #3219

Revision 85706e06
Added by Ward Vandewege almost 6 years ago

Merge branch '3219-further-docker-improvements'

refs #3219

Revision 171f4d29 (diff)
Added by Ward Vandewege almost 6 years ago

Sanitize how we handle the --ssh option in the api docker container.

refs #3219

Revision 56e82fff (diff)
Added by Ward Vandewege almost 6 years ago

Be smarter about building and the --realclean option:

  • remove intermediate containers after a successful build
  • --realclean removes only images labelled arvados/*
  • --realclean removes only arvados containers

refs #3219

Revision b93a012b (diff)
Added by Ward Vandewege almost 6 years ago

Update the documentation to reflect the new, safer build.sh --realclean
behavior.

refs #3219

Revision 8b185ea4
Added by Ward Vandewege almost 6 years ago

Merge branch '3219-further-docker-improvements'

refs #3219

Revision 652aec8c (diff)
Added by Ward Vandewege almost 6 years ago

First set of changes to add crunch-dispatch and compute nodes. This is not complete yet.

refs #3219

Revision 031db3fc (diff)
Added by Ward Vandewege almost 6 years ago

Two compute nodes work at the slurm level. Crunch-dispatch runs.

refs #3219

Revision ee0896bf (diff)
Added by Ward Vandewege almost 6 years ago

Compute nodes can run jobs - almost. The final arv-put --raw in the
collation step still fails (file descriptor error?).

refs #3219

Revision cb63ae17 (diff)
Added by Ward Vandewege almost 6 years ago

Compute nodes can now run jobs. Also, small fixes all over the place.

refs #3219

Revision 150f495e (diff)
Added by Ward Vandewege almost 6 years ago

Add local git repository support.

refs #3219

Revision c58d996c (diff)
Added by Ward Vandewege almost 6 years ago

Further changes to make pipeline running work reliably.

refs #3219

Revision 94c6d21a (diff)
Added by Ward Vandewege almost 6 years ago

Whitespace cleanup.

refs #3219

Revision 33f52182 (diff)
Added by Ward Vandewege almost 6 years ago

  • Add shell node to docker installation.
  • Remove documentation warning about missing crunch support.
  • Various small improvements

refs #3219

Revision 18717f22 (diff)
Added by Ward Vandewege almost 6 years ago

arvdock starts 2 compute nodes, always.

refs #3219

Revision 97f3ab85 (diff)
Added by Ward Vandewege almost 6 years ago

  • Update docs to reflect that crunch is now supported.
  • Rename --mrproper to --deepclean

refs #3219

Revision ff1b1d43 (diff)
Added by Ward Vandewege almost 6 years ago

  • move the downloading of skydns and skydock from arvdock to
    build_tools/Makefile
  • add echo statements to the start of build steps in
    build_tools/Makefile

refs #3219

Revision 69369eeb (diff)
Added by Ward Vandewege almost 6 years ago

Whitespace fixes

refs #3219

Revision aaafb72c (diff)
Added by Ward Vandewege almost 6 years ago

Make the build script prompt for a shell login name.

refs #3219

Revision ba4d612f (diff)
Added by Ward Vandewege almost 6 years ago

Make --deepclean also remove the crosbymichael/skydns and crosbymichael/skydock images.

refs #3219

Revision 5e062b44 (diff)
Added by Ward Vandewege almost 6 years ago

Suppress warnings and spurious error messages for the clean, realclean
and deepclean targets.

refs #3219

Revision 386aae03 (diff)
Added by Ward Vandewege almost 6 years ago

  • add argument help to build.sh
  • fix clean/realclean/deepclean invocation syntax in documentation

refs #3219

Revision 6c7373c9 (diff)
Added by Ward Vandewege almost 6 years ago

Fix the auto_admin_user feature; it was broken by accident in
5f651773

refs #3219

Revision 735d69e6
Added by Ward Vandewege almost 6 years ago

Merge branch '3219-further-docker-improvements'

refs #3219

History

#1 Updated by Ward Vandewege almost 6 years ago

  • Subject changed from further docker image improvements to [LOCAL] further docker image improvements

#2 Updated by Ward Vandewege almost 6 years ago

  • Target version changed from 2014-07-16 Sprint to 2014-08-06 Sprint

#3 Updated by Ward Vandewege almost 6 years ago

  • Assigned To set to Ward Vandewege

#4 Updated by Ward Vandewege almost 6 years ago

  • Subject changed from [LOCAL] further docker image improvements to [Docker] further docker image improvements

#5 Updated by Ward Vandewege almost 6 years ago

Review instructions:

check out the 3219-further-docker-improvements branch.

build the docs:

cd doc
rake

Then follow the 'installation with docker' instructions and see if it works for you. Building the images takes a while, ~20 min on my machine (and YMMV)

#6 Updated by Tim Pierce almost 6 years ago

From a fresh Docker build, after make realclean and deepclean, I cannot log in because my account is not automatically made active. My config.yml file has:

API_AUTO_ADMIN_USER: 

and after attempting to log in, the following User record is created in the database (reformatted for readability):

2.1.2 :003 > User.find_by_email("twp@curoverse.com")
  User Load (1.7ms)  SELECT "users".* FROM "users" WHERE "users"."email" = 'twp@curoverse.com' LIMIT 1
 => #<User id: 34,
      uuid: "twp01-tpzed-ty5p61c28oxznuf",
      owner_uuid: "twp01-tpzed-000000000000000",
      created_at: "2014-07-28 16:25:41",
      modified_by_client_uuid: nil,
      modified_by_user_uuid: "twp01-tpzed-000000000000000",
      modified_at: "2014-07-28 16:25:41",
      email: "twp@curoverse.com",
      first_name: "Tim",
      last_name: "Pierce",
      identity_url: "https://www.google.com/accounts/o8/id?id=AItOawkbYj...",
      is_admin: nil,
      prefs: {},
      updated_at: "2014-07-28 16:25:41",
      default_owner_uuid: nil,
      is_active: false> 

#7 Updated by Ward Vandewege almost 6 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF