Project

General

Profile

Actions

Bug #21999

closed

Support compute nodes with /tmp mounted with "noexec" flag

Added by Lucas Di Pentima 5 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Deployment
Story points:
-
Release:
Release relationship:
Auto

Description

One of the advisories from the Center for Internet Security regarding hardening hosts is mounting the /tmp filesystem with noexec. This produces the following issue when creating a compute node image:

Compute AMI creation fails: The compute image creation base script attempts to execute a program in /tmp

tools/compute-images/scripts/base.sh:135:  unzip -q /tmp/awscliv2.zip -d /tmp && $SUDO /tmp/aws/install


Subtasks 1 (0 open1 closed)

Task #22118: Review 21999-packer-fixesResolvedLucas Di Pentima10/10/2024Actions

Related issues 2 (0 open2 closed)

Related to Arvados - Feature #22029: arvados-dispatch-cloud option to use a different directory than /tmp for staging the crunch-run binaryResolvedTom CleggActions
Blocked by Arvados - Support #22030: Have a testing environment where /tmp is noexecResolvedLucas Di PentimaActions
Actions #1

Updated by Peter Amstutz 5 months ago

So if the problem is mainly using /tmp, we should be using a different directory, maybe /root or /var/spool/something ?

Actions #2

Updated by Lucas Di Pentima 5 months ago

Yes, I was thinking in just using $HOME/tmp for every case. Maybe a-d-c would benefit of a new config knob to set that directory to some sensible (or /tmp) default?

Actions #3

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-08-07 sprint to Development 2024-08-28 sprint
Actions #4

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-08-28 sprint to Development 2024-08-07 sprint
Actions #5

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-08-07 sprint to Development 2024-08-28 sprint
  • Description updated (diff)
Actions #6

Updated by Peter Amstutz 5 months ago

TODO split this into 3 tickets.

Actions #7

Updated by Peter Amstutz 4 months ago

  • Related to Feature #22029: arvados-dispatch-cloud option to use a different directory than /tmp for staging the crunch-run binary added
Actions #8

Updated by Peter Amstutz 4 months ago

  • Description updated (diff)
Actions #9

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-08-28 sprint to Development 2024-09-11 sprint
Actions #10

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-09-11 sprint to Development 2024-09-25 sprint
Actions #11

Updated by Peter Amstutz 4 months ago

  • Blocked by Support #22030: Have a testing environment where /tmp is noexec added
Actions #12

Updated by Lucas Di Pentima 3 months ago

  • Assigned To set to Lucas Di Pentima
Actions #13

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-09-25 sprint to Development 2024-10-09 sprint
Actions #14

Updated by Lucas Di Pentima 2 months ago

  • Status changed from New to In Progress
Actions #15

Updated by Lucas Di Pentima 2 months ago

21999-packer-fixes @ 72d634e3cf

packer-build-compute-image: #260

  • All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
    • Yes
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • This story doesn't explicitly mention that the packer scripts should work with a CIS L1 AMI (ami-0dc4762eb2bdd9a25). I've done some manual testing and it fails on the singularity build stage, unrelated to running things in /tmp.
  • Code is tested and passing, both automated and manual, what manual testing was done is described.
    • I've manually tested creating an AMI passing --workdir /home/admin and it worked fine.
  • New or changed UX/UX and has gotten feedback from stakeholders.
    • Just a new --workdir <path> argument
  • Documentation has been updated.
    • Yes
  • Behaves appropriately at the intended scale (describe intended scale).
    • N/A
  • Considered backwards and forwards compatibility issues between client and server.
    • Yes. The working directory is now parametrized, and defaults to /tmp so it's backwards compatible.
  • Follows our coding standards and GUI style guidelines.
    • N/A

Because this story is about being able to support base images with /tmp mounted with noexec, I made it possible for the user to specify another path instead of /tmp.
Additional work is required to make it work with CIS Level 1 Debian 11, because for some reason when trying to build singularity it doesn't detect the Go compiler (although there's no error output related to the Go compiler installation phase):

...
    amazon-ebs: Processing triggers for libc-bin (2.31-13+deb11u11) ...
    amazon-ebs: Configuring for project `singularity-ce' with languages: C, Golang
    amazon-ebs: => running pre-basechecks project specific checks ...
    amazon-ebs: => running base system checks ...
    amazon-ebs:  checking: host C compiler... cc
    amazon-ebs:  checking: host C++ compiler... c++
    amazon-ebs:  checking: host Go compiler (at least version 1.17)... not found!
    amazon-ebs: mconfig: could not complete configuration
==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored after 6 minutes 7 seconds: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]

If we're going to officially support these CIS issued images, I can continue investigating the cause of the failures and make an official pipeline on Jenkins for it on this ticket, or maybe on a separate one.

Actions #16

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-10-09 sprint to Development 2024-10-23 sprint
Actions #17

Updated by Brett Smith 2 months ago

Lucas Di Pentima wrote in #note-15:

21999-packer-fixes @ 72d634e3cf

Just one --workdir documentation nit: in the phrase "The directory on which," using the preposition "on" to describe stuff that happens "in" the directory sounds just slightly off to native English speaker ears. Suggest instead:

The directory where data files are staged and setup scripts are run

If you want to workshop it further in chat, happy to do that. Please go ahead and merge, thanks.

Actions #18

Updated by Lucas Di Pentima 2 months ago

Thanks for the suggestion, fixed and merged!

Actions #19

Updated by Lucas Di Pentima 2 months ago

  • Status changed from In Progress to Resolved
Actions #20

Updated by Peter Amstutz about 2 months ago

  • Release set to 70
Actions

Also available in: Atom PDF