Story #17103

Developer shell inside running container

Added by Peter Amstutz 10 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
01/01/2021
Due date:
03/31/2021
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

Ability to connect to a shell running inside a crunch container.

Draft implementation proposal:

1. When dispatcher starts crunch-run, it includes the hmac of systemroottoken and container id as a cli argument to crunch-run. Crunch-run can use that hmac to validate incoming shell connections (see below).

2. When crunch-run starts, it updates the container record with its ip and the port it listens on for shell connections. TBD: best name for this field.

3. Users can invoke this command to connect to a container:

arvados-client shell --user username container_uuid command...

(--user username is optional)

4. When invoked like that, arvados-client connects to controller on the standard port, using a new shell endpoint. Controller verifies the container record is readable by this user. If so, it uses connection hijacking (like we do in websockets) to set up bidirectional communication with arvados-client. That way controller doesn't need to listen on another port.

Controller then forwards the connection to the container, using the host and port recorded by crunch-run in the container record. It authenticates the request by sending along

hmac (hmac(systemroottoken and container id) and the container ip and the container shell port)

Crunch-run validates the hmac. We do hmac of the hmac for two reasons: it will make accidental connections to incorrect containers impossible. Also, it avoids malicious connections from another node on which the container was attempted to be started before: that node would also be able to generate hmac(systemroottoken and container id), but it wouldn't be able to know the ip + shell port combination of the container, so it wouldn't be able to compute the final hmac.

5. Assuming the connection validates, crunch-run then executes

docker exec -uid username -it docker_container_id command

or, if the username was not specified,

docker exec -it docker_container_id command

and it connects stdin/stdout/stderr with the incoming connection. The communication between arvados-client and crunch-run uses the ssh protocol under the hood, but that's transparent to the user. Because we are using docker exec between crunch-run and the container, we can't use ssh all the way through, which is unfortunate. The alternative would be to always have an ssh server inside the container, but that would be much more complicated.

Open questions:

  • connecting with ssh should probably mark the container as not eligible for reuse. How should we approach that?
  • a systemwide setting should probably govern the availability of the container shell feature (disabled/default off/default on ?)
  • maybe the availability of the shell feature should be a runtime constraint, like api_access?

Related issues

Related to Arvados - Feature #17170: Shell into container proof of conceptResolved01/14/2021

History

#1 Updated by Peter Amstutz 10 months ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 10 months ago

  • Start date set to 01/01/2021
  • Due date set to 03/31/2021

#3 Updated by Ward Vandewege 10 months ago

  • Description updated (diff)

#4 Updated by Ward Vandewege 10 months ago

  • Description updated (diff)

#5 Updated by Ward Vandewege 10 months ago

  • Description updated (diff)

#6 Updated by Ward Vandewege 10 months ago

  • Description updated (diff)

#7 Updated by Peter Amstutz 10 months ago

  1. We have ssh public keys for users, couldn't we use that for authentication?
  2. It looks like you're proposing tunnel the bare stdin/stdout streams over websockets (what about stderr?). Do we allocate a pty on the container host? Don't ptys have some out-of-band signaling, like communicating when ^C has been pressed? I'm pretty sure there's some hidden complexity in providing a good quality interactive session.

My idea for something like this would be:

  • arvados-client looks up the IP address and port of crunch-run shell service
  • construct the ssh command line and use that to connect to switchyard. (maybe we embed the switchyard functionality into controller?)
  • Switchyard tunnels to the shell service
  • crunch-run shell service authorizes the user based on their ssh public key, then runs "docker exec -it ..."

#8 Updated by Ward Vandewege 10 months ago

  • Description updated (diff)

#9 Updated by Ward Vandewege 10 months ago

Peter Amstutz wrote:

  1. We have ssh public keys for users, couldn't we use that for authentication?

The idea was to not require ssh keys and use native Arvados authentication instead.

  1. It looks like you're proposing tunnel the bare stdin/stdout streams over websockets (what about stderr?).

No, actually, websockets was only mentioned as a reference because we can use the same connection hijacking technique we use there. We were thinking of using ssh under the hood, I've updated the proposal accordingly.

Do we allocate a pty on the container host? Don't ptys have some out-of-band signaling, like communicating when ^C has been pressed? I'm pretty sure there's some hidden complexity in providing a good quality interactive session.

Indeed, which is why it would be ideal to use the ssh protocol as much as possible.

My idea for something like this would be:

  • arvados-client looks up the IP address and port of crunch-run shell service
  • construct the ssh command line and use that to connect to switchyard. (maybe we embed the switchyard functionality into controller?)
  • Switchyard tunnels to the shell service
  • crunch-run shell service authorizes the user based on their ssh public key, then runs "docker exec -it ..."

This sounds similar to the proposal. Effectively, we'd be building something like switchyard into controller, but not native ssh. That's because of the final `docker exec` step to get from the compute node to the container, which makes it impossible to go the whole way.

#10 Updated by Peter Amstutz 10 months ago

Ward Vandewege wrote:

Peter Amstutz wrote:

  1. We have ssh public keys for users, couldn't we use that for authentication?

The idea was to not require ssh keys and use native Arvados authentication instead.

  1. It looks like you're proposing tunnel the bare stdin/stdout streams over websockets (what about stderr?).

No, actually, websockets was only mentioned as a reference because we can use the same connection hijacking technique we use there. We were thinking of using ssh under the hood, I've updated the proposal accordingly.

I see, you mean doing a protocol upgrade (e.g. https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header) where it logically switches to ssh.

Is the idea that the client can do something like

ssh -o "ProxyCommand arvados-client shell my-container-uuid" blahblah.arvadosapi.com

And that connects to controller, and controller figures out the container's address and proxies the connection to the container?

This sounds similar to the proposal. Effectively, we'd be building something like switchyard into controller, but not native ssh. That's because of the final `docker exec` step to get from the compute node to the container, which makes it impossible to go the whole way.

I feel like the server side (using the Go ssh server) of this more straightforward, it just needs to accept the connection, verify the user (using their public key) and then invoke "docker exec" with the PTY properly connected up.

#11 Updated by Peter Amstutz 10 months ago

Minimum viable feature

  • Use token as password authentication
  • Connect to crunch-run by ssh
  • Crunch-run will update the container record with the IP address and port where it is listening
  • Connecting to crunch-run will run "docker exec -ti" on the container with stdin/stdout/stderr connected.

Initial proof of concept: go program that runs ssh server and forks and runs bash, with pseudoterminal support so that control characters work as expected.

https://godoc.org/golang.org/x/crypto/ssh#example-NewServerConn

#12 Updated by Tom Clegg 10 months ago

  • Related to Feature #17170: Shell into container proof of concept added

#13 Updated by Peter Amstutz 9 months ago

From 23 December 2020 discussion:

  • General architecture
    • arvados-client provides a command that connects to controller & forwards stdin/stdout
    • controller forwards ssh connection to crunch-run
    • crunch-run accept ssh connection & supports "session", "pty-req", "env" and "shell" messages
  • For port forwarding, support "tcpip-forward" message
  • User convenience command that invokes ssh -o "ProxyCommand arvados-client ..." for you to get a shell or a port forward.

for reference, ssh messages: https://tools.ietf.org/html/rfc4254

#14 Updated by Tom Clegg 9 months ago

  • Related to Feature #17206: crunch-run reverse proxies HTTP requests to container added

#15 Updated by Tom Clegg 9 months ago

  • Related to Story #17207: External access to web services running in containers added

#16 Updated by Peter Amstutz 8 months ago

  • Status changed from New to In Progress

#17 Updated by Peter Amstutz 7 months ago

  • Related to deleted (Feature #17206: crunch-run reverse proxies HTTP requests to container)

#18 Updated by Peter Amstutz 7 months ago

  • Related to deleted (Story #17207: External access to web services running in containers)

#19 Updated by Peter Amstutz 7 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF