Arvados » Arvados Epics

1. We have ssh public keys for users, couldn't we use that for authentication?
2. It looks like you're proposing tunnel the bare stdin/stdout streams over websockets (what about stderr?). Do we allocate a pty on the container host? Don't ptys have some out-of-band signaling, like communicating when ^C has been pressed? I'm pretty sure there's some hidden complexity in providing a good quality interactive session.

My idea for something like this would be:

• arvados-client looks up the IP address and port of crunch-run shell service
• construct the ssh command line and use that to connect to switchyard. (maybe we embed the switchyard functionality into controller?)
• Switchyard tunnels to the shell service
• crunch-run shell service authorizes the user based on their ssh public key, then runs "docker exec -it ..."

Peter Amstutz wrote:

1. We have ssh public keys for users, couldn't we use that for authentication?

The idea was to not require ssh keys and use native Arvados authentication instead.

1. It looks like you're proposing tunnel the bare stdin/stdout streams over websockets (what about stderr?).

No, actually, websockets was only mentioned as a reference because we can use the same connection hijacking technique we use there. We were thinking of using ssh under the hood, I've updated the proposal accordingly.

Do we allocate a pty on the container host? Don't ptys have some out-of-band signaling, like communicating when ^C has been pressed? I'm pretty sure there's some hidden complexity in providing a good quality interactive session.

Indeed, which is why it would be ideal to use the ssh protocol as much as possible.

My idea for something like this would be:

• arvados-client looks up the IP address and port of crunch-run shell service
• construct the ssh command line and use that to connect to switchyard. (maybe we embed the switchyard functionality into controller?)
• Switchyard tunnels to the shell service
• crunch-run shell service authorizes the user based on their ssh public key, then runs "docker exec -it ..."

This sounds similar to the proposal. Effectively, we'd be building something like switchyard into controller, but not native ssh. That's because of the final `docker exec` step to get from the compute node to the container, which makes it impossible to go the whole way.

Ward Vandewege wrote:

Peter Amstutz wrote:

1. We have ssh public keys for users, couldn't we use that for authentication?

The idea was to not require ssh keys and use native Arvados authentication instead.

1. It looks like you're proposing tunnel the bare stdin/stdout streams over websockets (what about stderr?).

No, actually, websockets was only mentioned as a reference because we can use the same connection hijacking technique we use there. We were thinking of using ssh under the hood, I've updated the proposal accordingly.

I see, you mean doing a protocol upgrade (e.g. https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header) where it logically switches to ssh.

Is the idea that the client can do something like

ssh -o "ProxyCommand arvados-client shell my-container-uuid" blahblah.arvadosapi.com

And that connects to controller, and controller figures out the container's address and proxies the connection to the container?

This sounds similar to the proposal. Effectively, we'd be building something like switchyard into controller, but not native ssh. That's because of the final `docker exec` step to get from the compute node to the container, which makes it impossible to go the whole way.

I feel like the server side (using the Go ssh server) of this more straightforward, it just needs to accept the connection, verify the user (using their public key) and then invoke "docker exec" with the PTY properly connected up.

Minimum viable feature

• Use token as password authentication
• Connect to crunch-run by ssh
• Crunch-run will update the container record with the IP address and port where it is listening
• Connecting to crunch-run will run "docker exec -ti" on the container with stdin/stdout/stderr connected.

Initial proof of concept: go program that runs ssh server and forks and runs bash, with pseudoterminal support so that control characters work as expected.

https://godoc.org/golang.org/x/crypto/ssh#example-NewServerConn

From 23 December 2020 discussion:

• General architecture
  • arvados-client provides a command that connects to controller & forwards stdin/stdout
  • controller forwards ssh connection to crunch-run
  • crunch-run accept ssh connection & supports "session", "pty-req", "env" and "shell" messages
• For port forwarding, support "tcpip-forward" message
• User convenience command that invokes ssh -o "ProxyCommand arvados-client ..." for you to get a shell or a port forward.

for reference, ssh messages: https://tools.ietf.org/html/rfc4254