Clojurians Log v2

For maintenance access to private networks, there are 2 common patterns: • have a hardened “jump box” in a public subnet which can route to your private networks • Have a vpn that allows you to make the private networks routable when connected. AWS has a service you can use for this but it’s on the expensive side. It’s also common to stand up openvpn or cisco instances in public subnets, there are amis in the marketplace that make this easy to do • Use one of the aws services that help with this: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-eic.html and https://dev.to/aws-builders/how-to-set-up-session-manager-and-enable-ssh-over-ssm-43k9. Both of these don’t require additional infrastructure but have limitations on where they can be used. They’re more secure because the ssh connection is made over AWS’s API layer rather than having one of your instances exposed to the internet. The jump box approach is the simplest to set up but least secure. VPNs offer the greatest flexibility and ease of use for devs at the cost of being more complex to set up. The aws services work great for just instances but wouldn’t let you eg connect to a database (I think, I’m less familiar with these). I think a nice hybrid approach for small projects might be to set up a jump box that you connect to via one of the aws services

I just typed this quickly on my phone so I didn’t go into much depth, but this should be enough to get you started. Happy to answer more qs too

From a security perspective the simplest stance is to never expose any machines to the internet that you are responsible for patching, ie only expose aws services or at least marketplace amis specifically hardened for it

ssm-agent can do port forwarding, so you can also have a jump box inside private network (without access from internet) and run ssm-agent on the jump box, and allow traffic from jump box security group to backend security group to nrepl port depending how you run the backend, if it is ec2 vms that you autoscale, you could also run ssm-agent on those instances too If you are using ECS, it has https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html, which allows to run commands and shell via docker exec, inside the backend containers. Useful for debugging, but I haven't looked if it would be possible to tunnel nrepl with that yet

Hmm, seems that there are tools for tunneling to ecs tasks https://github.com/winebarrel/ecs-exec-pf, https://github.com/alastairmccormack/ecs-tunnel Haven't tried these yet. But anyway, vpn or a jump box, and with ssm-agent, the jump box doesn't need to be accessibe from public internet, and ssm-agent could be run alongside backend

Small update on my prior comment since I had fun reading into instance connect: there’s a new offering called ec2 instance connect endpoint which seems generally similar to SSM in that you can do an api call to get an aim-authenticated ssh tunnel, except you don’t need a jump box in the middle and you don’t have to install agents anywhere. The main tradeoff seems to be that you only get network-level control over who can connect to what vs ssm’s granular role-based rules

I remember that too now, this one? https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2-instance-connect/open-tunnel.html

That’s the one

I think when it came out, they initially allowed to connect to any remote port, but then limited to port 22 only. But seems that it now allows for other ports too

Which is neat

Yeah it’s a relatively new offering. When I was googling around I had to specifically search for eic endpoint rather than just instance connect

Seems like it came out in 2023?

Yup, back then (well now too) have a case where backend is lambda and there is a rds aurora database, and needed to proxy to database. Back then it would have meant running database onbport 22 :P, but would not need that any longer

Kind of funny they don't advertise instance connect endpoint that much

Thanks a lot. I think instance connect endpoint will be great. My understanding is I will simply listen for nrepl connections on my app on my ec2 instance on port abcd inside the private subnet, open a tunnel between localhost:defg and app:abcd and connect emacs to nrepl on localhost:defg. I don't see a need for ssh but maybe I am missing that.