Clojurians Log v2

Hi Sean, quick question on your setup if you wouldn’t mind. I see that you occasionally jack in to production environments, which could be super useful in certain circumstances. I’m set up to do that as well for our prod env, but I never do it because I’m terrified that through the misapplication of muscle memory I’ll screw something up. For example, if I was to connect to a prod system using a socket repl which was loaded in my IDE (I use cursive), I could unintentionally reload namespaces into prod by triggering my usual reload key sequence - just because I do it all the time when developing. Are there techniques or conventions you use to minimise such risks?

@denis.mccarthy.kerry For starters, I have no "reload" workflow -- and I generally advocate strongly against using such things 🙂 But, yes, connecting a REPL to production carries the risk of blowing up your production system in all sorts of ways -- just as having a live connection to your production database can be perilous. I have a particular namespace that I work in when I connect to production. It contains a little bit of setup code and, specifically, sets up a readonly datasource that I can safely use for any interactions that access the database. Our production JARs are AOT-compiled with direct linking which also limits the "damage" you can do by (accidentally) redefining things since the calls sites are not automatically updated and you have to be explicit about those redefinitions up the call tree. Our deployment pipeline is sufficiently fast and easy to work with that making a local fix and having it go through CI and on to production isn't a big deal.

Thanks Sean, very useful. When you say you don’t have a reload workflow, what do you do when you are actively developing? As I write functions, I typically test them in a rich comment block, and once I’m happy with them I will load the namespace into my REPL (using an IntelliJ shortcut) and give the function a quick once-over in situ. Your recommendation against reloading namespaces would go against this if I understand you correctly. If I do understand correctly, how do you manage this aspect of the workflow? Feel free to tell me to rtfm if you’ve blogged about this piece elsewhere! On a related point, I’m curious about whether you go straight to production with fixes or do you have a manual step before code hits users? We have a deployment flow that could support either, and I haven’t made up my mind about the wisest policy here.

I eval code changes into a running REPL. Period. No "reloading". Nothing destructive. As for fixes going to production, when we commit (or merge) a change to our main branch, CI runs incremental tests (via Polylith) and builds JARs for any changed services/processes and automatically deploys those to staging, and from there we select artifacts to automatically deploy to our production cluster.

I put up https://github.com/seancorfield/vscode-calva-setup/pull/3 which should enable the dual window setup when connecting with the remote-repl

@djblue I'm not sure that fits my usage. I have Portal running as a headless HTTP server in the remote processes, when they boot up. But I'll experiment with the setup at some point, probably after Conj.

Running headless (starting the server on a fixed port) shouldn't conflict with this approach. This should enable calling (portal.api/open {:launcher :vs-code}) on the remote repl and getting it to open a vscode tab locally

Good to know. Thanks. And I need the very latest Portal, for this to work, right?

No, this should work with any version since the port mapping is being done manually via the ssh command 👌

Ah, OK. And why -R for the extension port?

(the others are all -L)

That will proxy the extension port from your local machine to the remote server so the portal runtime code can communicate with the extension.

This way all the code can pretend it's on the same box

Right, but the other ports are proxied via -L -- I was asking why the extension port uses -R instead.

I thought L was from remote to local and R was local to remote

Oh... direction... I had always thought of the forwarding as bidirectional...

It might be, not sure 🤔

I found this info here https://www.ssh.com/academy/ssh/tunneling-example#remote-forwarding

I'll save any other Qs until I've actually tried this! 🙂

OK, I had a bit of "down" time over lunch to test this: works great! Thank you! My old setup (which you hadn't seen, since it isn't in that repo), assumed I would add-tap/remove-tap so there was no Portal attached to the QA/production process unless I was actively debugging... It's not a big deal since we rarely leave tap> calls in deployed code but it is something I might have to think about since those tap> calls would send data to Portal, which would continue to conj them into the results array... I can of course manage that by adding code to my custom submit to limit history... Overall, tho', it is so slick to be able to do ctrl+alt+b q and then ctrl+alt+space p and have it start the tunnel, connect to the remote nREPL, then fire up Portal in VS Code and have it connect and run "just like dev"!

q for QA, I also have p for production 🙂

That's awesome! Thanks for taking the time to test it out thanks3

I don't think I actually need to start the Portal HTTP server in the remote server since the code I eval launches it -- but that does beg a question of if/when that Portal server would shutdown? I guess I'm also thinking, when does .portal/vs-code.edn get a different port number?

I think calling portal.api/close with no args will close all UI instances and shutdown the server. The edn file should mirror the info for your local vscode instance, so anytime you restart vscode it would need to change, but that should be automatically happening currently.

Having a clean up function should be as easy as calling close and removing all tap handlers 🤔

Hmm, I eval'd (p/close) and it seemed to shut things down -- but now I can't get it to start back up...

My guess would be it might have started back up on the wrong port

I get the Portal window open in VS Code but it has no content. I did not restart VS Code.

I think you might need to specify the :port in portal.api/open or portal.api/start

Ah, ok... and that's the one from the vs-code.edn file?

I thought it was supposed to pick that up automatically?

The port for the server on the jvm, I hate how many ports are involved here lol

You will also need to re-shutdown the jvm runtime server if it's bound to wrong port with portal.api/close

(portal.api/close) (portal.api/open {:port portal-port})

So the port I was starting the Portal server on was 7311, and it needs that port and it needs the one from vs-code.edn essentially?

Yes. However, the vs-code.edn one you shouldn't have to worry about

Now, I call open twice in my (client-initiated) Portal code -- once for logging/middleware output and once for plain tap> -- so I called open three times in total...

The startup open specifies port 7311 but the subsequent open calls do not specify a port -- so am I correct that once it starts on port 7311, the subsequent open calls reuse that?

Yup, that's correct

If the server has already started, if just goes ahead and uses. Otherwise when starting the server it will use the provided port or a random one.

It's one of those things where it mostly works when everything is local and you don't care about port mapping... but when you care it sucks 😆

OK, I recovered it! I did (p/start 7311) and that (re)started the http-kit server on the correct port.

So calling (p/close) is pretty dangerous...

Do you mean (p/start {:port 7311})? I would say delicate, but easily fixable

Yes, sorry, that's what I meant.

I think perhaps start / open could indicate / print something if the server is already started and on another port to help aid in debugging

So I really only want to close the new interactive instances I open'd in my VS Code session for debugging things... luckily, I do save both of those in a dev namespace...

(defn close [portal]
  (if (= portal :all)
    (c/request {:op :portal.rpc/close})
    (c/request (:session-id portal) {:op :portal.rpc/close}))
  (when (or (= portal :all) (empty? @c/connections))
    (future
      (some-> server deref :http-server http/server-stop!)
      (reset! server nil)))
  (swap! rt/sessions dissoc (:session-id portal))
  (swap! rt/sessions select-keys (keys @c/connections)))

I think portal still might kill the server if you have no clients when you call close

If you are cool keeping the server running, killing the tabs in vscode is probably the way to go

Yeah, I think that's fine.

However, maybe adding a explicit portal.api/stop would be better here 🤔 then I would remove the auto shutdown, feels more confusing than its worth.

It's also more symmetric, start/stop open/close

I restarted VS Code, reconnected to QA, and started Portal again with my regular command sequence and it worked, using the different port from vs-code.edn

Where is that c/connections var? I want to see how many sessions it thinks it has now...

Found it...

Confirmed that when I close each VS Code window, that connection goes away in the connections atom. When they're all closed, the atom is empty -- but the server stays running at that point -- and I can shutdown VS Code, restart it, and reconnect just fine... so it's all good at this point.

My dev setup for daily work -- my logging/middleware output window and my regular tap> window have different themes 🙂

And now my remote setup is near identical 🙂

I wonder if I should add vscode theme variants, so it can match but be different 🤔

I think that would be overkill 🙂

True, but it would be relatively easy and might be helpful when the default mapping doesn't work well for a given theme

I've updated my blog post to reflect the Pull Request from @djblue and the much-improved workflow that gives me -- so slick to copy .portal/vs-code.edn from the "client" machine running VS Code to the remote machine running the REPL! Thank you!