Fork me on GitHub
Victor Mataré14:07:05

Hey... I'm having some issues with remote debugging over a kubernetes port-forwarding. It works perfectly when I connect directly to a local JDWP socket (i.e. on localhost), but when I run the exact same jar in a container and connect to the JDWP port through a kubernetes port forwarding, I can't set breakpoints on clojure code or step through it. I know that the JDWP does work in principle because I can interrupt execution (with the pause button), which will take me to a decompiled Java frame, from where I can step through the Java code, but can't seem to get to the clojure level in any way. When I try to set a breakpoint, the red dot appears and everything seems fine, only the debugger will never stop there. When monitoring the TCP connection with tcpdump, I can see that setting the breakpoint causes an endless back-and-forth on the JDWP port (thousands upon thousands of packets if I let it go for 10-15 minutes), which doesn't appear to ever terminate. When setting a breakpoint locally (successfully), I see a few (maybe 10-20) packets going back and forth on the connection. Is there some known issue here or does anyone know a better way to debug what's going wrong on the JDWP connection?


Hmm, this is probably a bit specialised, sorry - I really don’t know. Do breakpoints work in Java over the port forwarding?

Victor Mataré10:07:11

Yes, I can debug the underlying Java code completely fine. Breakpoints work, values are displayed, I can see all threads, etc. I can even set breakpoints in the Java implementation of Clojure functions and then step out of the Java level and into our Clojure application code from there. Amazingly, I can then step through our Clojure code, see its locals and it all looks fine. It appears that I just can't set any breakpoints. I even put a breakpoint on the exact line the program is stopped at, and it just triggers this endless network traffic. The little check mark also never appears btw. Is there some additional logging on the Cursive debugger implementation that I could activate? Or any other way to figure out what's going on there?

Victor Mataré11:07:47

Oh wow, I've just let it run for almost half an hour and it seems that the breakpoint is finally set and the debugger is breaking there. Now I wanted to disable the breakpoint (not remove, just disable), and again I'm seeing huge amounts of network traffic that will probably again take ages to complete. Seems to me like there's a lot of unnecessary overhead going on there. In particular, why could there be so much more traffic than when I'm debugging a locally started JAR? I really need some way to find out what information the Cursive debugger is exchanging (or trying to exchange) with the debug server here...


So, I can’t fully explain this, but what I suspect is that you’re being bitten by the fact that Clojure bytecode is poorly designed for debugging using JDI. My memory of this is that to set a breakpoint, you have to specify the class that the breakpoint is in. In Java this is trivial, but in Clojure it’s impossible since class names tend to have autogenerated numbers in them. To work around this, I seem to recall that I could use a simple wildcard, but that means that I basically had to do namespace$* for the classname, which can match a lot of classes. I suspect this is why debugging Clojure code is so slow.


I’ve been meaning to reach out to JetBrains to see if they could help me fix this - I’ll try to get around to this since I would like to know definitively if there’s a better way. I can’t be the only one with this problem.

Victor Mataré13:07:26

Right, I watched your talk about the Cursive debugger on youtube and I remember you briefly mentioning this issue. So basically what I'm seeing here is Cursive querying all (generated) class names that exist in our namespace(s)? Several things come to mind: • You query everything every time I manipulate a breakpoint because code might have changed? If this true, can't we have a switch to disable that if we know that we won't be live-changing any code? • You said there's no way to predict the generated class names. Maybe there is a way to query them locally? At the very least there could be a way for Cursive to simply look at the compiled class files and find all the required information in there...?


So, maybe? But I’m not so familiar with the debugger infrastructure, and it’s very complex these days. I’ll reach out to JetBrains and see if I can get some help.