As quick example, lets say I'd like the callback function to be a variable defined in a let

I imagine you could combine something like https://github.com/IGJoshua/farolero with a custom restart handler

Hi, @U064X3EF3 — hope you're doing well! I had listened to @U5NCUG8NR on defn podcast, and had cursorily studied Farolero, but didn't have enough expertise to asses to what extent it could support this type of working style. So this is super helpful! If I hear you correctly, the exception handlers could be potentially modified to support a "restart the read / eval" loop from the specified point... Joshua, is there a better channel to have this discussion — otherwise, maybe we could continue the discussion in this thread? PS: Your episode on defn was fabulous and mind expanding on many fronts. I had no idea such human created languages existed outside of Esperanto, etc.!

Well we do have #farolero. There's a good portion of this style of development that can happen when using farolero, but unfortunately without getting into rewriting bytecode at runtime or the other sorts of magic that IntelliJ and others do for debugging Java code, or without e.g. a fork of Clojure that's built with something like this from the ground up, you can't get farolero quite to the level of these other languages. That is, this style of debugging is well-supported. You run your code, an error hits, program execution halts, you have a full repl, fix the problem, continue as if nothing went wrong. However, exceptions are a sticky business and I can't realistically catch them at their source like I would need to when providing this sort of experience, so libraries (like the Clojure compiler with exceptions on unresolved vars) that already throw exceptions can only be wrapped in a way to provide partial explicit support for this sort of development. That said, I do have underway some experiments like nrepl integration and more that will hopefully get Clojure as close to the likes of Common Lisp and Smalltalk as possible.

with Java agents, you can hot replace loaded bytecode

I'm not saying that any of that path is easy of course :)

Ah, that's good to know! I need to look into that at some point, I might consider making an extension library for farolero that is able to hijack some of the errors at the bottom of Clojure. Definitely don't want to that to be a part of the main library.

@U5NCUG8NR This is so cool — I'm going to try farolero for a bunch of scenarios, including exceptions. I've never used the Java debugger before in Cursive, but I'll give that a try, too. (I mentioned that I had watched @cfleming's earliest video on Cursive which highlighted the debugger, but this was so long ago, I didn't even have a basis to understand what he was showing.) Alas, I'm during a peak busyness period right now, so it may be a couple of weeks. Joshua: maybe after 4 weeks, could we perhaps get on a call? I'd love to better understand the problems that seem important to you, and see if there's any areas that I could help with. (Compilers were my area of study in graduate school, and some of my worst memories involved using hardware debuggers on Win 3.1 to implement multicast network protocols, with no memory protection, etc.) After having seen what is possible in Smalltalk, suddenly this area seems so ripe for resurrecting in the Clojure context. Thx!

@U064X3EF3 Super interesting. Any chance you could send me a URL on hot reloading code for Java agents? I'm not even sure I'm googling for the right thing... :)

I'm always happy to talk with other Clojurians about just about anything. I'm definitely willing to schedule a call at some point. Just let me know when you'll be more free so we can schedule something then.

It's the Java Instrumentation API - you use the Java Attach API to do dynamic loading with a ClassFileTransfomer

https://docs.oracle.com/en/java/javase/17/docs/api/java.instrument/java/lang/instrument/Instrumentation.html

Thanks, Alex — just read this: https://technology.amis.nl/software-development/java/java-agent-rewrite-java-code-at-runtime-using-javassist/ The mind reels at the possibilities. 😀. Thanks for sending this — literally had no idea this type of things were possible in the JVM. (And starting to realize why and how so many PhD dissertations were done on/around/on top of the JVM.) (Until now, I was thinking that some of the topics above could only be done in an interpreted environment, such as interpreted Java on TruffleVM [Espresso]. But I think you're suggesting that things like this make the JVM might be sufficiently dynamic.)

(Colin: jaw is on the floor, watching in your video how easy it is to start a "debug REPL." I'm going to try that tonight.)

Yes, if you’re using the debugger, I don’t think you need agents, you can just reload a class using JDPI. I’ve been meaning to improve Cursive’s debugger support for a while now, it has some annoying bugs that have crept in as the IntelliJ platform has improved, and there’s a lot more good stuff that could be done (e.g. a debug REPL for when the process is stopped at a breakpoint, breaking when vars are called, etc). But breaking on an exception is by far my most-used feature, and AFAIK you can’t really replicate that in any good way except with the debugger.

I’d also love to integrate with the stream debugger for seqs and transducers: https://www.jetbrains.com/help/idea/analyze-java-stream-operations.html

Right, if I want to integrate something like this into farolero I can't leverage intellij unfortunately, as it has to be editor-agnostic, and ideally even work over terminal-based repl with no additional dependencies

Yes, and it’s tricky since the JVM has to be started in debug mode, and you then need an external debugging process to control that.

If it’s something that is designed to work from e.g. a REPL client that’s probably ok, but if farolero is something that just works in any Clojure process, then that’s much harder.

@U6VPZS1EK I’m jealous though, Glamorous Toolkit is really a fantastic system. I’ve been meaning to sit down and learn it well enough to try to do something with it, but since that means learning a new language, ecosystem, editor, and way of working, the bar to entry is high enough that I’ve never actually done it.

I’ve been watching it from a distance for a while now though.

I saw a demo of someone working with Pharo at Curry On, it was pretty amazing.

@cfleming I finally re-watched your video from 2015 on Debugging in IntelliJ/Cursive, and my head literally exploded numerous times. When you showed how you can “drop frame,” which basically pops the stack, and program state is now from before you called the function. And seeing how all local variables have values rendered in the editor. 🤯 I’m blown away by this black magic / sorcery!! This is incredible!! (Colin, this was the video that made me decide in 2016 to use Cursive, which has been utterly wonderful. In hindsight, I see only grokked about 5% of what you were presenting. This is mind-expanding to see what is possible.) https://youtu.be/ql77RwhcCK0

@U6VPZS1EK Yeah, I think that frequently you need some decent amount of time feeling the pain before the solution makes sense! The same is true of Clojure itself I think - I’ve had the experience of trying to explain the benefits of immutability to someone who can program but it’s not their main thing (my stepfather, who’s a mathematician and has programmed for ages, but always hacking stuff together in the service of something else). He really was skeptical, until he tried to program some christmas lights on a RPi using multithreaded Python - after a couple of months he at least appreciated what I was talking about 🙂

I’m sitting down to try a debug REPL for the first time ever — I’m thinking I might get almost everything I hoped for just by using the IntelliJ debugger. Half-joking: it reminds me of this video, where this woman discovers a way to do something that has involved so much toil, and you can see the anguish on her face, part joy, and even angry/resentment, realizing how much time she’s wasted. It’s priceless. In my case, it’s all joy and delight — because there is no flipping way I could have used a debugger effectively until now. 🙂 https://twitter.com/cabel/status/1296213351479930880

I should warn you, there are some frustrating bugs that have worked their way in over time, and I think they’re due to changes in the platform. I’ve been meaning to fix them for ages, and you might have just prompted me to do it!

PS: I did almost cry when you described that you’re working inside of IntelliJ, “which is millions of lines of Java code”, and your characterization of it being at times bewildering, and that was even before it was rearchitected to be a platform for other languages/IDEs, resulting in some crazy situations. But it’s amazing that Cursive can leverage all of that R&D effort. I have a very clear notion of what I want to get out of the experience: a way to deal with exceptions besides looking at stack traces and restarting the program — I’m excited by what I see!

Yes, things have changed a lot, and JetBrains (to their credit) have done a huge amount to make plugin development easier compared to how it used to be. And to think - people developed plugins even before IntelliJ was open source! The mind truly boggles.

Going back to the debugger, personally I think that even if nothing else at all works, just being able to break on exceptions is worth the price of entry.

(the rest is nice too, of course)

There is still a huge amount of stuff in the JVM debugger that I’m sure I could leverage, if only I knew it was there, or how it worked. I have plenty of Julia moments still to come, I’m sure!

Holy crap — I feel a little like that woman in Photoshop video. It only takes one menu click to start a Debug REPL?!? The REPL just… started…. Holy crap. Stand by — I’ve gotta see what happens when an exception is thrown…. OMG…

OMG. Ability to navigate the call stack, and then see the exact line of code where the exception was thrown. I’m actually tearing up… Thank you, Colin!!!!

Seriously, is there a reason not to run Debug REPL mode all the time?!?

I would imagine it has some pretty significant performance implications

since the JIT can't work anywhere near as hard, if it works at all. I don't know how IntelliJ does it, but it's entirely possible it's falling back on bytecode interpretation all the time in debug mode.

It’s actually not too bad, and depends on what you’re doing. When I’m developing Cursive, I pretty much always run it in debug mode.

That's good to know

No, it’s not bytecode interpretation. This is a built-in JVM feature, not something that IntelliJ implements.

What I do find is that sometimes the JVM can get into a funky state, that happens from time to time, so I do end up restarting more often when debugging.

Ah, I see

Sorry, I mixed up my acronyms there, JDPA is the platform bit, and JDI is the interface you would almost certainly use to it: https://www.baeldung.com/java-debug-interface

But the JVM is actually very efficient at debugging. I wouldn’t want to run my prod systems in debug mode, but for development it’s a no-brainer IMO.

Holy cow. I’m seeing the local vars in the editor. Seeing if I can change that b to (inc b), and rerun that division operation…

Oh, no… I can’t do that, because there’s no running JVM to recompile the JVM bytecode, when I reload the namespace, is that right, @cfleming?

I.e., I can’t just reload this namespace, “drop frame”, and resume program, correct?

Well, the JVM is paused, correct. That’s what I mean when I talk about a debug REPL, it would be a REPL that would evaluate code over the debug interface rather than the REPL interface, since the REPL isn’t currently listening.

There’s some UI trickiness to work out there, since a REPL is a single conceptual context, and the debugger is really lots of them. What I mean by that is that you can evaluate expressions at any level in the stack frame, and in each one your locals etc are different.

So I’m not 100% sure how to expose that to the user. You might want to be able to evaluate REPL commands ten levels down in your stacktrace to see what was going on at that time.

Sorry — lots of neurons firing right now. What you’re describing would be amazing — like, brainmelting amazing, right? I feel like I need to think about what would be most useful before I even opine on this — can I share with you some excerpts of the Glamorous Toolkit sessions, to frame some of the aha moments that even led to asking about the debugger? I think it may trigger some thoughts on what people are talking about when they gush about what’s been lost in the post Smalltalk era. (I’ve been working on a blog post to try to articulate and share these aha moments behind GT, membrane, cljfx, and ultimately Fulcro and RAD. I think it’s be around 8K words.)

In the meantime, thanks for helping me understand the power of the debugger — I know this will change the way I problem solve issues. As you said, “hammocks are good,” but in many situations, “debuggers are better,” because you can actually see what you’re doing!

Colin, I’d recommend watching at the 20m mark in this video — you can see how quickly Tudor is able to get tweet images displayed inside GT, something difficult (impossible?) to do in typical IDE — which is why I use Portal so rabidly now, as well as Clerk. I’ll try to find the video where they keep yelling at me to stop closing the Pharo debugger! 🙂

https://twitter.com/RealGeneKim/status/1469389359249956869

Yes, I’m always interested in understanding how other systems work! I’ll take a look at that in a minute, BRB

One thing I have considered is whether to just let people open a REPL on their Cursive instance (which is just a JVM instance, after all) and go wild. The risk is that you bork your IDE terribly and have to restart, but it opens up more possibilities in terms of using the same UI for the editor and the app, which both Smalltalk and Common Lisp systems tend to do. Unfortunately it also does mean that you can’t debug, though.

(PS: Armed with my newfound ability to write UIs quickly using Fulcro, I have all Zoom recordings in XTDB… I’m searching for my Zoom sessions w/Eric and Tudor, so I can upload them to YouTube. Finding these recordings are such a pain w/o something like it!)

Surprisingly I feel like I don’t ever use debug mode in Cursive because of how tight the feedback loop is, i.e. an exception thrown can be handled by just a simple workspace reload away. To the point where I always forget to use debug mode when working with Java.

Being able to code handler on the fly sounds really amazing but I struggle to see how useful it is, or may be that’s just because I haven’t seen it in action yet

@U6VPZS1EK I think I’m going to have to go through those videos when I have more time, and probably try to replicate them myself in GT. Just watching them without the background of what is going on, I feel like I miss a lot without understanding the basics, and those videos are pretty long to catch up 🙂

Unfortunately their getting started doc is pretty much “Here’s how you open your IDE, have at it!” which isn’t really enough to get started, as a curious bystander.

No worries at all — I’ll send you 2 or 3 more selected excerpts which I think you’ll find interesting. Sorry they’re not edited down, they don’t have much background. I do put in a tweet a summary. I’ll send those your way, as well. (And you’ll definitely see the rough edges around GT — it was tough sledding at times, esp for Eric, who seemed to step on every landmine there was. 🙂

Only watch if it gives you joy. 🙂 I will try to narrow down 10m that I think would be of specific interest w.r.t our discussion above! (In next couple of days.)

Thanks, I’d definitely be interested to see them. Are you aware of any good getting-started resources, apart from getting personally tutored by the creator? 🙂

(that looks like a great way to do it BTW)

As food for thought, making the feedback loop more granular can, in some ways, be better than an imperative debugger. Especially with a functional language like clojure, you get all the incremental feedback, but without having to manually step because functions are pure and time isn't an input. I'm not sure how to best integrate side effects, but it's unclear if the "debugger approach" handles including side effects gracefully either.

What sorcery are you potentially unleashing unto the world, @U7RJTCH6J ?!? That's utterly wild. I'm not even sure what I'm looking at, but it's amazing!! 🤯🤯 Written in membrane? And some sort of container for ns forms, with live evaluation? Tell us more!!!

@U7RJTCH6J Are you going to ever tell us what magic you’re working on, that you’re showing us above? 🙂

I've been reluctant to share because I haven't thought of a cool name for the project. I'm working on a pure clojure, coding environment. Some of the goals are: • strong support for direct manipulation • create programs incrementally and reduce burden on your working memory • run everything local and in-process • leverage the full capabilities of jvm clojure and ecosystem • make the leap from text editing to code editing It's still a work in progress. I gave a short demo at one of the visual tools meetings: https://youtu.be/tse4gKFj45c?t=2585

@U7RJTCH6J Holy cow. That is freaking incredible!! I feel like you are definitely onto something amazing. This moment blew my mind, when you talked about feeling like this is one level closer to your code, than even being in the REPL. “Imagine not being able to work in a REPL. That's how I'm starting to feel about working in this environment, if i has to go back to a regular REPL.” And to do it all within your membrane environment, inside the JVM. No browser. It's really 🤯🔥