Fork me on GitHub

watching Sean Grove's talk on Dato -- -- an RFC framework he and Daniel Woelfel were working on. The idea of using datomic straight from the client is so darn cool -- like a firebase on steroids


What are the hurdles to building a PDB-style debugger for Clojure?


what’s the usecase? I find debugging needed in mutable environments where it’s important to stop the world and see current state of different pieces of environment. With clojure’s immutability by default, I can just tap> /`prn` interesting stuff on interesting points in time


tap> / prn is a poor man's debugger. @U15RYEQPJ have you tried cider's debugger?


@U47G49KHQ I really like being able to just pause and step through code, especially when it's written by others and is cryptic. And preferably without any external tooling


@U0P1MGUSX yeah I've used both Cider's and Cursive's debuggers


The question isn't as much "how do I debug Clojure?" as "how would one go about building a pure Clojure stepping debugger (and why hasn't anybody built one?)"


i can relate to wanting to pause and step through code. the closest things i've found are debug-repl by georgejahad: and a more recent attempt:


neat, thanks for those links @UG1C3AD5Z


debug-repl looks a lot like what I’d want


np -- if you find anything better or related, i'd love to know 🙂


yeah, I think I might try to build something 🙂


the main problem with debug-repl afaict is that it doesn’t allow stepping


I wonder how one would go about adding something like that…


you can do stepping with cider's debugger


yes, i have used that and cursive's, though i prefer the latter as it can dip into java stuff too and i find the ui for debugging nicer -- i also don't use cider.


in there is some mentioning of step-by-step debugging, fwiw


yeah I saw that — the problem afaict is that it only permits step-by-step debugging within the dbg-all* macro


being able to step out of the frame/scope where the debugger is called would be preferable


sure -- i thought about it a bit before, but unfortunately, didn't come up with anything great.


i think the best i came up with was instrumenting a lot 🙂


i think there have also been some attempts using low level jvm things too.


ah, i knew there was another one (pure clojure iirc): -- step is a todo for this one


cool, those look great


unfortunately there’s no discussion that I can find on how they planned on implementing (step)


perhaps it's possible one or the other of them might respond to queries?


i suppose it's possible there's something relevant regarding stepping in cider's debugging functionality. i don't really know how that one works.


yeah I suppose


any idea how one could find the relevant source code in cider?


my recollection is that a fair bit of the work was by:


okay yeah, doing a blame on that code seems to suggest that too


i think i remember seeing some bits discussed in some of the pull requests for adding the debugging stuff. taking a look now.


i'm not sure how much changed since this: but it has some explanation of some of how things worked at one point


it seems like the heavy lifting is happening in some other file (most of the code in cider-debug.el seems to end up calling


ah, there's some explanation in that file


lol, looks like it's the "instrument all the things" approach


ok "of interest"


it could be that Ivana or razum2um might have some ideas


that are different from instrumenting everything


i'm not sure how that works, but when i looked into java debugging stuff before, i think what i found is that jvm bytecode is instrumented at runtime.


yeah that sounds about right


clojure produces jvm byte code, so i suppose it might be possible to change it to emit different byte code


I imagine Cursive’s approach is probably something along those lines?


what i recall about it is that it uses "official" interfaces to jvm debugging -- but i don't recall clearly.


or may be there was some intellij-way to do it.


hmm yeah, I could imagine that the Cursive debugger is built on top of InteilliJ’s Java one


search for PDA and JDI


hah yeah I just found that same talk :)


Cursive is closed source right?


yes, i believe so


ah nice, there's an associated repository too:


maybe I could just start off writing a Java debugger in Clojure using JDI, and then see if I can tack on Clojure debugging functionality on top of that


perhaps so -- that other link from before: -- may be relevant


ah, one thing -- that project uses jdk8 and tools.jar.


so if you want a more recent jdk, you may need to do something different for the tools.jar part.


it seems like one limitation with JDI might be that you need two separate JVMs (one for the main process and one for the debugger) — so dropping into a debugger from the main process (like PDB does) might be hard


I don't know if relevant for your use case. For debugging, I use the combination of scope capture and Cursive/Intellij debugger. Although you can set breakpoints and rebind/inspect vars using just scope capture.


neat, thanks, scope-capture looks really cool


a huge gotcha with these sorts of debugging: you need to turn off locals clearing, which breaks some code badly


if you leave locals clearing on, the things you want to access are garbage collected, if you turn it off, functions that should be fine can blow up the heap


@noisesmith would that apply to the JDI approach as well? Also, how is locals clearing controlled?


right, this effects anything which directly controls real execution, there's a startup flag which controls locals clearing


if you use clojure as intended, storing immutable data and using it later is less complex than single stepping a vm, and the real risks of side effects (IO, system calls in general, etc.) aren't handled properly in a step debugging environment any better than they are with manually recreated calls


got it, thanks


@noisesmith thanks for the note about locals clearing and debugging. i see it appears to be mentioned here:


I shared that same link


ah you did! sorry i missed it 😅

Mikael Andersson01:07:45

If you are interested in poking around with JDI, there is also cdt of which at least a fork (indiejames) was updated in 2017. and then there are the JVMTI low level interfaces, and related on debuggers on the JVM, one must of course mention (apache licensed) as a reference of advanced usage, and for connecting to a running VM (GPL)

Mikael Andersson01:07:59

If locals clearing is implemented to keep stack size down and if it's the stack size that blows up without it, -Xss jvm option can be used to give you more stack headroom while playing around in a debugger/repl/what-ever-your-taste-is.


no, it's about heap allocation, and blowing up the max heap


(let [els (map produce-large-result (range))]
  (run! f els))
with locals clearing that can run indefinitely, without it fills up the heap


i found this rh post about fine-grained locals clearing:!msg/clojure/FLrtjyYJdRU/1gzChYsmTpsJ iiuc, the code from noisesmith demonstrates the following (slightly edited) quote from the post: > Unfortunately, on the JVM, [snipped], in many cases, [snipped] - the local is considered a live reference and is thus not GCed. This yields the infamous 'holding onto the head' problem, and subsequent Out Of Memory errors on large data sets.

Mikael Andersson01:07:24

Yes locals are always live references, that's how the VM determines liveness. Depends on the GC, but essentially it walks the stack(s) and marks all objects it can see, and then purges the rest. I've only just dove into learning a bit more of the internals, mostly because I don't like being surprised by the runtime. However, it kind of sounds like the locals clearing is closely related to lazy sequences. I can't really figure out any other way to have collection that grows boundlessly in size in a deep stack frame running a language using almost exclusively immutable data.

Mikael Andersson01:07:52

And since lazy seq's are only pretending to be immutable?


locals are not always live - clojure clears them if it can prove they don't escape the closure


the vm doesn't determine this, the clojure compiler does (via the way it structures the byte code implementing the closure)


the GC does not walk the stack, it walks the heap

Mikael Andersson21:07:06

Maybe I was unclear, locals in this context was referring to values on the jvm stack (primitives, and references), which referenced values are always considered live. Which is why clojure needs to null them in the case they reference a lazy sequence or similar mutable structures which otherwise will behave as ever expanding singly linked lists. You are right of course that the GC doesn't only walk values on the stack, but eg. a mark and sweep walks the live graph starting from the stacks, which together with some additional references (static variables, JNI references, and threads) serve as the "GC root"

Mikael Andersson21:07:00

I mean, you know that, I guess we're just talking across each other a bit.

Mikael Andersson21:07:33

By the way, I think I forgot earlier, but T thanks for the pointers about the local clearing, much appreciated! The clojure compiler still clears quite a few locals even with local clearing disabled, but now when I know what to look for I can easily make a tweaked version if need be.


Cool, thanks for all the info and links! I played around with JDI a bit, and I have to say it seems comparatively straightforward. I’ll try to use it to build something proper when I get some free time :)


@U15RYEQPJ if you get around to it, please let me know 🙂

Joshua Suskalo13:07:38

@hiredman I did mention that I am using newlines to denote the end of a form, and I've also tried chaining multiple newlines at the end to see if that had any impact.

Joshua Suskalo13:07:20

Yet despite that, it still never produces a value it seems


anyone familiar with a websocket client library compatible with jdk 11? 😕


What are idiomatic ways/libs to sort a vector without using sort-by?


What are you trying you trying to solve?


sort and sort-by are the idiomatic options, they use the underlying vm for the algorithm


Trying to sort a vector of maps. No, sort-by does not work always, sometimes it just breaks with a nasty exception


the exception means you need a better comparator


Giving me stuff like "Can't compare <print of my map here> with 1"


any other sort mechanism would require the same thing


And there most certainly is no 1 anywhere in my vector, nor does my comparator ever return a number


I'm confused, a comparator is by definition a function taking two items and returning a number


"case isn't broken" (or is it switch?)


is this a reference I'm missing?


It's a reference to people saying "the compiler is broken" when it's nearly always their own code.


oh, right, of course - I hadn't heard it put this way


solution: write a compiler


"It is rare to find a bug in the OS or the compiler, or even a third-party product or library. The bug is most likely in the application." -- that's where I remember "compiler" from I guess 🙂

🚮 3
💾 3

it's similar to the more general "if you hear hooves, assume a horse not a zebra"


check the common case if all else is equal, in software the common case is an error in the most recently written code


(of course, if you actually are a compiler writer -- as I used to be in the '80s/'90s 🙂 -- then the bug could well be in your compiler!)


right - but even then, it's more likely to be in newer code than older code, unreleased vs. released code, etc. - as a good first heuristic at least

💯 3

The story in the book is nice. Kinda. Engineer was convinced case was broken. Actually they didn't understand it. Not dissimilar from in clojure really.


I remember hearing it as "select isn't broken" a reference to SQL.


Not to be the “well akshuwally” guy, but the anecdote in the book had to do with the POSIX system call “select” (I know because I have my first edition print of the book that I purchased in 1997 right here in front of me 😉)


That is an awesome bit of info 🙂 Appreciate you sharing.


for example, if you require totally heterogenous data, you could make a comparator that says a number is always less than a map


It is very homogenous data, a vector where every map has the same keys


could you post a code sample?


(defn- vendor-comparator [a b]
  (cond (and (empty? (:vendor/approved a)) (seq (:vendor/approved b)))
        (and (seq (:vendor/approved a)) (empty? (:vendor/approved b)))
        (compare (:vendor/name a) (:vendor/name b))))
This is my comparator


that's a bad comparator, it needs to return a number for all branches


Doc says it has to return whatever input is "bigger"


So that is what I did


where do you see that doc?


Don't really remember. But returning numbers simplifies it


clojure's IFn is kind enough to implement Comparator for you, but you still need to return the right type if using your function that way


"Comparator. Returns a negative number, zero, or a positive number when x is logically 'less than', 'equal to', or 'greater than' y."


the only place "bigger" as a string appears in the clojure git history is in a comment for the asm library that is used to generate jvm bytecode

😂 3

And "If no comparator is supplied, uses compare. comparator must implement java.util.Comparator."


Weird enough that it actually worked at the beginning


Oh, I didn't find that guide, thanks


Quick note: Launched "Journal Together" -- a lil app I was working on to play with clojure again on HN (It's on the Show page). Noted the experience of using clojure in there -- tl:dr repl + jvm == a loot of fun. Big shoutouts to the community here, got a lot of help in pushing this out -- esp @seancorfield and @noisesmith

👍 15