This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-07-21
Channels
- # aws (14)
- # babashka (35)
- # beginners (163)
- # boot (2)
- # calva (5)
- # cider (30)
- # clojure (143)
- # clojure-colombia (1)
- # clojure-europe (5)
- # clojure-nl (11)
- # clojure-spec (1)
- # clojure-uk (16)
- # clojurescript (71)
- # community-development (2)
- # conjure (1)
- # cursive (6)
- # datomic (30)
- # duct (4)
- # figwheel-main (11)
- # fulcro (28)
- # graalvm (3)
- # graphql (23)
- # hoplon (36)
- # jackdaw (24)
- # kaocha (16)
- # lambdaisland (1)
- # leiningen (4)
- # luminus (3)
- # meander (4)
- # observability (1)
- # off-topic (10)
- # pathom (5)
- # re-frame (27)
- # reitit (7)
- # remote-jobs (1)
- # sci (17)
- # shadow-cljs (22)
- # spacemacs (14)
- # sql (61)
- # testing (3)
- # tools-deps (27)
- # vim (2)
- # xtdb (18)
- # yada (2)
watching Sean Grove's talk on Dato -- https://www.youtube.com/watch?v=BiplJ4AFwCc -- an RFC framework he and Daniel Woelfel were working on. The idea of using datomic straight from the client is so darn cool -- like a firebase on steroids
What are the hurdles to building a PDB-style debugger for Clojure?
what’s the usecase? I find debugging needed in mutable environments where it’s important to stop the world and see current state of different pieces of environment. With clojure’s immutability by default, I can just tap>
/`prn` interesting stuff on interesting points in time
tap>
/ prn
is a poor man's debugger. @U15RYEQPJ have you tried cider's debugger?
@U47G49KHQ I really like being able to just pause and step through code, especially when it's written by others and is cryptic. And preferably without any external tooling
@U0P1MGUSX yeah I've used both Cider's and Cursive's debuggers
The question isn't as much "how do I debug Clojure?" as "how would one go about building a pure Clojure stepping debugger (and why hasn't anybody built one?)"
i can relate to wanting to pause and step through code. the closest things i've found are debug-repl by georgejahad: https://github.com/GeorgeJahad/debug-repl and a more recent attempt: https://github.com/Ivana-/bb-debug
neat, thanks for those links @UG1C3AD5Z
debug-repl looks a lot like what I’d want
yeah, I think I might try to build something 🙂
the main problem with debug-repl afaict is that it doesn’t allow stepping
I wonder how one would go about adding something like that…
yes, i have used that and cursive's, though i prefer the latter as it can dip into java stuff too and i find the ui for debugging nicer -- i also don't use cider.
in https://github.com/Ivana-/bb-debug/blob/master/src/bb_debug/example.clj there is some mentioning of step-by-step debugging, fwiw
yeah I saw that — the problem afaict is that it only permits step-by-step debugging within the dbg-all*
macro
being able to step out of the frame/scope where the debugger is called would be preferable
sure -- i thought about it a bit before, but unfortunately, didn't come up with anything great.
ah, i knew there was another one (pure clojure iirc): https://github.com/razum2um/clj-debugger/ -- step
is a todo for this one
cool, those look great
unfortunately there’s no discussion that I can find on how they planned on implementing (step)
i suppose it's possible there's something relevant regarding stepping in cider's debugging functionality. i don't really know how that one works.
yeah I suppose
any idea how one could find the relevant source code in cider?
my recollection is that a fair bit of the work was by: https://github.com/Malabarba
okay yeah, doing a blame on that code seems to suggest that too
i think i remember seeing some bits discussed in some of the pull requests for adding the debugging stuff. taking a look now.
i'm not sure how much changed since this: https://github.com/clojure-emacs/cider-nrepl/pull/220 but it has some explanation of some of how things worked at one point
it seems like the heavy lifting is happening in some other file (most of the code in cider-debug.el
seems to end up calling https://github.com/clojure-emacs/cider/blob/master/cider-debug.el#L440-L458)
may be here? https://github.com/clojure-emacs/cider-nrepl/blob/master/src/cider/nrepl/middleware/debug.clj
yeah 😕
fwiw https://docs.oracle.com/javase/7/docs/technotes/tools/windows/jdb.html seems to do this but for Java
i'm not sure how that works, but when i looked into java debugging stuff before, i think what i found is that jvm bytecode is instrumented at runtime.
yeah that sounds about right
clojure produces jvm byte code, so i suppose it might be possible to change it to emit different byte code
I imagine Cursive’s approach is probably something along those lines?
what i recall about it is that it uses "official" interfaces to jvm debugging -- but i don't recall clearly.
hmm yeah, I could imagine that the Cursive debugger is built on top of InteilliJ’s Java one
there is a bit on it here i think: https://eventil.com/talks/rneSXd-colin-fleming-debugging-clojure-code-wtih-cursive/transcript/
hah yeah I just found that same talk :)
Cursive is closed source right?
this seems to be a good intro to JDI: https://www.baeldung.com/java-debug-interface
ah nice, there's an associated repository too: https://github.com/eugenp/tutorials/tree/master/java-jdi
sweet
maybe I could just start off writing a Java debugger in Clojure using JDI, and then see if I can tack on Clojure debugging functionality on top of that
perhaps so -- that other link from before: https://github.com/indiejames/clojure-debug-demo -- may be relevant
right
so if you want a more recent jdk, you may need to do something different for the tools.jar part.
yeah okay
it seems like one limitation with JDI might be that you need two separate JVMs (one for the main process and one for the debugger) — so dropping into a debugger from the main process (like PDB does) might be hard
I don't know if relevant for your use case. For debugging, I use the combination of scope capture and Cursive/Intellij debugger. Although you can set breakpoints and rebind/inspect vars using just scope capture. https://github.com/vvvvalvalval/scope-capture
neat, thanks, scope-capture looks really cool
a huge gotcha with these sorts of debugging: you need to turn off locals clearing, which breaks some code badly
if you leave locals clearing on, the things you want to access are garbage collected, if you turn it off, functions that should be fine can blow up the heap
@noisesmith would that apply to the JDI approach as well? Also, how is locals clearing controlled?
right, this effects anything which directly controls real execution, there's a startup flag which controls locals clearing https://clojure.org/reference/compilation#_locals_clearing
if you use clojure as intended, storing immutable data and using it later is less complex than single stepping a vm, and the real risks of side effects (IO, system calls in general, etc.) aren't handled properly in a step debugging environment any better than they are with manually recreated calls
got it, thanks
@noisesmith thanks for the note about locals clearing and debugging. i see it appears to be mentioned here: https://cursive-ide.com/userguide/repl.html#starting-a-debug-repl
I shared that same http://clojure.org link
If you are interested in poking around with JDI, there is also cdt of which at least a fork (indiejames) was updated in 2017. https://github.com/GeorgeJahad/cdt/network and then there are the JVMTI low level interfaces, and related on debuggers on the JVM, one must of course mention https://github.com/JetBrains/intellij-community/tree/master/java/debugger/impl/src/com/intellij/debugger/jdi (apache licensed) as a reference of advanced usage, and https://github.com/JetBrains/jdk-sa-jdwp for connecting to a running VM (GPL)
If locals clearing is implemented to keep stack size down and if it's the stack size that blows up without it, -Xss jvm option can be used to give you more stack headroom while playing around in a debugger/repl/what-ever-your-taste-is.
no, it's about heap allocation, and blowing up the max heap
(let [els (map produce-large-result (range))]
(run! f els))
with locals clearing that can run indefinitely, without it fills up the heapi found this rh post about fine-grained locals clearing: https://groups.google.com/forum/#!msg/clojure/FLrtjyYJdRU/1gzChYsmTpsJ iiuc, the code from noisesmith demonstrates the following (slightly edited) quote from the post: > Unfortunately, on the JVM, [snipped], in many cases, [snipped] - the local is considered a live reference and is thus not GCed. This yields the infamous 'holding onto the head' problem, and subsequent Out Of Memory errors on large data sets.
Yes locals are always live references, that's how the VM determines liveness. Depends on the GC, but essentially it walks the stack(s) and marks all objects it can see, and then purges the rest. I've only just dove into learning a bit more of the internals, mostly because I don't like being surprised by the runtime. However, it kind of sounds like the locals clearing is closely related to lazy sequences. I can't really figure out any other way to have collection that grows boundlessly in size in a deep stack frame running a language using almost exclusively immutable data.
And since lazy seq's are only pretending to be immutable?
locals are not always live - clojure clears them if it can prove they don't escape the closure
the vm doesn't determine this, the clojure compiler does (via the way it structures the byte code implementing the closure)
the GC does not walk the stack, it walks the heap
Maybe I was unclear, locals in this context was referring to values on the jvm stack (primitives, and references), which referenced values are always considered live. Which is why clojure needs to null them in the case they reference a lazy sequence or similar mutable structures which otherwise will behave as ever expanding singly linked lists. You are right of course that the GC doesn't only walk values on the stack, but eg. a mark and sweep walks the live graph starting from the stacks, which together with some additional references (static variables, JNI references, and threads) serve as the "GC root"
I mean, you know that, I guess we're just talking across each other a bit.
right, yes
By the way, I think I forgot earlier, but T thanks for the pointers about the local clearing, much appreciated! The clojure compiler still clears quite a few locals even with local clearing disabled, but now when I know what to look for I can easily make a tweaked version if need be.
Cool, thanks for all the info and links! I played around with JDI a bit, and I have to say it seems comparatively straightforward. I’ll try to use it to build something proper when I get some free time :)
@U15RYEQPJ if you get around to it, please let me know 🙂
@hiredman I did mention that I am using newlines to denote the end of a form, and I've also tried chaining multiple newlines at the end to see if that had any impact.
Yet despite that, it still never produces a value it seems
sort and sort-by are the idiomatic options, they use the underlying vm for the algorithm
Trying to sort a vector of maps. No, sort-by does not work always, sometimes it just breaks with a nasty exception
the exception means you need a better comparator
any other sort mechanism would require the same thing
And there most certainly is no 1 anywhere in my vector, nor does my comparator ever return a number
I'm confused, a comparator is by definition a function taking two items and returning a number
is this a reference I'm missing?
It's a reference to people saying "the compiler is broken" when it's nearly always their own code.
oh, right, of course - I hadn't heard it put this way
"It is rare to find a bug in the OS or the compiler, or even a third-party product or library. The bug is most likely in the application." -- that's where I remember "compiler" from I guess 🙂
it's similar to the more general "if you hear hooves, assume a horse not a zebra"
check the common case if all else is equal, in software the common case is an error in the most recently written code
(of course, if you actually are a compiler writer -- as I used to be in the '80s/'90s 🙂 -- then the bug could well be in your compiler!)
right - but even then, it's more likely to be in newer code than older code, unreleased vs. released code, etc. - as a good first heuristic at least
The story in the book is nice. Kinda. Engineer was convinced case was broken. Actually they didn't understand it. Not dissimilar from in clojure really.
Not to be the “well akshuwally” guy, but the anecdote in the book had to do with the POSIX system call “select” https://man7.org/linux/man-pages/man2/select.2.html. (I know because I have my first edition print of the book that I purchased in 1997 right here in front of me 😉)
for example, if you require totally heterogenous data, you could make a comparator that says a number is always less than a map
(defn- vendor-comparator [a b]
(cond (and (empty? (:vendor/approved a)) (seq (:vendor/approved b)))
b
(and (seq (:vendor/approved a)) (empty? (:vendor/approved b)))
a
:else
(compare (:vendor/name a) (:vendor/name b))))
This is my comparatorthat's a bad comparator, it needs to return a number for all branches
@zilti the doc string says you need to implement java.util.Comparator https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Comparator.html
clojure's IFn is kind enough to implement Comparator for you, but you still need to return the right type if using your function that way
"Comparator. Returns a negative number, zero, or a positive number when x is logically 'less than', 'equal to', or 'greater than' y."
the only place "bigger" as a string appears in the clojure git history is in a comment for the asm library that is used to generate jvm bytecode
And "If no comparator is supplied, uses compare. comparator must implement java.util.Comparator."
I guess this sums it up nicely: https://clojure.org/guides/comparators
Quick note: Launched "Journal Together" -- a lil app I was working on to play with clojure again on HN (It's on the Show
page). Noted the experience of using clojure in there -- tl:dr repl + jvm == a loot of fun. Big shoutouts to the community here, got a lot of help in pushing this out -- esp @seancorfield and @noisesmith