Fork me on GitHub
#clojure
<
2020-07-21
>
stopa04:07:08

watching Sean Grove's talk on Dato -- https://www.youtube.com/watch?v=BiplJ4AFwCc -- an RFC framework he and Daniel Woelfel were working on. The idea of using datomic straight from the client is so darn cool -- like a firebase on steroids

wombawomba08:07:26

What are the hurdles to building a PDB-style debugger for Clojure?

vlaaad09:07:13

what’s the usecase? I find debugging needed in mutable environments where it’s important to stop the world and see current state of different pieces of environment. With clojure’s immutability by default, I can just tap> /`prn` interesting stuff on interesting points in time

chrisblom09:07:30

tap> / prn is a poor man's debugger. @U15RYEQPJ have you tried cider's debugger?

wombawomba09:07:05

@U47G49KHQ I really like being able to just pause and step through code, especially when it's written by others and is cryptic. And preferably without any external tooling

wombawomba09:07:53

@U0P1MGUSX yeah I've used both Cider's and Cursive's debuggers

wombawomba09:07:09

The question isn't as much "how do I debug Clojure?" as "how would one go about building a pure Clojure stepping debugger (and why hasn't anybody built one?)"

sogaiu10:07:33

i can relate to wanting to pause and step through code. the closest things i've found are debug-repl by georgejahad: https://github.com/GeorgeJahad/debug-repl and a more recent attempt: https://github.com/Ivana-/bb-debug

wombawomba11:07:01

neat, thanks for those links @UG1C3AD5Z

wombawomba11:07:30

debug-repl looks a lot like what I’d want

sogaiu11:07:56

np -- if you find anything better or related, i'd love to know 🙂

wombawomba11:07:50

yeah, I think I might try to build something 🙂

wombawomba11:07:17

the main problem with debug-repl afaict is that it doesn’t allow stepping

wombawomba11:07:27

I wonder how one would go about adding something like that…

chrisblom11:07:18

you can do stepping with cider's debugger

sogaiu11:07:14

yes, i have used that and cursive's, though i prefer the latter as it can dip into java stuff too and i find the ui for debugging nicer -- i also don't use cider.

sogaiu11:07:28

in https://github.com/Ivana-/bb-debug/blob/master/src/bb_debug/example.clj there is some mentioning of step-by-step debugging, fwiw

wombawomba11:07:33

yeah I saw that — the problem afaict is that it only permits step-by-step debugging within the dbg-all* macro

wombawomba11:07:27

being able to step out of the frame/scope where the debugger is called would be preferable

sogaiu11:07:54

sure -- i thought about it a bit before, but unfortunately, didn't come up with anything great.

sogaiu11:07:23

i think the best i came up with was instrumenting a lot 🙂

sogaiu11:07:57

i think there have also been some attempts using low level jvm things too.

sogaiu11:07:16

ah, i knew there was another one (pure clojure iirc): https://github.com/razum2um/clj-debugger/ -- step is a todo for this one

wombawomba11:07:25

cool, those look great

wombawomba11:07:41

unfortunately there’s no discussion that I can find on how they planned on implementing (step)

sogaiu11:07:10

perhaps it's possible one or the other of them might respond to queries?

sogaiu11:07:23

i suppose it's possible there's something relevant regarding stepping in cider's debugging functionality. i don't really know how that one works.

wombawomba12:07:51

yeah I suppose

wombawomba12:07:15

any idea how one could find the relevant source code in cider?

sogaiu12:07:01

my recollection is that a fair bit of the work was by: https://github.com/Malabarba

wombawomba12:07:37

okay yeah, doing a blame on that code seems to suggest that too

sogaiu12:07:20

i think i remember seeing some bits discussed in some of the pull requests for adding the debugging stuff. taking a look now.

sogaiu12:07:44

i'm not sure how much changed since this: https://github.com/clojure-emacs/cider-nrepl/pull/220 but it has some explanation of some of how things worked at one point

wombawomba12:07:57

it seems like the heavy lifting is happening in some other file (most of the code in cider-debug.el seems to end up calling https://github.com/clojure-emacs/cider/blob/master/cider-debug.el#L440-L458)

sogaiu12:07:24

ah, there's some explanation in that file

sogaiu12:07:57

lol, looks like it's the "instrument all the things" approach

sogaiu12:07:12

ok "of interest"

sogaiu12:07:36

it could be that Ivana or razum2um might have some ideas

sogaiu12:07:46

that are different from instrumenting everything

sogaiu12:07:46

i'm not sure how that works, but when i looked into java debugging stuff before, i think what i found is that jvm bytecode is instrumented at runtime.

wombawomba12:07:38

yeah that sounds about right

sogaiu12:07:56

clojure produces jvm byte code, so i suppose it might be possible to change it to emit different byte code

wombawomba12:07:16

I imagine Cursive’s approach is probably something along those lines?

sogaiu12:07:01

what i recall about it is that it uses "official" interfaces to jvm debugging -- but i don't recall clearly.

sogaiu12:07:25

or may be there was some intellij-way to do it.

wombawomba12:07:20

hmm yeah, I could imagine that the Cursive debugger is built on top of InteilliJ’s Java one

sogaiu12:07:23

search for PDA and JDI

wombawomba12:07:51

hah yeah I just found that same talk :)

wombawomba12:07:49

Cursive is closed source right?

sogaiu12:07:32

yes, i believe so

sogaiu12:07:56

ah nice, there's an associated repository too: https://github.com/eugenp/tutorials/tree/master/java-jdi

wombawomba12:07:35

maybe I could just start off writing a Java debugger in Clojure using JDI, and then see if I can tack on Clojure debugging functionality on top of that

sogaiu12:07:58

perhaps so -- that other link from before: https://github.com/indiejames/clojure-debug-demo -- may be relevant

sogaiu12:07:00

ah, one thing -- that project uses jdk8 and tools.jar.

sogaiu12:07:27

so if you want a more recent jdk, you may need to do something different for the tools.jar part.

wombawomba12:07:38

it seems like one limitation with JDI might be that you need two separate JVMs (one for the main process and one for the debugger) — so dropping into a debugger from the main process (like PDB does) might be hard

joaohgomes12:07:04

I don't know if relevant for your use case. For debugging, I use the combination of scope capture and Cursive/Intellij debugger. Although you can set breakpoints and rebind/inspect vars using just scope capture. https://github.com/vvvvalvalval/scope-capture

wombawomba13:07:40

neat, thanks, scope-capture looks really cool

noisesmith15:07:11

a huge gotcha with these sorts of debugging: you need to turn off locals clearing, which breaks some code badly

noisesmith15:07:45

if you leave locals clearing on, the things you want to access are garbage collected, if you turn it off, functions that should be fine can blow up the heap

wombawomba18:07:30

@noisesmith would that apply to the JDI approach as well? Also, how is locals clearing controlled?

noisesmith18:07:53

right, this effects anything which directly controls real execution, there's a startup flag which controls locals clearing https://clojure.org/reference/compilation#_locals_clearing

noisesmith18:07:19

if you use clojure as intended, storing immutable data and using it later is less complex than single stepping a vm, and the real risks of side effects (IO, system calls in general, etc.) aren't handled properly in a step debugging environment any better than they are with manually recreated calls

wombawomba19:07:49

got it, thanks

sogaiu00:07:01

@noisesmith thanks for the note about locals clearing and debugging. i see it appears to be mentioned here: https://cursive-ide.com/userguide/repl.html#starting-a-debug-repl

noisesmith01:07:26

I shared that same http://clojure.org link

sogaiu01:07:16

ah you did! sorry i missed it 😅

Mikael Andersson01:07:45

If you are interested in poking around with JDI, there is also cdt of which at least a fork (indiejames) was updated in 2017. https://github.com/GeorgeJahad/cdt/network and then there are the JVMTI low level interfaces, and related on debuggers on the JVM, one must of course mention https://github.com/JetBrains/intellij-community/tree/master/java/debugger/impl/src/com/intellij/debugger/jdi (apache licensed) as a reference of advanced usage, and https://github.com/JetBrains/jdk-sa-jdwp for connecting to a running VM (GPL)

Mikael Andersson01:07:59

If locals clearing is implemented to keep stack size down and if it's the stack size that blows up without it, -Xss jvm option can be used to give you more stack headroom while playing around in a debugger/repl/what-ever-your-taste-is.

noisesmith16:07:30

no, it's about heap allocation, and blowing up the max heap

noisesmith16:07:37

(let [els (map produce-large-result (range))]
  (run! f els))
with locals clearing that can run indefinitely, without it fills up the heap

sogaiu22:07:53

i found this rh post about fine-grained locals clearing: https://groups.google.com/forum/#!msg/clojure/FLrtjyYJdRU/1gzChYsmTpsJ iiuc, the code from noisesmith demonstrates the following (slightly edited) quote from the post: > Unfortunately, on the JVM, [snipped], in many cases, [snipped] - the local is considered a live reference and is thus not GCed. This yields the infamous 'holding onto the head' problem, and subsequent Out Of Memory errors on large data sets.

Mikael Andersson01:07:24

Yes locals are always live references, that's how the VM determines liveness. Depends on the GC, but essentially it walks the stack(s) and marks all objects it can see, and then purges the rest. I've only just dove into learning a bit more of the internals, mostly because I don't like being surprised by the runtime. However, it kind of sounds like the locals clearing is closely related to lazy sequences. I can't really figure out any other way to have collection that grows boundlessly in size in a deep stack frame running a language using almost exclusively immutable data.

Mikael Andersson01:07:52

And since lazy seq's are only pretending to be immutable?

noisesmith17:07:07

locals are not always live - clojure clears them if it can prove they don't escape the closure

noisesmith17:07:49

the vm doesn't determine this, the clojure compiler does (via the way it structures the byte code implementing the closure)

noisesmith17:07:19

the GC does not walk the stack, it walks the heap

Mikael Andersson21:07:06

Maybe I was unclear, locals in this context was referring to values on the jvm stack (primitives, and references), which referenced values are always considered live. Which is why clojure needs to null them in the case they reference a lazy sequence or similar mutable structures which otherwise will behave as ever expanding singly linked lists. You are right of course that the GC doesn't only walk values on the stack, but eg. a mark and sweep walks the live graph starting from the stacks, which together with some additional references (static variables, JNI references, and threads) serve as the "GC root"

Mikael Andersson21:07:00

I mean, you know that, I guess we're just talking across each other a bit.

Mikael Andersson21:07:33

By the way, I think I forgot earlier, but T thanks for the pointers about the local clearing, much appreciated! The clojure compiler still clears quite a few locals even with local clearing disabled, but now when I know what to look for I can easily make a tweaked version if need be.

wombawomba12:07:50

Cool, thanks for all the info and links! I played around with JDI a bit, and I have to say it seems comparatively straightforward. I’ll try to use it to build something proper when I get some free time :)

sogaiu12:07:17

@U15RYEQPJ if you get around to it, please let me know 🙂

Joshua Suskalo13:07:38

@hiredman I did mention that I am using newlines to denote the end of a form, and I've also tried chaining multiple newlines at the end to see if that had any impact.

Joshua Suskalo13:07:20

Yet despite that, it still never produces a value it seems

g18:07:24

anyone familiar with a websocket client library compatible with jdk 11? 😕

zilti18:07:06

What are idiomatic ways/libs to sort a vector without using sort-by?

dpsutton18:07:29

What are you trying you trying to solve?

noisesmith18:07:49

sort and sort-by are the idiomatic options, they use the underlying vm for the algorithm

zilti18:07:52

Trying to sort a vector of maps. No, sort-by does not work always, sometimes it just breaks with a nasty exception

noisesmith18:07:07

the exception means you need a better comparator

zilti18:07:13

Giving me stuff like "Can't compare <print of my map here> with 1"

noisesmith18:07:16

any other sort mechanism would require the same thing

zilti18:07:50

And there most certainly is no 1 anywhere in my vector, nor does my comparator ever return a number

noisesmith18:07:14

I'm confused, a comparator is by definition a function taking two items and returning a number

dominicm18:07:56

"case isn't broken" (or is it switch?)

noisesmith18:07:37

is this a reference I'm missing?

seancorfield18:07:00

It's a reference to people saying "the compiler is broken" when it's nearly always their own code.

noisesmith18:07:42

oh, right, of course - I hadn't heard it put this way

g18:07:46

solution: write a compiler

seancorfield18:07:58

"It is rare to find a bug in the OS or the compiler, or even a third-party product or library. The bug is most likely in the application." -- that's where I remember "compiler" from I guess 🙂

🚮 3
💾 3
noisesmith18:07:01

it's similar to the more general "if you hear hooves, assume a horse not a zebra"

noisesmith18:07:28

check the common case if all else is equal, in software the common case is an error in the most recently written code

seancorfield18:07:49

(of course, if you actually are a compiler writer -- as I used to be in the '80s/'90s 🙂 -- then the bug could well be in your compiler!)

noisesmith18:07:10

right - but even then, it's more likely to be in newer code than older code, unreleased vs. released code, etc. - as a good first heuristic at least

💯 3
dominicm20:07:38

The story in the book is nice. Kinda. Engineer was convinced case was broken. Actually they didn't understand it. Not dissimilar from in clojure really.

the2bears15:07:13

I remember hearing it as "select isn't broken" a reference to SQL.

deactivateduser23:07:04

Not to be the “well akshuwally” guy, but the anecdote in the book had to do with the POSIX system call “select” https://man7.org/linux/man-pages/man2/select.2.html. (I know because I have my first edition print of the book that I purchased in 1997 right here in front of me 😉)

the2bears02:07:13

That is an awesome bit of info 🙂 Appreciate you sharing.

noisesmith18:07:59

for example, if you require totally heterogenous data, you could make a comparator that says a number is always less than a map

zilti18:07:03

It is very homogenous data, a vector where every map has the same keys

g18:07:28

could you post a code sample?

zilti18:07:40

(defn- vendor-comparator [a b]
  (cond (and (empty? (:vendor/approved a)) (seq (:vendor/approved b)))
        b
        (and (seq (:vendor/approved a)) (empty? (:vendor/approved b)))
        a
        :else
        (compare (:vendor/name a) (:vendor/name b))))
This is my comparator

noisesmith18:07:02

that's a bad comparator, it needs to return a number for all branches

zilti18:07:28

Doc says it has to return whatever input is "bigger"

zilti18:07:33

So that is what I did

hiredman18:07:12

where do you see that doc?

zilti18:07:43

Don't really remember. But returning numbers simplifies it

noisesmith18:07:01

clojure's IFn is kind enough to implement Comparator for you, but you still need to return the right type if using your function that way

seancorfield18:07:24

"Comparator. Returns a negative number, zero, or a positive number when x is logically 'less than', 'equal to', or 'greater than' y."

hiredman18:07:54

the only place "bigger" as a string appears in the clojure git history is in a comment for the asm library that is used to generate jvm bytecode

😂 3
seancorfield18:07:55

And "If no comparator is supplied, uses compare. comparator must implement java.util.Comparator."

zilti18:07:59

Weird enough that it actually worked at the beginning

zilti18:07:43

Oh, I didn't find that guide, thanks

stopa22:07:47

Quick note: Launched "Journal Together" -- a lil app I was working on to play with clojure again on HN (It's on the Show page). Noted the experience of using clojure in there -- tl:dr repl + jvm == a loot of fun. Big shoutouts to the community here, got a lot of help in pushing this out -- esp @seancorfield and @noisesmith

👍 15