I tried to profile it with async while it was doing a lot of work and apart from GC there's a lot of work done inside orchard

you are using JDK8, correct? One line of research if checking if the error simply goes away by using a more recent JDK

And another is using JVM flags to avoid OOMs, here are mine (for a 64GB MBP) https://github.com/reducecombine/.lein/blob/f01aec663ff4ac8f8e83f8eca93a13b2cf99aa4b/profiles.clj#L10-L43 People would be surprised by how many issues go away with these flags - Clojure can be hungry in general and it can show in arbitrarily many libs

will try to make sense of the flamegraph anyway 🙏 maybe something is problematically lazy

Thanks for the prompt response. I'm using JDK 8 and I'm a bit stuck with this for a while (a legacy app). I haven't tuned it much but it's got 1GB heap (to mimic production settings). I'll keep an eye on it and will see if this becomes a frequent issue.

From your settings, I would say only setting large Xmx and Xss could have a positive effect.

XX:CompressedClassSpaceSize also (although not so much for this specific problem) In general I don't recommend mimicking production settings. A dev repl can have much more in it e.g. cider-nrepl, refactor-nrepl, kondo (when used as a jvm lib) etc so that already makes the memory requirements different Perhaps in CI it's more apt

Another line of research is that gc overhead limit exceed is an unusual wording, perhaps it's specific to a given GC? Not sure if you are explicitly choosing one. Swapping impls might do the trick. Also in the second screenshot I'd need to be able to read what is said in the rightmost panes

Hopefully this will fix it, at the same time everything I said stands :) https://github.com/clojure-emacs/orchard/pull/154 working on a 1GB jvm is not guaranteed

Thanks for the prompt action! Note that gc overhead limit exceed is a very common thing nothing special: The rule tends to be something like : >  too much time (e.g. 98%) of a time interval (e.g.  no less than 1 minute) during which multiple (e.g. at least N) full GC cycles have run was spent in  stop-the world GC. (https://groups.google.com/u/1/g/mechanical-sympathy/c/TpyjsaPhM5U)

Also, the app has been working with 1gb just fine for a couple of years. I'm adding a couple more pictures, the second one is at the very far right but it also consumed only a tiny piece of CPU time

That said, I'm really impressed by the prompt fix you did 🙂

(If you saw it already, forget what I said above - it was a stupid mistake on my side and I deleted the messages)

I guess I was lucky because I didn't see them :)

>  too much time (e.g. 98%) of a time interval (e.g.  no less than 1 minute)  what I find peculiars about this is that memory hasn't literally run out, but rather an heuristic is being hit

> Also, the app has been working with 1gb just fine for a couple of years. But tooling changes, so it's a bet that can fail at any time (as it has for me with various other tooling or libs)

> That said, I'm really impressed by the prompt fix you did cheers The changes made it to the latest CIDER snapshot, would be curious as to whether they make a difference

Snapshots are published to MELPA? That is I can just upgrade the emacs cider package?

depends on what melpa are you using

a stable release will be cut within a week

C-c C-s b is sesman-link-with-buffer which does this.

thank you!

@U5NCUG8NR has been building https://github.com/IGJoshua/farolero and I believe nrepl integration is on the roadmap

It is definitely on the roadmap. Unfortunately it will never come all the way to what common lisp can do because I can't inject usage of farolero into clojure core, and I'm not currently aware of a way to hijack a thread when an exception is thrown before it unwinds, so it'll be limited to where you place wrap-exceptions calls and similar.

Any library or application using farolero for its stuff will get a breakloop-style debugging experience for stuff that they've written to support that workflow, though.

Very cool! As awesome as it would be to have full a CL condition system in Clojure, I was imagining something lighter-weight built on CIDER’s existing debugger — IIUC the CIDER debugger https://docs.cider.mx/cider/debugging/debugger.html#using-the-debugger before running it, and I was imagining this could similarly wrap everything in a try/catch, and drop to the debugger whenever it caught an exception. > I’m not currently aware of a way to hijack a thread when an exception is thrown before it unwinds I wonder whether you could temporarily capture the stack before evaluating each expression, and then if the expression resulted in an exception, restore the saved stack and drop to the breakloop. That might result in user-visible side effects happening twice, but seems like it’d still be very useful. I may be being painfully naive here; like I said I’ve just started to think about this problem.

I’m basically wondering what if anything I could accomplish in this vein as an experienced Clojure programmer with a couple of weeks between jobs to devote to it.

That "capture a stack" is continuations, and the JVM does not currently expose a way to do that.

and try/catch misses the bit where you have to catch things before they unwind, although just wrapping all functions in this type of try/catch and then retrying the function may be possible if the cider debugger already does this, but I want to note that the cider debugger doesn't wrap most functions, just the ones you explicitly wrap.

> That “capture a stack” is continuations, and the JVM does not currently expose a way to do that. Gotcha, makes sense, not something I’ve ever looked into with the JVM. > although just wrapping all functions in this type of try/catch and then retrying the function may be possible That’s definitely what I was imagining. > the cider debugger doesn’t wrap most functions, just the ones you explicitly wrap. cider-debug-defun-at-point is what I’m thinking of; per the CIDER docs that “will insert as many breakpoints as possible into the form”.

It does certainly sound like there’s no easy intervention that would enable the true CL breakloop with restarts, but it seems like “break to debugger on exception” would have some value on its own, and I’m hoping that the “wrap every single expression and subexpression with try/catch” could prevent the unwinding to keep the state of the stack as local as possible. Might still be being too naive though 😆

right, my main thought is that often when an exception appears out of nowhere and it would be the most useful to have this kind of breakloop is when it's already 10 layers deep in the callstack and now you have to go find the function that threw the exception, mark it as a debugged function, and then retry your whole thing up to that point to get it to fire.

That said, I don't want to discourage you from trying, if you're able to get something that works for that usecase I think it'd help me out quite a bit in my personal workflow.

> right, my main thought is that often when an exception appears out of nowhere and it would be the most useful to have this kind of breakloop is when it’s already 10 layers deep in the callstack and now you have to go find the function that threw the exception, mark it as a debugged function, and then retry your whole thing up to that point to get it to fire. My ideal would be to call something like toggle-debug-on-exception and then every expression would be transparently wrapped. > That said, I don’t want to discourage you from trying, if you’re able to get something that works for that usecase I think it’d help me out quite a bit in my personal workflow. Not discouraging at all! This is super-helpful info that’ll save me at least one big blind alley, and I really appreciate you taking time to talk about it. It definitely doesn’t have nearly as much potential as Farolero! But I’m hoping that it could be done in a relatively lightweight way and wouldn’t require writing/rewriting the code to specifically take advantage of it. Worst case it’ll be a fun way to spend a couple of weeks even if nothing comes of it 😆

curious if project loom will lift these limitations

loom has continuations at a low level, but I don't know how much that would actually help if there isn't a reasonable way to do a "break on exception" style facility. I know that some debuggers can do this, so maybe it's feasible?

Honestly my main thought would be that I suspect that normal threads would dodge this by not actually having continuations, since they're only needed on the virtual threads.

can the uncaught exception handler be such a facility?

> can the uncaught exception handler be such a facility? https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Thread.UncaughtExceptionHandler.html the stack would be fully unwound by the time it hit the uncaught exception handler, so it would be of limited use for something like a debugger.

yes

Oh, hmm. I see @U0GN0S72R has already created https://github.com/gfredericks/debug-repl, implemented as nREPL middleware. I think that means that automatically dropping to a debug repl on exception would “just” require automatically wrapping each call (or maybe even just the top-level call?) in the https://github.com/gfredericks/debug-repl/blob/master/src/com/gfredericks/debug_repl.clj#L256 macro. Although I’m a bit confused as to how/why debug-repl is middleware, since at least in its https://github.com/gfredericks/debug-repl/blob/master/README.md the breakpoints are being set manually. So playing with that will be my next step 🙂

Oh, I think I see, maybe it’s middleware https://nrepl.org/nrepl/design/middleware.html#sessions its own set of dynamic vars (eg *1, *e).

My vague recollection is it's more fundamental than that

It's gnarly code though. Can't figure out at a glance what problems it was trying to solve

Ah, gotcha. It might make it extra-useful for what I’m trying to do anyhow, not sure yet. I’ll play with it. Thanks for making it!

@U0GN0S72R wow, that really is some genuinely gnarly code 😅

Oops

Super cool though. I’m having fun trying to reverse engineer it a bit 😁

I usually wrap them in a do if I think I’m going to frequently want to eval them all.

This is exactly how I use rich comment forms. I can safely evaluate the namespace without worrying about code I do not want to load in the comment form

> I usually wrap them in a `do` ie

(comment
  (do
    (foo)
    (bar)
    (baz)
    ))

@U077BEWNQ thanks for the quick reply! I’ll try that.

@U05254DQM yeah, it makes sense (just trying to get my head around all the things about Clojure :)

You can definitely inspect locals without unwinding, and you can pop out a level (although from that latter I don’t think you can get back inward). Locals being shown on left:

This bit shows the debugger commands, which are documented https://docs.cider.mx/cider/debugging/debugger.html:

Not sure what you mean by unwinding but every now and then, I wish it was possible to step out of the instrumented function and continue debugging higher up the stack. I don't think this is doable today. Like here, If I instrument only baz I would like to be able to jump out and continue stepping through the code in bar and possibly foo https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/debugging.clj#L6

(defn baz [z]
  (let [zz (+ 10 z)
        zzs (repeat zz z)]
    (mapv inc zzs)))

(defn bar [y]
  (let [yy (+ 5 y)]
    (baz yy)))

(defn foo [x]
  (bar (inc x)))

(comment
  (foo 3)
  ,)

The other thing that doesn't really work for me is *s*tacktrace. When I press s nothing happens, at least I don't see anything - I though it would print the stacktrace to stdout.

I'm not sure how in works but it apparently is able to step through a function which wasn't explicitly instrumented. I would love to see a variant of out that would have the same capability.

Ok, I thought that perhaps s isn't working because I use spacemacs in evil mode. But it still doesn't seem to do anything after I switch to emacs mode. I'm wondering where the "s" shortcut is defined - it seems most of them are here, but not "s": https://github.com/clojure-emacs/cider/blob/master/cider-debug.el#L400-L424

@U06BE1L6T working fine for me in spacemacs/evil (CIDER 1.1.1), so I don’t think that’s your problem. The binding to s might actually be https://github.com/clojure-emacs/cider/blob/master/cider-debug.el#L191 in cider-debug-prompt-commands.

@U077BEWNQ so it actually prints the stacktrace in the REPL buffer or what exactly it does?

For me it pops up the *cider-error* buffer and shows it there.

I see, thanks - have no idea why it doesn't work for me 😞 Ah, maybe... Oh yes! It's a sideffect of me disabling the error buffer via cider-show-error-buffer set to never. (I found it quite slow and annoying) Definitely unintuitive!

in works by recompiling the function with instrumentation before you call it

at least I'm fairly sure

Out can't do that because you can't recompile code that's already been partially run

Yes, makes sense.