I had no idea that you can export SHELLOPTS like that, that's very useful

I'm just getting:

$ (set -x; export SHELLOPTS; clj -Sforce -M -e 42)
+-zsh:8> export SHELLOPTS
+-zsh:8> clj -Sforce -M -e 42
42

perhaps bash isn't inheriting this from zsh or so

ah yes, bash -c '(set -x; export SHELLOPTS; clj -Sforce -M -e 42)' worked

It's not that bash wasn't picking up zsh's env. It's that zsh doesn't record set in SHELLOPTS

$ (echo $0; echo $SHELLOPTS; set -x; echo $SHELLOPTS)
bash
braceexpand:hashall:histexpand:history:interactive-comments:monitor:vi
+ echo braceexpand:hashall:histexpand:history:interactive-comments:monitor:vi:xtrace
braceexpand:hashall:histexpand:history:interactive-comments:monitor:vi:xtrace

$ (echo $0; echo $SHELLOPTS; set -x; echo $SHELLOPTS)
zsh

+zsh:2> echo

Have you tried -Sdeps '{:mvn/local-repo "the-path"} ?

Ahha! That is what I was missing. Thanks!

The repl typically stores up to 3 recent values in *1, *2, and *3. I often work with large values in the REPL and personally, I wouldn't want references to more than 3 values unless it was opt in.

Similarly, I was thinking that 4 would be a more cautious default because of the potential memory usage. But too low of a value could prevent users from seeing how data flows / evolves over time.

I am a fan of many* of the builtin cider tools (like cider-inspect ). Maybe any tools that use the history would have an easy way to see the history length and an easy way to extend it if they're using those tools.

My main objection to keeping any more references than 3 is for anyone that would be trying to debug an issue related to extra automatic references. To debug the issue, you need to know specifics about general repl tooling and specifics about cider's tooling (which may or may not end up in a bug description).

> I am a fan of my of the builtin cider tools (like cider-inspect ). Maybe any tools that use the history would have an easy way to see the history length and an easy way to extend it if they're using those tools. In case you missed it from the issue I linked to, I am precisely trying to bridge taps and the inspector :) If you believe that such an integration should be fundamentally different, a post in that GH thread would be much appreciated

...Maybe both the tap activation and its size are better explicitly specified by the user. OTOH one tends to want to create things that "just work". It's a hard balance, if one makes it too easy, one may be accused of authoring 'magic workflows'

I'd say it should be opt in - my own repl helpers also have similar tools for working with tap: https://github.com/lukaszkorecki/rumble/blob/7bead7852ad3c2b22c578925bc3497880f7a95bd/src/r.clj#L201-L222

(yes, I know that I reinvented bits of CIDER)

Yeah, probably it should be disabled by default, but its activation should be very easy, via defcustom

As a heavy user of Portal who relies on CIDER middleware and Portal middleware, and who wires up tools.logging to tap> into Portal, I would definitely want this new CIDER feature disabled by default. In Portal, it's easy for me to keep as much history as I want but also easy for me to clear that history too. Portal also makes it straightforward to provide your own "tap-log" so you can control how/where tap> values are actually stored (I use two different tap logs already: one for middleware and one for explicit tap> calls).

Got it, appreciated. Curious, what did you mean by middleware?

https://github.com/clojure-emacs/cider-nrepl and https://cljdoc.org/d/djblue/portal/0.48.0/doc/guides/nrepl are the two middleware I use.

While a small queue will alleviate potential memory issues, it will also potentially make it hard to debug something if you have tap> in a loop or lots of tap> calls executing in the call chain -- so you'll definitely want easy ways to expand/contract/clear the tap log.

I'm wondering how you'd go about exposing those sorts of controls? Perhaps in the inspector itself it might be sufficient to turn this on/off and clear the queue -- and use an unlimited queue when it is enabled (disabled by default).

FWIW, tap> internally uses a 1,024 queue: https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L8031

(`tap>` drops values if that queue ever gets full -- I don't think I've ever overrun that queue, even with really heavy tap> usage tho')

> I'm wondering how you'd go about exposing those sorts of controls? It might be usable enough with M-x cider-whatever or also with the handy CIDER inspector key bindings (https://docs.cider.mx/cider/debugging/inspector.html#usage). Clients other then Emacs could have a dedicated nREPL op - easy stuff

I think tap> is awesome 🙂 I've been using it heavily ever since it dropped in an early alpha of 1.10. Glad to hear you're thinking of supporting it in the CIDER inspector! I used REBL (now Morse) for a while, then Reveal, and now Portal -- makes a massive difference to my workflow.

I got fed up of typing (def dbg (atom nil)) so many times 😇 There had been a few attempts of a better tap experience in cider, but sometimes it's hard to get things driven to the finish line.

> FWIW, tap> internally uses a 1,024 queue If one taps x and nobody's listening, will x remain in the queue?

No, the queue is emptied continuously. See tap-loop just below where I linked to the queue.

It will only "back up" if someone adds a tap listener that is slow or gets blocked (since the tap-loop runs on a single thread).

> I would definitely want this new CIDER feature disabled by default. Most likely this will be the case. Anyway, is there a big reason? As in, possible drawbacks / dangers? (I'm seeking to possibly gain a less obvious insight)

I was thinking of the case where the queue was unbounded (which you may well need for a debugging session), so you want it off by default.

(I don't think a bounded queue is going to be all that useful in real world debugging, unless that bound is pretty large)

I think keeping each value inside SoftReferences is a fine tradeoff. Values would be there most of the time, but it won't break the program if huge values start coming through the tap and fill up the heap.

> it won't break the program if huge values start coming through the tap and fill up the heap. It's not uncommon for me to test to make sure I'm not holding onto references to make sure my program isn't leaking memory. I believe holding soft references would make that harder to test.

I agree that holding extra references for dev tooling can be useful, but I'm not sure it's a good default behavior for every development environment that uses a cider repl.

Thanks! > (I don't think a bounded queue is going to be all that useful in real world debugging, unless that bound is pretty large) It may be a reasonably frequent usage pattern to look at the latest thing? Or the latest few things. It is the type of usage I give to bare atoms, and that I was giving to tap> these days. Maybe once in a while I would want to look a "needle in a haystack" where indeed, anything but a large/unbounded data structure will be useless. Maybe one can document that (when (interesting? x) (tap> x)) is a pattern that would be useful with CIDER's would-be tap. Ultimately, of course, users can always grow, shrink and remove the bound of the data structure.

How do you perform such a test usually?

> It's not uncommon for me to test to make sure I'm not holding onto references to make sure my program isn't leaking memory. Most certainly your test (more so your performance-test) setup should be decoupled from CIDER. Not that CIDER necessarily adds a lot of bloat... but I wouldn't run a serious perf test with CIDER in

> How do you perform such a test usually? It depends. Sometimes I use print statements in a finalizer, sometimes, it's via stat tracker that counts allocations and deallocations so I can make sure they match.

(Portal uses an unbounded list but it is "off" until you add Portal as a tap listener and you can clear the list easily -- it's a simple model and works well: off, on/unbounded, clear -- so you only need on/off and clear: two controls)

@U45T93RA6 I think the size of the queue is a moot point regarding the memory consumption. 100 small objects may be smaller than 1 big object. It makes sense to limit the queue to some reasonable value, but there is no "safe length" for it

@U7RJTCH6J I see; but then, you would have to tap> those objects in order for your testing to break, right?

Also, from the linked issue:

(defn bounded-conj
  [^long n coll x]
  (let [b (== (count coll) n)]
    (cond-> (conj coll x)
      b pop)))

There's a bounded-count that could be used but I don't know what Clojure version added it...

> I see; but then, you would have to tap> those objects in order for your testing to break, right? (edited) For some reason, I was under the impression that the proposal would both be adding a tap and automatically tapping results from forms sent to the repl. 😳 nvmd.

The way I would implement bounded-conj is:

(defn bounded-conj
  [^long n coll x]
  (let [b (>= (count coll) n)]
    (cond-> (conj coll x)
      b pop)))

bounded-conj assumes that coll never exceeds the bound which I think is a footgun.

We tend to leave tap> calls in the code since it's a no-op if there are no tap listeners active. We have several calls active in production, but until we attach Portal, any tap>'d data is thrown away. Then we can debug, with Portal, and then remove the Portal listener and it goes back to being a no-op.

The idea of automatically adding a tap that is opt-out seems potentially problematic, especially since there's no public API to inspect the list of current taps (afaik).

It will be opt-in - that became clear after all this talk :) I can imagine it being particularly bad in a production server (where some people might be tempted to be running cider-nrepl... which I don't recommend at all), given the 'unconditional taps' pattern that Sean describes Lots of good input in this thread - thanks all! :man-bowing: I'll try to make something sensible out of it.

For me personally, an OK experience would be if the tap only started working after I do M-x cider-inspect-tap or something. Once activated, the tapped values are stored in a queue of SoftRefs, and the queue can be navigated. The queue can be quite long since memory woes would be handled by softrefs.

I'd also control initialization with a defcustom, so that it's running from the beginning. It makes sense for a codebase that already has taps - one may want to see stuff tapped before the first use I'm in for SoftRefs. It might slightly complicate the impl and UI when a SoftRef was actually cleared (as it would produce a queue that could look inconsistent: [val, val, <no val because GC>, val, val])

> as it would produce a queue that could look inconsistent Yes, it would. For me, navigating the history of values at the same time as GC is constantly triggering has an inherent "here be dragons" disclaimer. Perhaps, somebody would want a different experience.

Two things I'd want from this ability to "install" cider's taps, with defcustom to do so automatically, and a different buffer using the inspector's mode, so I can enjoy both inspection and taps

And thank you for taking the time to work on this 🙏

from my experience using and developing #C03KZ3XT0CF everyday, which retains every pointer while your app runs in a timeline, having an easy way of starting/stopping this recording, clearing it, and looking at the heap usage has been enough. Imho if it is easily controllable, and "stopped" by default so no surprises, is a nice feature

It sounds like you want to export (and later slurp) resources rather than files in the filesystem. Are you familiar with them?

nope

The text file would be a resource in the original project, and in the consumer project, you (slurp (io/resource "a.txt")) (don't specify src/) The whole notion of resources is fairly easy to pick up, probably chatgpt can give a decent primer

ah that was great, thanks!

Include files in the resources directory and add that directory on the project path, then use `http://clojure.java.io/resource to define the file location (relative to resources directory) https://clojuredocs.org/clojure.java.io/resource