I don't like :injections much you can run arbitrary code in a project.clj by placing it before defproject, https://github.com/clojure-emacs/orchard/blob/master/project.clj shows this particularly

Interesting, thanks. I ended up using a shell script which is run by Make before lein uberjar

do you know why pred is been called after first true?

if it is about lazy-collection been chunked by default then have a look at https://clojuredocs.org/clojure.core/chunk#example-5c9cebc3e4b0ca44402ef6ec

(defn re-chunk [n xs]
  (lazy-seq
    (when-let [s (seq (take n xs))]
      (let [cb (chunk-buffer n)]
        (doseq [x s] (chunk-append cb x))
        (chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))

(first
  (filter (fn [l]
            (prn l)
            (even? l))
    (re-chunk 1 (range 1 1000))))

I don't in fact know that it is. However, I read a response from alexmiller saying that clojure does not guarantee that this pattern halts the traversal when the predicate returns true.

wrt unchuck, I found the following function in my odds-and-ends utils.clj file.

;; code thanks to 
(defn unchunk [s]
  (when (seq s)
    (lazy-seq
      (cons (first s)
            (unchunk (next s))))))

I am finding many places in my code base where I'm depending on my mis-understanding of filter

(defn conversion-C12
  "AXBC + !X = ABC + !X"
  [self]
  (let [combos (filter gns/combo? (operands self))
        duals (setof [td combos] (dual-combination? self td))
        comp (filter (fn [n]
                       (and (gns/not? n)
                            (exists [td duals]
                                    (member (operand n) (operands td)))))
                     (operands self))]
    (if (empty? comp)
      self
      (letfn [(f [td]
                (cond (not (gns/combo? td))
                      td

                      (not (dual-combination? self td))
                      td

                      :else
                      (create td (remove-element (operand (first comp))
                                                 (operands td)))))]
        (create self (map f (operands self)))))))

Why would that unchunk work at all? If s is chunked lazy sequence it will still load 32 elements when first s is called

you are wrapping a chunked sequence into an additional lazy sequence which doesn’t solve anything

The "unchunk" would be used on a collection before handing it to an expensive lazy operation. But there's a reason there's no "unchunk" in core - if you need to control how many times an operation is performed you shouldn't be using laziness in the first place.

you can also use transducers to avoid laziness:

user=> (first (map prn [1 2 3 4 5]))
1
2
3
4
5
nil
user=> (into [] (comp (map prn) (take 1)) [1 2 3 4 5])
1
[nil]

it's not the transducers doing that though, it's using take before map

user=> (map prn (take 1 [1 2 3 4 5]))
(1
nil)

comp works in "reverse" order with transducers, so map is being done first in my example

My personal approach is not to use first/filter, but rather some

(some #(when (pred %) %) lazy-collection)

the problem with some, if I'm not mistaken, is that it returns the Boolean value returned by the predicate, not the value on which the predicate was called. The additional problem with the #(when (pred %) %) trick is that it cannot be used to find false nor nil in a sequence.

You could use transducers instead:

(transduce
 (comp
  (filter #(> % 10))
  (take 1))
 #(or %2 %1)
 (range 100))
;> 11

what's the connection to memoize that you've noticed. I do make heavy use of memoize, but I don't see the connection here.

This would be to avoid re-execution of the predicate when the value has already been seen. That way if the filter executes the memorized predicate the second time, it just becomes a map lookup and doesn’t execute the predicate code

ah, I sort of see what you mean. But I don't see how that would work in the global scope of the program. Are you suggesting that every predicate argument to filter should be a global function which is memoized for the duration of the program, or are you suggesting that I should just wrap the predicate argument of filter in a call to memoize? The former would be difficult as I often pass a closure to filter . The latter would be difficult because it would only have an effect if the predicate is called multiple times within the same call to filter.

silly demo:

(defn ^{:sweet 1} sour [])

(add-watch
  #'sour ::watch
  (fn [_k v _old _new]
    (print (-> v meta :sweet))
    ;; separate thread
    (future (println " .." (-> v meta :sweet)))))

(defn ^{:sweet 2} sour []) ;; =prints=> 1..2
(defn ^{:sweet 3} sour []) ;; =prints=> 2..3

looking at the Java classes, I would say no, but some non-hacky workarounds?

def executes all the watches before setting the metadata of the Var. I don't have much experience with reflection but maybe ASM (https://asm.ow2.io/asm4-guide.pdf) has a way to modify .setMeta of the Var instances. I am assuming you do not want to change any of the defn calls to something else so that the changes in metadata are detected without changing any code that changes the metadata.

Hmm.. maybe the Var could be proxied/reified instead of adding a watcher :thinking_face:

The problem with it is that the proxied Var object cannot be be placed into the current namespace object. It can be retrieved by using #'symbol-name . If Clojure or Java had a simple mechanism to advice existing methods (similar to Emacs Lisp's advice mechanism: https://www.gnu.org/software/emacs/manual/html_node/elisp/Advising-Functions.html ), we could advice the setMeta method and make it execute something every time it is called. The only option that might achieve this would be to modify the clojure.lang.Var class before it is loaded (or maybe it is also possible to modify it after it is loaded).

If I can shamelessly plug https://github.com/Tyruiop/needle (because I made it :p and because it's a ~simple defn replacement)

thank you, but it is not a tool for my case. We have memory leak on production in old complex system and I have to debug it.

I’ve used visualvm in the past for Scala. It’s pretty decent, though I’ve never tried it with clojure

I am watching YT and so far jprofiler looks like show information in simple way. But I think it is not free (still researching)

not for clojure particularly, but I have used thread dumps and https://www.eclipse.org/mat/ to pin point memory problems on java

Why eclipse vs visualvm?

The evil part of visualvm (and maybe other profiles) is thousands of:

Profiler Agent Warning: Cannot convert 0x7fc988c13320
Profiler Agent Warning: Cannot convert 0x7fc988c13328
Profiler Agent Warning: Cannot convert 0x7fc988c13330
Profiler Agent Warning: Cannot convert 0x7fc988c13338
Profiler Agent Warning: Cannot convert 0x7fc988c13318
Profiler Agent Warning: Cannot convert 0x7fc988c13320
Profiler Agent Warning: Cannot convert 0x7fc988c13328
Profiler Agent Warning: Cannot convert 0x7fc988c13330
Profiler Agent Warning: Cannot convert 0x7fc988c13338

YourKit, you won't regret it :) it's scary at first but you don't have to use every tab. You can grow in knowledge as-you-go 90% of the time I use the Threads tab, that one is a fairly simple UI

rationale: it's powerful, just works, doesn't look like a cheap Swing app

considering licences it is not free too, but I agree it can be good choice

BTW From youtube jprofiler was looking even better

but it is only first impression

every xmas they make a deal for 100 bucks

still expensive, I need it first time to fix memory leak in about 5-6 years of coding Clojure

:) well, each to his own... after you acquire one of these hammers you'll see nails more often there's hardly a week where YK doesn't help me in some way. TIL about jprofiler btw!

this week's: I could quickly visualize that a thread pool wasn't being stopped, so my Reloaded workflow was leaking threads on each (reset)

There is also XRebel worth to check

please let me know if you will do research about them

oh not atm, I feel quite well served it's good to know about them though!

for me all of them are new stuff and it is hard to judge for me

YK and JP look fairly similar in areas of focus. maybe JP looks a little bit easier? that will be subjective of course. You could simply try their demos and see which feels more intuitive

XRebel seems a little more tailor-made for webapps and microservices instead of being a generic JVM thing which is still interesting, but probably more than one needs for debugging a mem leak

I am trying jprofiler right now, but the main issue the out of memory exception is happening only on production

I can’t recreate it locally or even on preprod

I have plan to use XX:+HeapDumpOnOutOfMemoryError maybe

But it will be my first time

normally these can connect to a remote process

I have only used YourKit but it was very impressive and easy to use. I would recommend it. It also has a “15 day fully functional trial”. (I have no relationship to the company)

visualvm is a profiler, eclispse-mat is a memory (thread-dump) analyzer, the later can produce a "leak suspects" report from a thread-dump, I don't remember if it can connect to a running jvm. Also, OutOfMemory errors not necessarily implies a memory leak it may be that your current workload has increased, and well, your memory is not enough under current gc configuration. Monitoring jvm-memory spaces and app throughput/workload/rpm may hint this. Also you can look at the gc logs for hints of gc memory problems.

> “leak suspects” Can YourKit / jprofiler / XRebel do the same?

> OutOfMemory errors not necessarily implies a memory leak In this specific case we can observe increasing heap in time and constantly repeating OOM after a few days.

but still using core async?

yes

so it’s not impactful enough in your app or did you work around it by avoiding alts!?

Hard to say, the leak is actually subtle and needs some precise circumstances to trigger. When our core.async heavy app was initially deployed it was very rocky and needed a lot of work to stabilize for a variety of reasons. I am not sure how much this leak played into that, but as part of stabilizing the application we removed a lot usages of timer channels just going on the idea that timers are global and potentially holding references in some way.

I didn't make the connection with alts! until much later

We also have a sort of custom channel that only weakly references callbacks that is used strategically in a few places that may also make us less suspectable

this is very helpful, thank you!

Hi. I have some code like this:

(defonce buf (async/chan (async/sliding-buffer 1)))
(defonce running? (atom true))
...
(defonce
  pusher
  (async/go-loop [do-sleep false]
    (if @running?
      (do
        (when do-sleep
          (async/alts! [(async/timeout 200)]))
        (let [timeout (async/timeout 1000)
              [v ch] (async/alts! [buf timeout])]
          (if (nil? v)
            (recur false)
            (do
              (doseq [c @clients]
                (s/put! c (json/generate-string v)))
              (recur true)))))
      (log/info "exiting!"))))

I can't say for sure, but I think it is very unlikely your leak is related to that alts! (Using a loop in a go block like that mitigates parts of it, and you just have one instance of that loop, and you aren't closing over big chunks of memory)

Thanks

Which system? A java framework?

It's a CMS library called Bread https://github.com/breadsystems/bread-cms/

and it uses annotations to dispatch?

sorry, not talking about annotations in the reader sense, I'm just talking about extra runtime data

https://github.com/breadsystems/bread-cms/blob/main/src/systems/bread/alpha/core.cljc#L369

"annotate" = call with-meta/`ex-info`

you should prefer ex-info

unless you can represent failures as plain old data (maps)