This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-08-10
Channels
- # announcements (1)
- # babashka (18)
- # beginners (122)
- # calva (18)
- # cider (3)
- # cljs-dev (2)
- # cljsrn (3)
- # clojure (102)
- # clojure-europe (15)
- # clojure-france (2)
- # clojure-nl (1)
- # clojure-portugal (1)
- # clojure-spec (3)
- # clojure-uk (8)
- # clojurescript (46)
- # clojureverse-ops (5)
- # code-reviews (1)
- # conjure (2)
- # cursive (15)
- # datalog (13)
- # datomic (18)
- # emacs (4)
- # fulcro (8)
- # helix (8)
- # instaparse (1)
- # introduce-yourself (2)
- # jobs (4)
- # leiningen (23)
- # lsp (26)
- # malli (21)
- # off-topic (34)
- # pedestal (21)
- # polylith (6)
- # reitit (5)
- # remote-jobs (3)
- # schema (1)
- # sci (8)
- # shadow-cljs (8)
- # spacemacs (3)
- # sql (30)
- # testing (31)
- # tools-deps (21)
- # vim (25)
- # xtdb (8)
do you tap> users have multiple monitors? what value does it have over cider-inspect?
@tessocog I only have one monitor (27"): I have VS Code 3/4 screen on the left and Reveal 1/4 screen on the right. The value of tap>
is that it can stay in the code all the time, including in production code, and you can attach (and remove) listeners for it any time you want -- and in particular you can attach multiple listeners at different times.
For example, you could have a plain prn
attached to see values at the console, you can have Reveal attached, you can have Cognitect's REBL attached, you can have Portal attached -- in any combination you want, at any time.
ofc if directly interacting with code, cider-inspect buffer will always present the last evaluation
Also, cider-inspect kind of requires you're using Emacs 🙂 I switched away from Emacs some years ago (after using it on and off for nearly two decades).
I'm using leiningen and want's to create a minimal html file containing git head sha before the uberjar is bundled.
So my idea was to create the resources/version.html
file as here: https://gist.github.com/gleenn/a8a604ebfa807bcd2a8c3341b3573657
But that doesn't seem to work - the problem might be that :injections
aren't executed for jar/uberjar: https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L273-L276
;; Forms to prepend to every form that is evaluated inside your project.
;; Allows working around the Gilardi Scenario:
;; Note: This code is not executed in jars or uberjars.
:injections [(require 'clojure.pprint)]
I don't like :injections much
you can run arbitrary code in a project.clj by placing it before defproject
, https://github.com/clojure-emacs/orchard/blob/master/project.clj shows this particularly
Interesting, thanks. I ended up using a shell script which is run by Make before lein uberjar
I just discovered a gotcha in clojure which was different than I had supposed for a long time. (first (filter pred lazy-collection))
might call pred
again after the first time it has returned true
. This could have an unfortunate effect on performance if pred
has a high computational complexity.
To remedy this, I am refactoring 10 to 20 call-sites in my application to avoid this dangerous pattern. The approach I have chosen (for the moment) is a scheme-like continuation-passing-style (CPS) pattern. Given a predicate, a collection, and an if-found function, call the if-found function on the first item of the collection which satisfies the predicate, and return a call-site-given return value if no such element exists. The functional version of this is somewhat awkward, but could be made more beautiful with some macrology if necessary.
(defn call-with-found
"Call the given predicate, pred, on successive element of the collection
until the first time pred returns a truthy value, at which time if-found
is called with that element of the collection, and call-with-found returns
the return value of if-found. If no such element of collection is found
(including if collection is empty) then the value if-not-found (defaulting
to false) is returned."
([pred coll & {:keys [if-found if-not-found]
:or {if-found (constantly true)
if-not-found false}}]
(reduce (fn [_ item]
(if (pred item)
(reduced (if-found item))
if-not-found))
if-not-found
coll)))
Here is an example of the call-site.
(call-with-found (fn [x] (dual-combination? this x)) operands
:if-not-found this
:if-found (fn [dual-td]
(create-dual this (map (fn [y]
(create this (map (fn [x]
(if (= x dual-td)
y x))
operands)))
(rest dual-td)))))
previously
(let [[dual-td & _ :as tds] (filter (fn [x] (dual-combination? this x))
operands)]
(if (empty? tds)
this
(create-dual this (map (fn [y]
(create this (map (fn [x]
(if (= x dual-td)
y x))
operands)))
(rest dual-td))))))
do you know why pred is been called after first true?
if it is about lazy-collection been chunked by default then have a look at https://clojuredocs.org/clojure.core/chunk#example-5c9cebc3e4b0ca44402ef6ec
(defn re-chunk [n xs]
(lazy-seq
(when-let [s (seq (take n xs))]
(let [cb (chunk-buffer n)]
(doseq [x s] (chunk-append cb x))
(chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))
(first
(filter (fn [l]
(prn l)
(even? l))
(re-chunk 1 (range 1 1000))))
prints only 2 items instead of 32I don't in fact know that it is. However, I read a response from alexmiller saying that clojure does not guarantee that this pattern halts the traversal when the predicate returns true.
wrt unchuck, I found the following function in my odds-and-ends utils.clj
file.
;; code thanks to
(defn unchunk [s]
(when (seq s)
(lazy-seq
(cons (first s)
(unchunk (next s))))))
although this link no longer works for me, so I'm not really 100% what it is doing. However, it looks very much like it might be useful here.I am finding many places in my code base where I'm depending on my mis-understanding of filter
(defn conversion-C12
"AXBC + !X = ABC + !X"
[self]
(let [combos (filter gns/combo? (operands self))
duals (setof [td combos] (dual-combination? self td))
comp (filter (fn [n]
(and (gns/not? n)
(exists [td duals]
(member (operand n) (operands td)))))
(operands self))]
(if (empty? comp)
self
(letfn [(f [td]
(cond (not (gns/combo? td))
td
(not (dual-combination? self td))
td
:else
(create td (remove-element (operand (first comp))
(operands td)))))]
(create self (map f (operands self)))))))
Why would that unchunk work at all? If s
is chunked lazy sequence it will still load 32 elements when first s
is called
you are wrapping a chunked sequence into an additional lazy sequence which doesn’t solve anything
The "unchunk" would be used on a collection before handing it to an expensive lazy operation. But there's a reason there's no "unchunk" in core - if you need to control how many times an operation is performed you shouldn't be using laziness in the first place.
you can also use transducers to avoid laziness:
user=> (first (map prn [1 2 3 4 5]))
1
2
3
4
5
nil
user=> (into [] (comp (map prn) (take 1)) [1 2 3 4 5])
1
[nil]
it's not the transducers doing that though, it's using take before map
user=> (map prn (take 1 [1 2 3 4 5]))
(1
nil)
comp
works in "reverse" order with transducers, so map
is being done first in my example
My personal approach is not to use first/filter, but rather some
(some #(when (pred %) %) lazy-collection)
this only calls pred
on each element once, and does a recur
on the next
when the predicate returns falsey, so it’s not lazy.
The bonus of some
is that a lot of predicates return the value being tested when they pass, so the when
wrapper isn’t neededthe problem with some, if I'm not mistaken, is that it returns the Boolean value returned by the predicate, not the value on which the predicate was called. The additional problem with the #(when (pred %) %)
trick is that it cannot be used to find false
nor nil
in a sequence.
You could use transducers instead:
(transduce
(comp
(filter #(> % 10))
(take 1))
#(or %2 %1)
(range 100))
;> 11
The function call-with-found
uses reduce
to efficiently iterate through the collection making no assumption about whether it is list-like or vector-like, and uses reduced
to terminate the iteration when finished.
@jimka.issy Is this perhaps a case where memoize
can prove useful?
what's the connection to memoize
that you've noticed. I do make heavy use of memoize
, but I don't see the connection here.
This would be to avoid re-execution of the predicate when the value has already been seen. That way if the filter executes the memorized predicate the second time, it just becomes a map lookup and doesn’t execute the predicate code
ah, I sort of see what you mean. But I don't see how that would work in the global scope of the program. Are you suggesting that every predicate argument to filter
should be a global function which is memoized for the duration of the program, or are you suggesting that I should just wrap the predicate argument of filter
in a call to memoize
? The former would be difficult as I often pass a closure to filter
. The latter would be difficult because it would only have an effect if the predicate is called multiple times within the same call to filter
.
Is there a way to capture Var meta-data changes? add-watch
sees the Var before the meta-data has been changed.
silly demo:
(defn ^{:sweet 1} sour [])
(add-watch
#'sour ::watch
(fn [_k v _old _new]
(print (-> v meta :sweet))
;; separate thread
(future (println " .." (-> v meta :sweet)))))
(defn ^{:sweet 2} sour []) ;; =prints=> 1..2
(defn ^{:sweet 3} sour []) ;; =prints=> 2..3
def
executes all the watches before setting the metadata of the Var. I don't have much experience with reflection but maybe ASM (https://asm.ow2.io/asm4-guide.pdf) has a way to modify .setMeta
of the Var instances.
I am assuming you do not want to change any of the defn
calls to something else so that the changes in metadata are detected without changing any code that changes the metadata.
Hmm.. maybe the Var could be proxied/reified instead of adding a watcher :thinking_face:
The problem with it is that the proxied Var object cannot be be placed into the current namespace object. It can be retrieved by using #'symbol-name
. If Clojure or Java had a simple mechanism to advice
existing methods (similar to Emacs Lisp's advice mechanism: https://www.gnu.org/software/emacs/manual/html_node/elisp/Advising-Functions.html ), we could advice the setMeta
method and make it execute something every time it is called.
The only option that might achieve this would be to modify the clojure.lang.Var
class before it is loaded (or maybe it is also possible to modify it after it is loaded).
What java profiler do you recommend to use today for Clojure? Do you feel some of them are simpler (more efficient) than others? Which one do you use and why? Why not others?
If I can shamelessly plug https://github.com/Tyruiop/needle (because I made it :p and because it's a ~simple defn
replacement)
thank you, but it is not a tool for my case. We have memory leak on production in old complex system and I have to debug it.
I’ve used visualvm in the past for Scala. It’s pretty decent, though I’ve never tried it with clojure
I am watching YT and so far jprofiler looks like show information in simple way. But I think it is not free (still researching)
not for clojure particularly, but I have used thread dumps and https://www.eclipse.org/mat/ to pin point memory problems on java
The evil part of visualvm (and maybe other profiles) is thousands of:
Profiler Agent Warning: Cannot convert 0x7fc988c13320
Profiler Agent Warning: Cannot convert 0x7fc988c13328
Profiler Agent Warning: Cannot convert 0x7fc988c13330
Profiler Agent Warning: Cannot convert 0x7fc988c13338
Profiler Agent Warning: Cannot convert 0x7fc988c13318
Profiler Agent Warning: Cannot convert 0x7fc988c13320
Profiler Agent Warning: Cannot convert 0x7fc988c13328
Profiler Agent Warning: Cannot convert 0x7fc988c13330
Profiler Agent Warning: Cannot convert 0x7fc988c13338
which make me to not trust resultsYourKit, you won't regret it :)
it's scary at first but you don't have to use every tab. You can grow in knowledge as-you-go
90% of the time I use the Threads
tab, that one is a fairly simple UI
still expensive, I need it first time to fix memory leak in about 5-6 years of coding Clojure
:) well, each to his own... after you acquire one of these hammers you'll see nails more often there's hardly a week where YK doesn't help me in some way. TIL about jprofiler btw!
this week's: I could quickly visualize that a thread pool wasn't being stopped, so my Reloaded workflow was leaking threads on each (reset)
YK and JP look fairly similar in areas of focus. maybe JP looks a little bit easier? that will be subjective of course. You could simply try their demos and see which feels more intuitive
XRebel seems a little more tailor-made for webapps and microservices instead of being a generic JVM thing which is still interesting, but probably more than one needs for debugging a mem leak
I am trying jprofiler right now, but the main issue the out of memory exception is happening only on production
I have only used YourKit but it was very impressive and easy to use. I would recommend it. It also has a “15 day fully functional trial”. (I have no relationship to the company)
visualvm is a profiler, eclispse-mat is a memory (thread-dump) analyzer, the later can produce a "leak suspects" report from a thread-dump, I don't remember if it can connect to a running jvm. Also, OutOfMemory errors not necessarily implies a memory leak it may be that your current workload has increased, and well, your memory is not enough under current gc configuration. Monitoring jvm-memory spaces and app throughput/workload/rpm may hint this. Also you can look at the gc logs for hints of gc memory problems.
> OutOfMemory errors not necessarily implies a memory leak In this specific case we can observe increasing heap in time and constantly repeating OOM after a few days.
Ok to figure out out of memory issue I think the best will be to first prepare app for which one I can control leak memory intentionally. Do you know good sophisticated memory leak code example in Clojure? Just something more tricky to debug, than “add to array”.
Hard to say, the leak is actually subtle and needs some precise circumstances to trigger. When our core.async heavy app was initially deployed it was very rocky and needed a lot of work to stabilize for a variety of reasons. I am not sure how much this leak played into that, but as part of stabilizing the application we removed a lot usages of timer channels just going on the idea that timers are global and potentially holding references in some way.
We also have a sort of custom channel that only weakly references callbacks that is used strategically in a few places that may also make us less suspectable
Hi. I have some code like this:
(defonce buf (async/chan (async/sliding-buffer 1)))
(defonce running? (atom true))
...
(defonce
pusher
(async/go-loop [do-sleep false]
(if @running?
(do
(when do-sleep
(async/alts! [(async/timeout 200)]))
(let [timeout (async/timeout 1000)
[v ch] (async/alts! [buf timeout])]
(if (nil? v)
(recur false)
(do
(doseq [c @clients]
(s/put! c (json/generate-string v)))
(recur true)))))
(log/info "exiting!"))))
and very often (async/>!! buf new-val)
is done.
Would this code (`pusher`) trigger a memory leak?
Edit: From this issue I think yes, but I am not sure. I think that the timeout channels will "hang" around (for 1s) and potentially grow and grow (if there is enough traffic).
I'm using org.clojure/core.async "0.3.443"
included by yada/async "1.2.16"
..
I've only recently added this code, and only recently experienced OOM (I think!) in production. And OOM only happens after some time in prod, and not in stage, it seems.I can't say for sure, but I think it is very unlikely your leak is related to that alts! (Using a loop in a go block like that mitigates parts of it, and you just have one instance of that loop, and you aren't closing over big chunks of memory)
Thanks
I'd like to annotate any Throwable
s thrown from a certain context with some additional context-specific info. This seems like a job for metadata. However, Throwables don't appear to implement IObj
:
(with-meta (Exception. "oh no") {:extra :context})
; (err) class java.lang.Exception cannot be cast to class clojure.lang.IObj
I could implement IObj
(there's only one method: withMeta()
) for Throwables, but would it be more idiomatic to throw an (ex-info ...)
wrapping the original Throwable? It seems cleaner to just add stuff to the original but there's no obvious way to do that with the https://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html. Lack of arbitrary annotations for errors seems like the central shortcoming that ex-info
and ex-data
were designed to solve, but I want to be sure before I add a bunch of indirection...you could implement withMeta
perhaps, but you couldn't really make it work correctly right?
This sound a little like chained exceptions, in the java world
Perhaps (ex-info "oh no!" {:key :val} original-exception)
would serve?
Ah, As you already said. That seems most idiomatic.
What are your indirection concerns?
We generally just wrap exceptions in ex-info
whenever we want to convey more information. They are also handled "better" by certain runners if they are otherwise unhandled (e.g., exec function execution prints a neat error message for ex-info
but a complete stacktrace for other exceptions).
Okay, that makes sense. No specific concerns with indirection, just wanted to make sure I was on the right track/not overthinking things. Thanks all.
this system has a hooks API so I want to annotate errors thrown inside hook callbacks with info about the hook being dispatched
It's a CMS library called Bread https://github.com/breadsystems/bread-cms/
sorry, not talking about annotations in the reader sense, I'm just talking about extra runtime data
https://github.com/breadsystems/bread-cms/blob/main/src/systems/bread/alpha/core.cljc#L369
does it make sense to use spec for separating part of data that conforms to a spec? say
(s/MATCH!!! spec mycollection)
(s/filter (s/valid? spec %) mycollection)