You could make a function for this pretty easily.

(defn when-pred
  [val pred then]
  (if (pred val)
    (then val)
    val))

(defmacro if->
  [val sym test clause]
  `(when-pred ~val (fn [~sym] ~test) (fn [~sym] ~clause)))

(-> data
    fn1
    fn2
    (when-pred :needs-doing needs-doing)
    fn4)

(-> data
    fn1
    fn2
    (if-> val (= (:some-key val) :some-thing) (update val :some-key some-update))
    fn4)

is there a run! which accepts two elements? 🙂

I would just implement this using loop probably

makes sense!

thanks

almost. the nil? check would not be enough if the seq also can contain nils, so I would use:

(loop [xs (seq xs0)]
  (let [next-xs (next xs)
         last? (not next-xs)]
   (recur next-xs)

You could also do it with a transducer/eduction

(run! identity (eduction (fn [rf] (fn ([] (rf)) ([r] (prn 'done r) (rf r)) ([r i] (rf r i)))) (range 5)))

it is interesting idea with a transducer thanks!

Could you run! Over but-last and special case the last one?

seems like custom loop as simplier solution

thanks for the input

you could also pair each element with the next one and look for where the paired elements run out:

(let [data (range 5)]
    (run! (fn [[a b]] (if (= a ::last) (do (prn b) b) b)) (map vector (concat (drop 1 data) [::last]) data)))

thanks

can you provide more context around the code that did that?

--illegal-access=debug @U5QB8D59A add ^ to your JVM to get some better stacktrace

https://clojure.org/guides/faq#illegal_access

The actual code creating the issue is this: (->> cert (.getSubjectDN) (.getName) (drop 3) (apply str))] cert is an X509CertImpl pulled from an sslContext Solution is to use a different method but it asked to warn someone..so here I am.

So, I have fixed the issue in my code to not use a deprecated function.

unfortunately, because this is in code generated by the compiler on your code's behalf, it will always get reported as if it is Clojure ;)

Okies. Thanks

the issue is not actually one of deprecation, but rather calling into a function that you cannot access due to module visibility.

And it's true that switching from getSubjectDN to getSubjectX500Principal still gives an error

this being one of the internal sun jdk methods, you're not supposed to use this class directly

can you post the full stacktrace from --illegal-access=debug?

I'd like to know if it's the Reflector at fault or something else

I'll have a look

yeah, would be curious to see the code

probably should be using java.security.cert.X509Certificate ?

ok here is what I got for you: > WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000800232040 (file:/C:/Users/sweb/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method sun.security.x509.X509CertImpl.getSubjectX500Principal() > at clojure.lang.Reflector.canAccess(Reflector.java:49) > at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:156) > at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438) > at url_check.core$wildcard_QMARK_.invokeStatic(core.clj:70) > at url_check.core$wildcard_QMARK_.invoke(core.clj:67) > at url_check.core$eval8394.invokeStatic(form-init16933106916591026780.clj:205) > at url_check.core$eval8394.invoke(form-init16933106916591026780.clj:205) > at clojure.lang.Compiler.eval(Compiler.java:7177) > at clojure.lang.Compiler.eval(Compiler.java:7132) > at clojure.core$eval.invokeStatic(core.clj:3214) > at clojure.core$eval.invoke(core.clj:3210) > at clojure.main$repl$read_eval_print__9086$fn__9089.invoke(main.clj:437) > at clojure.main$repl$read_eval_print__9086.invoke(main.clj:437) > at clojure.main$repl$fn__9095.invoke(main.clj:458) > at clojure.main$repl.invokeStatic(main.clj:458) > at clojure.main$repl.doInvoke(main.clj:368) > at clojure.lang.RestFn.invoke(RestFn.java:1523) > at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:79) > at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:55) > at nrepl.middleware.interruptible_eval$interruptible_eval$fn__3452$fn__3456.invoke(interruptible_eval.clj:142) > at clojure.lang.AFn.run(AFn.java:22) > at nrepl.middleware.session$session_exec$main_loop__3553$fn__3557.invoke(session.clj:171) > at nrepl.middleware.session$session_exec$main_loop__3553.invoke(session.clj:170) > at clojure.lang.AFn.run(AFn.java:22) > at java.base/java.lang.Thread.run(Thread.java:834) > WARNING: Illegal reflective access by clojure.lang.Reflector (file:/C:/Users/sweb/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method sun.security.x509.X509CertImpl.getSubjectX500Principal() > at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) > at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438) > at url_check.core$wildcard_QMARK_.invokeStatic(core.clj:70) > at url_check.core$wildcard_QMARK_.invoke(core.clj:67) > at url_check.core$eval8394.invokeStatic(form-init16933106916591026780.clj:205) > at url_check.core$eval8394.invoke(form-init16933106916591026780.clj:205) > at clojure.lang.Compiler.eval(Compiler.java:7177) > at clojure.lang.Compiler.eval(Compiler.java:7132) > at clojure.core$eval.invokeStatic(core.clj:3214) > at clojure.core$eval.invoke(core.clj:3210) > at clojure.main$repl$read_eval_print__9086$fn__9089.invoke(main.clj:437) > at clojure.main$repl$read_eval_print__9086.invoke(main.clj:437) > at clojure.main$repl$fn__9095.invoke(main.clj:458) > at clojure.main$repl.invokeStatic(main.clj:458) > at clojure.main$repl.doInvoke(main.clj:368) > at clojure.lang.RestFn.invoke(RestFn.java:1523) > at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:79) > at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:55) > at nrepl.middleware.interruptible_eval$interruptible_eval$fn__3452$fn__3456.invoke(interruptible_eval.clj:142) > at clojure.lang.AFn.run(AFn.java:22) > at nrepl.middleware.session$session_exec$main_loop__3553$fn__3557.invoke(session.clj:171) > at nrepl.middleware.session$session_exec$main_loop__3553.invoke(session.clj:170) > at clojure.lang.AFn.run(AFn.java:22) > at java.base/java.lang.Thread.run(Thread.java:834)

seems like https://github.com/davidkeen/url-check-clojure/blob/master/src/url_check/core.clj

line #s don't seem right there though

indeed I am doing something along the same lines. I hadn't seen this repo before though

or maybe that's your code? I'm just guessing from the ns

ha, no it's not although we have the same ns.

ok, well what's your code at line 70?

he seems to be checking for a well formed response from a server. I am checking cookie compliance, form usage, cert validity (where I am running into the interop issues) etc

(let [n (->> cert (.getSubjectX500Principal) (.getName) (drop 3) (apply str))]

as per your faq, changing it to: > (let [n (->> ^java.security.cert.X509Certificate cert > (.getSubjectX500Principal) > (.getName) > (drop 3) > (apply str))] worked without error.

https://clojuredocs.org/clojure.java.shell/sh perhaps does what you need?

I like babashka/process https://github.com/babashka/process as the name implies, it's included in babashka

Thanks. I'll look at both of them!

For now clojure.ava.shell.sh is enough. It is synchronous though. But I am sure I can just use Thread to make it non-synchronous.

how are you starting the java process? if you want to download code from the internet and add it to your classpath, you may want to look at something like the add-lib branch of the clj tool (which is currently work in progress)

My Clojure program will be a jar 🙂

https://insideclojure.org/2018/05/04/add-lib/

Cool! Will keep my eyes open for updates on the tool 😎

yeah ... that's the bunny 😉

I've used this a little in the past too: https://github.com/pallet/alembic

for what it's worth: nREPL 0.7 got sideloader: #97: Added a sideloader, a network classloader that allows dependencies to be added even when the source/class files are not available on the server JVM's classpath (e.g. supplied by the client). https://github.com/nrepl/nrepl/releases (I haven't tried this feature myself)

Cool to know about 🙂

I guess. This would require me to learn lots more about tooling. Up to this day I've just cargo-culted and used luminus-recipes XD

Thanks, btw.

It seems like this will be really easy with the clj command-line tool :thumbsup:

Yea exactly, I was gonna say may not be as hard as you think

I had only worked with lein and that would be a bit more involved. Plus it would add another rather large dependency.

Clojure does produce classes

Perhaps with VisualVM: https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/heapdump.html

bash-5.0# jcmd 1 GC.class_histogram -all > GC.class_histogram.csv bash-5.0# head GC.class_histogram.csv 1: num #instances #bytes class name (module) ------------------------------------------------------- 1: 96854702 3099350464 clojure.lang.LazySeq 2: 64655045 2068961440 clojure.lang.Cons 3: 5122755 394906984 [Ljava.lang.Object; ([email protected]) 4: 3382334 157481032 [B ([email protected]) 5: 3323357 79760568 java.lang.String ([email protected]) 6: 1509001 60360040 clojure.lang.PersistentVector 7: 2059526 49428624 clojure.lang.PersistentVector$Node

hmm, this seems to indicate holding onto the head of lazy sequences

yes, and because clojure lacks types it's hard to know where this is comming from

in Java, with types you would see the name of the class somewhere there

@U011NGC5FFY did you try to force a GC before taking this data?

no

if you take a thread dump can you see something in the list of things that the process is doing that's a common code path that looks like it might be holding onto the head of something large?

I don't have one now

thanks

it's a bit difficult since it's hard to reproduce the performance issue that triggers the issue and the app runs remotely inside docker

what kind of things is it connected to? if it's selecting large amounts of data from a db, and trying to serve that up over http it might be calling doall on something before trying to convert it to an http response?

Can you dump the heap on out of memory exception and have it written to a mounted docker volume to be inspected later? https://www.baeldung.com/java-heap-dump-capture#automatically

thanks - the app does not throw memory exception - it starts lots of threads and it slows down to a halt

at the same time memory consumption is very high and these are related

but because there is no OOM it might not be a leak ?!

are you catching and logging/printing uncaught thread exceptions?

if there's a leak, you'll eventually get an OomException

which gc algorithm are you using?

have you tried the old strategy of doubling the heap size and seeing what happens over longer periods of time?

@U011NGC5FFY > it starts lots of threads can you see what the app is doing in those threads?

@U0P0TMEFJ default with java 11 - G1

@U04V15CAJ: I am working on that

@UGJE0MM0W: not sure how to reply

I think @UGJE0MM0W meant are you doing this: https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions ?

thanks @U0P0TMEFJ, will check

thanks for the feedback so far, I'll see what I can do further

if you've got multiple docker processes to play with, you could always have one start up with the serial collector, which is a bit easier to follow what's getting cleaned up, because it just stops the world rather than trying to let all the threads progress ... (just as an idea of trying to work out what's going on 😉 )

Yes that's what I was suggesting

the performance issue is difficult to trigger, the app uses ~ 12 - 23 GB of ram and needs a cache, db and queue - these make things difficult to debug

class histo indicates a 96 million entry seq -- at least

you could take a memory dump and examine any instance of the LazySeq, check its .s field

I will do that, thanks

your step 1 class_histo was the absolute correct thing to do

thank you, I found a very good presentation on the subject: https://www.youtube.com/watch?v=_gineh_HcoQ (shorter version available online someplace)

wow 3hrs 🙂

jcmd <pid> GC.heap_dump file_destination

thanks, I am putting some load on the system now (trying, I can't seem to be able 😄 ) .

https://github.com/clojure-goes-fast/clj-async-profiler. I found the flame graph quite useful in analysing both space and time issues.

this looks interesting, thanks rmcv

@U011NGC5FFY if you can download the heapdump, you can offline analyze it with https://visualvm.github.io/download.html

I just re-tested it out and it was easy to tell what's in the LazySeq

thanks. Will try it out. I don't lknow how useful it will be for me now since I can't seem to reproduce the issue

I think it's affraid 🙂

you definitely have a thing hanging onto a LazySeq -- it's absolutely clear

seems I can't create a heap dump - size is 0. tried with both jcmd and jmap - files have size 0

Because the system is stalled or wrong pid? I did it like this some time ago:

$ docker exec my-service ps
PID   USER     TIME   COMMAND
    1 java       0:00 /dev/init -- java -Xmx8g -Dcom.sun.management.jmxremote.rmi.port=19090 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=19090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=localhost -jar /my-jar.jar
    8 java       0:26 java -Xmx8g -Dcom.sun.management.jmxremote.rmi.port=19090 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=19090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=localhost -jar /my-jar.jar
  256 java       0:00 ps

# Make a heap dump for PID 8
$ docker exec my-service jcmd 8 GC.heap_dump /tmp/docker.hprof

and then:

docker cp my-service:/tmp/docker.hprof .

it works now, I changed the path

I see there is a -Jopt. Will read up on the possible flags I can send there.

that just lets you use any jvm option

But I thought one of them might be relative-to or something 🙂

Most (all?) of the ExecutorService implementations allow you to provide a ThreadFactory which would allow you to set an uncaught exception handler

The error handling strategy will depend on the use case. Generally, it's hard to recover from unexpected errors from arbitrary processes.

As far as using something like core.async, I would generally turn all "outputs" (including exceptions) into data. That can mean wrapping work in a try/catch

+1 on this, the ideal here is to start all your processes with core.async and then use alt to wait for the first one to fail and then exit on that.

regarding using deref to test future failure, you can use the second and third args to deref to time out and provide a placeholder value if the future isn't ready, then move to the next future

contrived example:

(cmd)user=> (def futs (map future-call (repeat 10 #(doto (rand-int 20000) (Thread/sleep)))))
#'user/futs
(cmd)user=> (do (doall (take-while #{:waiting} (map #(deref % 1000 :waiting) (cycle futs)))) (map #(deref % 1000 :waiting) futs))
(6165 :waiting 7143 1631 :waiting 1091 :waiting :waiting :waiting :waiting)

core.async/merge

(one way to do it)

Yeah, I can think of several good ways to get it done, just don't want to recreate someone's prior art needlessly

Manifold has its own version of let , IIRC.

ooh, that's very promising, thanks

for a sorted-set you can get better time complexity for a split based on the ordering if you use subseq