Clojurians Log v2

I think https://cljdoc.org/d/metosin/reitit/0.6.0/doc/basics/route-conflicts describes your problem and a couple of ways to ignore conflicting routes

Thanks a lot! It helps )

https://clojure.org/reference/reader#_symbols it is not officially supported. So even if it works now doesn't mean it will work forever.

Ok, best to avoid then. Thanks!

^void typehint in front of run should help

(reify AsyncRunnable
            (^void run [_# ^AsyncExecution execution]
              (~body execution)))

@delaguardo It helped, thank you. It’s interesting, because https://clojurians.slack.com/archives/C03S1KBA2/p1653670299984069 the approach was to remove typehint to make it work.

that was a bit different case https://clojurians.slack.com/archives/C03S1KBA2/p1653671315346139?thread_ts=1653670299.984069&cid=C03S1KBA2

I’m surprised that that mattered or helped

I think the #_ there is removing not just the type hint but the whole arg so I suspect you are reifying the wrong signature?

it is _#

Oh nvm, I totally misread that

The problem is the map produced in your code snippet is extremely hard to process any further. I'd prefer to pmap a named function to a seq of all endpoints, with that function doing all nescessary processing. So something like

(def data (pmap snipe endpoints))

(def endpoints (let [mensas #{"/mensa-erzbergerstrasse" "/mensa-schloss-gottesaue" "/cafeteria-moltkestrasse-30" "/mensa-moltke" "/mensa-am-adenauerring"}
                     days #{"/montag" "/dienstag" "/mittwoch" "/donnerstag" "/freitag"}
                     extensions #{".html"}
                     combinations (combo/cartesian-product mensas days extensions)]
                 (map (partial apply str) combinations)))

(defn snipe [endpoint]
  (->> endpoint
       (request)
       (s/select (s/class "aw-meal-category"))
       (map parse-metadata)))

this was just meant to show that i don't believe the "pmap double the execution time"

Fair point

I'll continue to look into it, should work eventually

The easiest way is to use pmap. But if you want to have even a modicum of control over the parallelism, you should use something else. Plenty of discussions here with multiple references if you search for "pmap".

Have a look at https://github.com/clj-commons/manifold it has deferreds and streams for async programming with a pretty clean api.

I'd say Manifold is an overkill in this situation, assuming there are no other places in that code where it could be used.

pmap does seem to work for 2 endpoints at least, I'll check if it works more generally as well

When scaled up, it counterintuitively slows down the program by a factor of 2

Not easily possible as it seems: https://stackoverflow.com/questions/24980014/can-i-make-a-fully-non-blocking-backend-application-with-http-kit-and-core-async

@port19 you can use a client based on java.net.http and supply a fixed thread pool (e.g. of two).

user=> (import 'java.util.concurrent.Executors)
java.util.concurrent.Executors
user=> (def executor (Executors/newFixedThreadPool 2))
#'user/executor
user=> (require '[babashka.http-client :as http])
nil
user=> (def client (http/client {:executor 1}))

and then make requests with that client + :async true

(http/get "" {:client client :async true})

httpkit should also support supplying your own executor

clj-http-lite should work for that probably, I may look into it

how are you using pmap ? it's not the ideal solution because you don't control the thread pool (and probably don't care about the order) but it shouldn't double the execution time. I would not bother with non blocking for 25 requests, threads are good enough

like for you 5 requests:

dev> (time (do (into {} (map (fn [[k v]] [k (client/get (str "" v))])
                             {:monday "/mensa-erzbergerstrasse/montag.html"
                              :tuesday "/mensa-erzbergerstrasse/dienstag.html"
                              :wednesday "/mensa-erzbergerstrasse/mittwoch.html"
                              :thursday "/mensa-erzbergerstrasse/donnerstag.html"
                              :friday "/mensa-erzbergerstrasse/freitag.html"}))
               nil))
"Elapsed time: 383.66711 msecs"
;; => nil
dev> (time (do (into {} (pmap (fn [[k v]] [k (client/get (str "" v))])
                             {:monday "/mensa-erzbergerstrasse/montag.html"
                              :tuesday "/mensa-erzbergerstrasse/dienstag.html"
                              :wednesday "/mensa-erzbergerstrasse/mittwoch.html"
                              :thursday "/mensa-erzbergerstrasse/donnerstag.html"
                              :friday "/mensa-erzbergerstrasse/freitag.html"}))
               nil))
"Elapsed time: 80.316591 msecs"
;; => nil

The issue was solved by adding JVM option -XX:MaxMetaspaceSize=512m. Turns out, code generated by (eval) doesn't have a GC root. So when we limit the metaspace, its GC is triggered as expected.

> I know that one option is to try to rewrite the code without (eval), You could consider using #sci which doesn't have this problem

@shkertik say more about what you're doing, what is being evaled?

@ghadi we are creating some filtering functions with it. As I said, it could be rewritten without (eval). But rather than a big rewrite, a much better solution would be to add a couple lines that allow (eval) generated classes to be unloaded.

You can replace eval with:

(sci/eval-form (sci/init {}) the-form)

(sci/eval-string the-input-string)

(def ctx (sci/init {:namespaces {' {'your-fn your-fn}}}))
(sci/eval-form ctx the-form)

@borkdude thanks, I'll try that.

before making changes, I'd try to show an example. Classes that are not referenced can be garbage collected

are you defining vars ? I'm curious if something like that could cleanup stuff:

(defn isolated-eval [form]
  (let [tmp-ns (gensym)]
    (create-ns tmp-ns)
    (try (binding [*ns* (the-ns tmp-ns)] (eval form))
         (finally (remove-ns tmp-ns)))))

or maybe it makes things worse

you're not using memoize by any chance? which could prevent garbage collection

From what I see, there are tons of instances of Clojure's DynamicClassLoader. Classes cannot be unloaded unless their class loader is garbage collected. So the question here is how can we dereference these loaders.

right

No memoize anywhere.

i'm surprised. From my understanding we are creating a new classloader everytime we eval, but classes generated by that classloader can still be garbage collected, so it's only a small memory leak. Otherwise I would just blow my repl's memory in a few hours no ?

If you have lots of requests with "eval"s, this may be interesting too:

user=> (time (dotimes [i 100000] (eval '(fn []))))
"Elapsed time: 17816.421042 msecs"
nil
user=> (time (dotimes [i 100000] (sci/eval-form (sci/init {}) '(fn []))))
"Elapsed time: 1021.765 msecs"
nil

@rolthiolliere Well, somehow our code generates thousands of classloader instances each minute. And it results in a slow metaspace usage creep, takes many hours to blow up. You can't really get this situation from typical REPL usage.

oh okay i misread, I though you were using eval once per inute and a single eval was generating 3.5k classes

paste some code, otherwise it's like playing 20 questions

people are throwing solutions at you without the specific problem being diagnosed

AFAIK, eval creates a new class loader, and those are not GC'ed in some cases. Have stumbled upon this before: https://stackoverflow.com/questions/71447267/clojure-memory-leaks-using-eval

It depends on how you published it. using deps.edn or lein?

deps.edn and deps-deploy

I see. I think this will help: https://github.com/clojure/tools.build/blob/f096376cd4e301dbf90ce447fee901397d3490c0/src/main/clojure/clojure/tools/build/api.clj#L383

example: https://github.com/nextjournal/clerk/blob/86fd28d119d3488ba6f8a1d959c98442ea3d7856/build.clj#L31

Thanks! I'll give that a shot

Another approach is to have a "template" pom.xml file with most of the data you need https://github.com/seancorfield/next-jdbc/blob/develop/template/pom.xml#L18-L22 and then in your build.clj specify :scm { :tag ... } and :src-pom https://github.com/seancorfield/next-jdbc/blob/develop/build.clj#L40-L48

(I prefer that approach in my projects so that build.clj is simpler and I can have as much static detail in pom.xml as I want)

How do I get a version tag for a snapshot version?

https://github.com/seancorfield/next-jdbc/blob/develop/build.clj#L20-L21

and https://github.com/seancorfield/next-jdbc/blob/develop/build.clj#L39

So I tell the build task whether I'm building a release or a snapshot... And that's all done automatically in GitHub Actions...

For a release workflow (after creating a release on GH): https://github.com/seancorfield/next-jdbc/blob/develop/.github/workflows/test-and-release.yml#L50

For a snapshot workflow (after pushing/merging any commits to the main branch): https://github.com/seancorfield/next-jdbc/blob/develop/.github/workflows/test-and-snapshot.yml#L48

That repository has an invalid git tree for snapshots https://github.com/seancorfield/next-jdbc/tree/v1.3.999-SNAPSHOT

Yes, true, because I don't tag the repo for snapshots

(because every commit produces a release)

why do you even push a snapsnot release to clojars? git deps ftw ;)

Some of my users still use Leiningen 😛

(and Java compilation in some cases I guess)

(although we can solve that using prep-deps :))

@eoogbe You could put the SHA in :scm > :tag for snapshots... I just haven't bothered yet

SNAPSHOT isn't really the way to go. Just publish an -alpha0 release if it's really preview

or 0.1.0, fine too

SNAPSHOT builds are optional and not reproducible -- you'll get whatever is the latest snapshot JAR at the time of the request.

I do it just so folks can test the very latest versions from Clojars if they want (I checked with the Clojars maintainers first to ensure that pushing a SNAPSHOT for every commit was acceptable to them).

I push SNAPSHOTs on every commit for clj-kondo too and you can get a reproducible version from that, clojure-lsp uses this all the time

True, if you use the full date/time stamp version that each snapshot resolves to...

yes

Sending clj-kondo/clj-kondo/2023.04.15-SNAPSHOT/clj-kondo-2023.04.15-20230418.173453-3.jar (361k)
    to 

It worked! tysm

Because it's better to be explicit about the local bindings you are introducing...

hmm. I think I can accept that.

(let [{:keys nil} {:if 1 :let 2 :do 3}] (if true (do (println "wat?") let))

(that's mostly a joke, but the point I'm making is that a hash map could inject arbitrary symbols and change the meaning of your code in unexpected ways)

another reason is that the compiler cannot know statically what is given dynamically

so if you would type:

(let [{:keys-all true} x]
  (assoc whatever-yo :foo :bar))

Heh, I remember reviewing some Python code where its author mutated locals(). It was impossible to make sense of, and the code wasn't working. The author claimed that there just wasn't any other way it could've been written without changing locals(). Needless to say, the author was wrong and the code without locals() ended up being drastically simpler.

ok

so if you were going to have that it would be better (though not necessarily good) in static type land

in static typing this would be easier to implement, but I haven't encountered any languages that have this

Try :refer :all for a taste of how awful this quickly becomes 😅

yeah i should follow the rules and say what is motivating this q

I have a multimethod and there's a couple bindings that appear at the top of every or almost every method implementation

trying to factor this out

you could do this using a macro (but even then I'd probably avoid it)

yeah that doesn't seem appropriate to me fwiw

and there's no way to access the dispatching value from within the method, right?

hmm

that seems like itwould be another bad idea

you could call the dispatch function on the this argument to get it?

Well, you can define the dispatch function seperately

just call it again

sure

you can also just change the part that is the multimethod (dependent on who the consumers are)

so lets say you have a multimethod like this

(defmulti do-thing #(get-in % [:metadata :type]))

(defn f [x]
  (get-in x [:something :age]))
(defn g [x]
  (get-in x [:whatever :abc]))
(defmethod do-thing :apple
  [apple]
  (let [a (f apple)
        b (g apple)]))

(defmethod do-thing :pear
  [pear]
  (let [a (f pear)
        b (g pear)]))

if that let [a ... b ...] part is something you want extracted you can always just change the multimethod part to be further down

(defmulti do-thing*
          (fn [obj & _] 
            (get-in obj [:metadata :type])))

(defmethod do-thing* :apple
  [apple a b])

(defmethod do-thing* :pear
  [pear a b])

(defn do-thing 
  [fruit]
  (do-thing* fruit (f fruit) (g fruit)))

if that makes sense

really depends on why you are using a multimethod though

I see what you mean

and if you actually expect it to be extended

there are other options too

but without going insane, they are all just around moving where and how dispatch and extraction happen

I think the approach you describe suits my needs;thankss for taking the time @emccue