And then it seems to work anyway, but maybe coverage reporting is off, not sure. I'm reporting this because I didn't get this before (Clojure 1.11.1), so I figured you might want to know.

Did you read the latest announcements regarding param tags? Looks like you need to type hint your parse int or specify param tags

I think cloverage drops metadata and type hints when it instruments your code. This used to just have a performance penalty but was generally fine as cloverage did some useful stuff. But now missing type hints are compile errors. And that’s what you are seeing here

Can you put this on https://ask.clojure.org ? Those Integer/parseInt calls are reflective but should still be valid so not sure why you are seeing this. Our intent was not to change anything about that. /cc @U050WRF8X

I’m out in the wild at the moment but a quick scan of cloverage gives me a hint where the problem lies. In pre-1.12alpha days static method calls were macroexpanded into dot calls but we no longer do that. cloverage wraps dot forms differently than function calls and so I expect that instrument is changing something about the call to parseInt that makes it fail. It’s that “something” that needs more investigation.

A little closer look hints that cloverage wraps things in invoke position with partial. This makes the call Integer/parseInt a value position affair meaning that we can’t infer the static call without param-tags thus the error. We’re noodling idea around error conditions for next alpha but for now cloverage probably needs changes around how it handles qualified method calls in invoke position.

Ok got it. So not needed to put it on http://ask.clojure.org I guess?

Maybe not. I’m reading cloverage code on my phone at the moment so I’m willing to admit that I may have misunderstood what’s happening. I’ll look closer when I get in front of a computer 😀

Sure thing, thank you for your responses. I also made a ticket on the cloverage github to make sure they are aware.

https://github.com/cloverage/cloverage/issues/345

If you have a spare moment it would be interesting to know if adding ^[String] param-tag in front of those calls makes the error go away. If not then they may have a further metadata propagation problem to fix also.

I mentioned it dropping metadata / type hints because we fail builds if they have reflection warnings. But when cloverage runs it has tons of reflection warnings. So i assume that it’s instrumentation fails to propagate them in some way. But that’s my only evidence or source for that claim. So don’t treat it as gospel

Understood

If you have a spare moment it would be interesting to know if adding ^[String] param-tag in front of those calls makes the error go away.Sure, but then I think I need a build of the clojure compiler executable 1.12.0-alpha8 I think? Can I download that somewhere? Because now I get:

Execution error (ExceptionInfo) at clojure.tools.reader.impl.errors/throw-ex (errors.clj:34).
[line 112, col 38] Metadata cannot be [String]. Metadata must be a Symbol, Keyword, String or Map.

Oh that’s needing a new tools.reader - we have that queued up but not released yet

You can use ^{:param-tags [String]} instead

That seems to work. By annotating all occurrences of parseLong I got the number of these errors down from 178 to 165. Still a way to go though 😅

In other words: lots of new annotations needed for this. Not necessarily a problem, but I'm also not sure if this is desirable, but fortunately it's not my decision 😉

In terms of readability I like the version without the annotations better, e.g.

(dec (Math/round (* 600 0.6))) [12 00] :low-critical
      (dec (Math/round (* 1400 0.6))) [18 00] :low-critical
      (dec (Math/round (* 3000 0.6))) [24 00] :low-critical

      (Math/round (* 600 0.6)) [12 00] :low-warning
      (Math/round (* 1400 0.6)) [18 00] :low-warning
      (Math/round (* 3000 0.6)) [23 59] :low-warning

(dec (^[double] Math/round (* 600 0.6))) [12 00] :low-critical
      (dec (^[double] Math/round (* 1400 0.6))) [18 00] :low-critical
      (dec (^[double] Math/round (* 3000 0.6))) [24 00] :low-critical

      (^[double] Math/round (* 600 0.6)) [12 00] :low-warning
      (^[double] Math/round (* 1400 0.6)) [18 00] :low-warning
      (^[double] Math/round (* 3000 0.6)) [23 59] :low-warning

you can capture the method as a value now though to remove the redundancy. (def round ^[double] Math/round) and then use that repeatedly

Or just use clojure.math/round :)

that doall is all I've personally ever seen

you could use transducers since those are eager

I guess I’m kind’a wishing for a map! 🙂

like run! but not 🙂

somehow I never have a need to do side effects in the middle of coll processing…

suppose you could define that yourself as (def map! (comp doall map))?

@U47G49KHQ I see that I need it less and less, and I’d argue/agree to that it is a code smell, and as such it shouldn’t be facilitated.

mapv is significantly faster than (doall (map ...))

oh?? thats not what i would have expected at all (not to say i've done experiments, just goes against my assumption that vectorifying the data would be "extra")

I would probably use transducers in that situation

here's the results from my 2018 macbook pro running clojure 1.11:

(doall (map inc nil)) => 30 ns
(doall (map inc (range 1000))) => 32 us
(doall (map inc (vec (range 1000)))) => 32 us

(into [] (map inc) nil) => 100 ns
(into [] (map inc) (range 1000)) => 23 us
(into [] (map inc) (vec (range 1000))) => 23 us

(mapv inc nil) => 86 ns
(mapv inc (range 1000)) => 15 us
(mapv inc (vec (range 1000))) => 15 us

(timed with criterium, and the ranges and vec ranges were pre-generated)

we use transducers to do side-effecting work such as sending records to kafka

mapv is also eager too, no? that could explain the performance difference

nvm, I think into would force eagerness as well. mapv most likely is doing less type reflection in exchange for being less general than map

Was interested in allocation per call, so I also ran Noah’s examples with time+:

(time+ (doall (map inc nil))) ;; alloc: 57b
  (time+ (doall (map inc (range 1000)))) ;; alloc: 111,643b
  (time+ (doall (map inc (vec (range 1000))))) ;; alloc: 115,868b

  (time+ (into [] (map inc) nil)) ;; alloc: 491b
  (time+ (into [] (map inc) (range 1000))) ;; alloc: 48,971b
  (time+ (into [] (map inc) (vec (range 1000)))) ;; alloc: 55,227b

  (time+ (mapv inc nil)) ;; alloc: 443b
  (time+ (mapv inc (range 1000))) ;;  alloc: 49,318b
  (time+ (mapv inc (vec (range 1000)))) ;; alloc: 54,429b

Just to get that right: The idiomatic transducer version of the original question

(->> stuff (map side-effecting-fn) (filter whatever?) ...)

(into [] (comp (map side-effecting-fn) (filter whatever?) ...) stuff)

see :watch-dir https://shadow-cljs.github.io/docs/UsersGuide.html#_css_reloading

What is that path supposed to be relative to? I tried that but it didn't seem to change anything

I tried relative to the "resources" directly and relative to the project root?

https://code.thheller.com/blog/shadow-cljs/2021/05/13/paths-paths-paths.html

see second part

from the layout of the project I suspect :watch-dir "resources/public" would do it

maybe :watch-dir "resources/public/electric_starter_app". I don't really know what you access over http and where the actual css file is

Yeah it's inside resources/public/electric_starter_app I'll try that an take a look

That worked, thank you!

Unfortunately, just using docker/containers isn’t a solution that will work for near-term. I actually have sandboxed classloaders, but dynamically loading clojure “against” one of these classloaders isn’t working.

if you can, don't do this, it is really tricky to get right, and even when you think you have it working you'll run into weird issues where classes loaded on a different thread use the wrong runtime, etc

lein had an eval in classloader feature (maybe still does), and polylith's test running tries to do this by default (we don't use the default at work exactly because it would break by using the wrong clojure runtime when loading classes on certain jvm built in threadpools)

https://github.com/polyfy/polylith/blob/3b37d2f58e6dfb2be369d8121940a163d0cdd106/components/common/src/polylith/clj/core/common/class_loader.clj#L45 is polylith's eval in classloader thing (which as written is just for running tests and the isolation is known to break under the right conditions)

I definitely hear the warning, thanks for the relevant pointer.

the thing is, the jvm won't load a class until it needs to, and having loaded the class won't initialize static fields until it needs to, and on a single thread you can sort of corral it to use the classloader you want using the context classloader stuff

but at work we use the build in java websocket client, which by default runs callbacks on a java provided thread pool, and what would happen is the static data in those callbacks would stored in the bytecode as a big string that clojure would turn back into data using read-string, and it would end up using read-string from the wrong runtime, and you would get fun errors about not being able to cast PersistentMap to PersistentMap

Yes, I’m definitely familiar with the cross-CL casting issues 😞

@U0NCTKEV8 Is there potentially an issue with AOT initial load and then attempting to dynamically load Clojure “source”? For example a gen-class/aot compiled code that depends on a clojure namespace like (ns my.pkg) (defn x [] 42) and then dynamically loading that later (load-string)? I suspect the original aot class is no longer dynamic, as if it has it’s own “copy” of that namespace/functions.

it is sort of complicated, you can aot compile with a flag called direct linking turned on, which will make that so, but generally no

if you aot compile some code, load it, and then redef the vars and stuff it will see new definitions of vars

re-def as in what would happen if you type def something in the repl and then def it again to a different value

(with-redefs does the same thing, it just changes the value back afterwards)

Polylith has an interesting line (binding [*use-context-classloader* false] Whereas the Clojure code seems to subvert that regardless

Var.pushThreadBindings(RT.map(USE_CONTEXT_CLASSLOADER, RT.T));
;; *use-context-classloader* is always bound to true regarldess of any 
;; outer binding setting and thus ignored.

Am I going to need to figure out the DynamicClassLoader instance that was used in the “re-def” in order to get the change? I see RT/Clojure do some interesting nesting of classloaders, that I fail to see the significance of, as the loaded classes are placed in a static instance variable.

classloaders are not sandboxes, they don't contain things like that

almost certainly if you are loading code and not getting an error, and not seeing the effects reflected in the clojure runtime you are looking at, it is because the code loaded into a different clojure runtime

It must be our special classloader then. There isn’t an error, it is afaict loading on the proper classloader. I was concerned about https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6 but you and Stuart I believe indicate AOT should not be an issue (it is gen-class in my case).

@U0NCTKEV8 In that Polylith code eval-in* [class-loader form] where is the class-loader coming from? I presume its a DynamicClassLoader instance.

I dunno, and my point it is, it is a lie, code doesn't run in a classloader

the best you can do is sort of say for a given thread "please think about using this classloader" using the context classloader stuff, but there is no way to say "for this code, and any threads it starts or any tasks it hands off to executors, please use this classloader"

Having instrumented the Clojure code though, it appears it loads single-threaded, and does, in fact, use the classloader set in the thread context. I’m currently trying to understand this block from Polylith

(defn eval-in [class-loader form]
  (eval-in* class-loader (clojure.main/with-bindings (eval '~form))))

the problem isn't with clojure code loading

the problem is with class loading

with-bindings sets up the bindings that are present in the standard clojure.main repl

you are missing some quotes in that snippet (or they got eaten by slack)

the code doesnt call eval there, it builds a datastructure where the symbol eval (actually clojure.core/eval because of syntax quote) is somewhere in there, and then passes that datastructure to eval-in*

But isn’t the form essentially getting evaled twice?

If I were attempting this, I'd load the AOT code in a parent DynamicClassLoader, and load the dynamic code into a separate DynamicClassLoader that references the parent (as its parent ClassLoader).

And yes, code runs in threads that reference to a context ClassLoader, not in a ClassLoader.

I have done some fairly interesting things with ClassLoaders in Java-land.

So by “AOT in a parent DynamicClassLoader”, you would instantiate one to load it into?

yep. New classes get loaded by the context class loader; however, an already-loaded class is searched up the parent hierarchy to the root ClassLoader. That's how Java segregates its bootstrap and root class hierarchies. And that's also similar to how JEE servers work…lots of ClassLoader management code in those, so they can be instantiated and thrown away at will at runtime.

So if you want to see the dynamic code changes, then, that “child” DynamicClassLoader would need to be used I presume (not the parent)?

I just think it's easier to manage using a child ClassLoader for more dynamic code, and keeping static code as a parent. I'd set the child ClassLoader as the ContextClassLoader for all my threads. Then any "new" code would affect get loaded in the child, but you can still reference all the classes loaded into the parent. I haven't deeply studied Clojure's DynamicClassLoader, though I've been aware of it and reviewed it some years ago. Rich may have overridden the standard class lookup behavior.

That’s interesting. I was seeing that Clojure creates a nested DynamicClassLoader structure when it does a load-string yet it places the loaded classes in a static instance variable. So I don’t understand what the nesting achieves.

So for a form it will create a DCL (DynamicClassLoader) for the top-level form, and then for each form inside it will nest a child off of the top-level (recursive), so it makes a tree.

In that case it's using the DCL's as scope/environment visible from a given form. Interesting usage!

depends what you mean by "TCO"

To my best knowledge, Clojure does not manage TCO. However, it lets you work around the need for TCO using the recur mechanism.

often in online discussions when you say such and such a language has TCO, you mean in that language tail calls are always optimized

and that is not the case in clojure, or I think any language on the jvm (kotlin, scala, etc)

Yes, that's what I'm asking. Kotiln too has an explicit tailrec keyword that converts the code to a while loop. I wanted to know what happens in Clojure with recur.

scala has a particular annontation where it will do a limited TCO on self calls

yeah, they all do the same thing

not really a while loop, but the bytecode a while loop might compile to

got it, thanks.

Scala does automatic TCO though, right? I read the annotation is to receive a compile error in case the code is not in tail position.

when last I looked, which was years ago, it was only supported for self calls (the same restriction recur has)

recur turns into a jump/goto instruction

Loop/ recur expands into a native Java for loop. You can decompile a class produced by Clojure and see.

You can see same examples here: https://github.com/clojure-goes-fast/clj-java-decompiler

@U1WAUKQ3E thanks for sharing this!