Fork me on GitHub
#clojure
<
2021-02-08
>
Narendra12:02:57

I am trying to understand the following type

user> (type in-ns)
clojure.lang.RT$1
user> 
I can see that RT.java defines in-ns (`clojure.lang.RT/IN_NS_VAR`) as a Var (with a value equal to Boolean/FALSE?) but where does this RT$1 class come from?

bronsa12:02:55

that's the naming scheme javac uses for anonymous classes

Narendra12:02:41

Thanks for the link. I couldn't find this relationship between IN_NAMESPACE and IN_NS_VAR. Now, it makes sense. 🙂

bronsa12:02:49

so RT$1 is the name assigned to the first anonymous class inside the RT class

👍 3
Tomas Brejla12:02:42

Hello. Does anyone here know about some workaround for this issue with clojure compiler https://clojure.atlassian.net/browse/CLJ-1852? It seems to be a cause of this issue in Fulcro I reported a few days ago: https://github.com/wilkerlucio/pathom/issues/187. Perhaps @alexmiller might have some workaround idea?

Alex Miller (Clojure team)13:02:38

The long name is the combination of nesting and existing names, so using shorter intermediate names and/or nesting less are the two main workarounds

Tomas Brejla13:02:18

That's what I thought as well. But if I'm looking correctly to that specific problem in compiling of pathom, it doesn't seem to use that-much of a nesting, does it? https://github.com/wilkerlucio/pathom/blob/v2.3.1/src/com/wsscode/pathom/core.cljc#L564

Tomas Brejla13:02:27

Maybe if one would macro-expand all that, it's actually nested a lot. But at first sight I don't see any obvious "low-hanging fruit" of easily reducing the nesting in that code.

mpenet16:02:02

isn't the code in question a bit problematic: nested go blocks

mpenet16:02:25

I suspect that could be one of the reasons why it explodes

noisesmith16:02:31

@U01LFP3LA6P hard agree with mpenet - the big culprits for massive method size and deep unexpected nesting after macroexpand are go, core.match, and for / doseq - if you nest those things enough you will run into issues

noisesmith16:02:50

usually it's a question of decomposing into functions to remove that nesting of "verbose" macros

noisesmith16:02:29

the "low hanging fruit" in your example is to replace the nested go block with a call to a function that makes a go block

noisesmith16:02:40

re-reading - it's probably actually to move the function definition out of the deep nesting where the current literal is (and factor any captured locals into a hash map arg)

mpenet16:02:30

I didn't dig deeper but for sure some refactoring might help

Tomas Brejla17:02:19

Thanks a lot for your ideas, guys. I'll pass this information to the mentioned pathom issue. And if I find some spare time, I'll try to look into the issue myself just to learn what can be done with it.

Tomas Brejla18:02:28

it'd be nice if the compiler itself didn't have that issue though 😉

noisesmith18:02:18

it's intrinsic to the compilation model - fn inside fn is a nested class

noisesmith18:02:58

since go rewrites things into fns, your nesting can get very deep

noisesmith18:02:13

fixing this compilation issue (without losing performance) would be one of those "clojure 2.0" things, and I suspect clojure 2 would never happen

Alex Miller (Clojure team)18:02:09

I don't think it needs to be of that level of significance. it's on the list to think about for 1.11 at least

noisesmith18:02:37

oh wow - I'm glad if that's the case

Alex Miller (Clojure team)18:02:02

I'm not saying it's likely :)

Tomas Brejla18:02:15

Awesome! I don't want to push my luck, but speaking of list of things to think about.. I wonder whether this is still a possibility? 😇🤞 > Rich and I have been kicking around some interesting ideas on a class compilation cache that would be integrated with core and that research is in scope for Clojure 1.10 but no guarantees on anything. Potentially it would combine the benefits of publishing source-only but the performance of AOT. It is inherently silly to recompile the same lib source file over and over when it’s not changing. https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/10

Alex Miller (Clojure team)18:02:26

you can get 90% of the benefits of that using the approach in https://clojure.org/guides/dev_startup_time right now

👍 3
Alex Miller (Clojure team)18:02:40

(and probably fewer of the downsides)

erwinrooijakkers17:02:01

I see this thread, https://clojureverse.org/t/what-about-goto-in-clojure/3943/19, is it possible to create a goto macro in Clojure? Make a label somewhere and use (`goto label)` to continue execution at the label?

noisesmith17:02:53

in theory, but it would all have to be in one method body for a literal vm goto

noisesmith17:02:26

otherwise you end up needing an interpreter, which clojure intentionally is not

noisesmith17:02:23

@erwinrooijakkers an option that I don't see in that thread: letfn plus trampoline allows a structured rewrite of any code block that uses goto

👍 3
noisesmith17:02:15

replace each segment of code between labels with a named fn inside letfn, and replace every fall-through to the next label with returning the function replacing that label

noisesmith17:02:32

similarly, replace each goto of some label with returning the function replacing that label

noisesmith17:02:54

it has the same semantics (but is a little slower thanks to function call overhead)

borkdude17:02:58

There's also an issue in JIRA about loop + recur-to I believe

noisesmith17:02:02

;; x = 1
;; a: inc x
;; b: if (x > 12) goto :c
;; goto :a
;; c: return x*x
(let [x (volatile! 1)]
  (letfn [(a []
            (vswap! x inc)
            b)
          (b []
            (if (> @x 12)
              c
              a))
          (c []
            (* @x @x))]
    (trampoline a)))

noisesmith17:02:24

it's a tedious way to write code, but if going through Knuth, like the example, the translation is very direct

noisesmith17:02:51

(returns 169)

noisesmith17:02:05

(nb. the gotcha with trampoline is you have to wrap a function in a container in order to return it)

noisesmith17:02:55

a version of trampoline that works on (function, value) tuples instead of just function might be a fun exercise

hiredman17:02:25

you could use a continuation monad to get an extremely literal translation of goto containing code https://gist.github.com/hiredman/235655ccde8689499576db882095959f is of what using a continuation monad can look like, it creates labels and uses goto to jump to them.

hiredman17:02:46

https://gist.github.com/779827c2f1055c5ab080a32bff31656b is the little continuation monad library that example code uses, it is kind of customized around the needs of that particular library, no docs on it of course

erwinrooijakkers13:02:26

I’m going to study this 🙂 never fully understood continuations

hiredman17:02:29

with continuations a label is "capture the continuation here" and goto is "replacing the continuation here with this captured one"

noisesmith17:02:08

@borkdude interesting that your link doesn't mention trampoline - I always thought that letfn (which it does mention) is pretty much pointless outside trampoline

p-himik18:02:47

letfn is useful even with a single recursive function - to avoid having to type its name twice.

noisesmith18:02:13

in a way that (fn foo [x] ...) isn't?

hiredman18:02:46

(let [foo (fn foo [x] ...)] ...)

hiredman18:02:55

is the double name thing

hiredman18:02:37

I disagree with the letfn hate, letfn is great, I prefer it to a regular let binding of functions

p-himik18:02:54

I do too, but only when there are no non-fn bindings involved, to avoid an extra level of nesting.

hiredman18:02:01

sure, creating a cyclic graph is trivial when you mutate stuff

noisesmith17:02:38

(and trampoline addresses the non-tail-recurring complaint in that post)

vemv18:02:23

Are people following the https://clojure.org/guides/dev_startup_time dev model at scale? It worked well for me in a small-medium project (where it shaved some 10s startup time) but in a larger app it would immediately cause issues. Which would not seem surprising when "AOT in dev" is kind of a frowned-upon thing (and a root cause diagnostic that I've seen given in this very channel many times) So I'm a bit conflicted as to what to believe

Alex Miller (Clojure team)19:02:11

in my experience, using pre-compiled libs is problematic (because that nails down use of a specific clojure version and to some degree fixes compilation order in a way that may be different than use), but using aot in dev is not inherently frowned upon - it is exactly the same work being done to compile at load time, just saved to disk

hiredman18:02:22

I think that guide undersells the tricky bits of making aot in dev work because it is written by someone with considerable clojure and clojure tooling expertise

👀 3
seancorfield19:02:40

I have a compile.clj script that, when loaded, will compile several top-level namespaces in our codebase to follow that "AOT in dev" model, but I don't update the compiled classes very often. I haven't noticed any unusual behavior with it but I'm probably not pushing it hard -- it does improve initial load time in dev noticeably but, OTOH, I only restart my REPL maybe once a week so I don't suffer that initial load time very often anyway...

👍 3
vemv19:02:14

thanks! do a lot of 3rd-party libs get compiled transitively?

vemv19:02:51

that's kind of good news as it means that many libs in the wild can survive AOT in dev? :) I think I had problems with two, although I'm not experienced in AOT to actually tell if they were wrong

Alex Miller (Clojure team)19:02:37

it would be helpful to file questions on https://ask.clojure.org if you run into them

Alex Miller (Clojure team)19:02:03

AOT compile is exactly the same compile as the not AOT compile, so generally all libs should "survive AOT" (because they are being compiled on load)

Alex Miller (Clojure team)19:02:45

there can be issues with load ordering, particularly around protocols being reloaded after instances have been loaded

caumond19:02:11

yes @U04V70XH6, about repl management, I can say I improved a lot my workflow thanks to your advices. I mostly start my repl for deps.edn update right now ! it has been a learning curve but the final experience is much more "sure", I feel like I am more mastering the state of my code, even during deep refactoring

👏 3
Alex Miller (Clojure team)19:02:07

if there are issues that people are running into doing this kind of aot dev env, I would love to be hearing about them so we can assess if there are things to fix/improve

seancorfield20:02:07

@U45T93RA6 FWIW, I just cleaned out my local classes folder and kicked off the compile script and that produces just over 13K .class files.

vemv13:02:10

Thanks everyone! I've given it another shot. Here's one specific blocker I have found out. It seems easy to reproduce: https://github.com/clj-commons/potemkin/issues/50#issuecomment-775954692 (I don't like / use Potemkin directly; but that dep can a bit hard to avoid in large/existing projects) *Edit*: might not be specific to Potemkin. Will update that issue after I'm done figuring out stuff

Alex Miller (Clojure team)14:02:41

The main reason you won’t see a class is because the class already exists on your classpath

👍 3
Alex Miller (Clojure team)14:02:10

Not sure if that’s what you’re seeing but something to rule out first

vemv14:02:00

Yes, I figured something was requireing potemkin before my 'compile' step So I'm doing

(binding [*compile-files* true]
  (when-not (-> *compile-path* (File. "potemkin") .exists)
    (require 'potemkin :reload-all))
  ;; proceed with other compilation...
  )

vlaaad20:02:25

(:foo (sorted-set 1 2 3)) => ClassCastException (can't cast Long to Keyword)
is there a generic way to check if something has key :foo without exceptions? (contains? (sorted-set 1 2 3) :foo) also throws...

manutter5120:02:13

(some #{:foo} (sorted-set 1 2 3))
=> nil
(some #{1} (sorted-set 1 2 3))
=> 1

vlaaad20:02:48

mm no, by generic I mean something that works on ANY object without throwing exceptions

vlaaad20:02:16

this requires my generic arg to be a collection

timsgardner20:02:25

You could just filter for coll? first

timsgardner20:02:11

I guess what's trickier is also avoiding a linear search

vlaaad20:02:25

I also want this stuff to be fast, e.g. O(1) on maps, not O(N)

p-himik20:02:06

sorted-map cannot be O(1). :)

p-himik20:02:49

(defn safe-contains? [maybe-coll val]
  (try
    (contains? maybe-coll val)
    (catch Throwable _ false)))

vlaaad20:02:52

I mean reasonobly fast

vlaaad20:02:20

yeah, probably catching is fine...

vlaaad20:02:32

thanks folks!

timsgardner20:02:08

Hey all, old question I'm sure but I had trouble finding an answer: is there a way to intern a new dynamic var? Like (intern (ns-name *ns*) (with-meta 'blorp {:dynamic true})), except it works

r20:02:51

hey there, i’m running into an opaque macroexpansion error on clojure 1.10.2 that i did not have on 1.10.1, and i was wondering if any folks had some ideas (details in thread):

r20:02:28

i have a project that depends on specter and methodical and a newer version of riddley than is specified in either of those projects

r20:02:03

i’ve tried adding an exclusion for riddley to both methodical and specter’s dependency vectors, but this error persists

r20:02:10

the error is specifically this:

r20:02:25

{:type clojure.lang.Compiler$CompilerException
   :message Unexpected error macroexpanding com.rpl.specter/path at (com/rpl/specter.cljc:980:9).
   :data #:clojure.error{:phase :macroexpansion, :line 980, :column 9, :source com/rpl/specter.cljc, :symbol com.rpl.specter/path}
   :at [clojure.lang.Compiler macroexpand1 Compiler.java 7023]}
  {:type java.lang.NoClassDefFoundError
   :message clojure/core$seq_QMARK___5406
   :at [riddley.walk$macroexpand$fn__214 invoke walk.clj 14]}
  {:type java.lang.ClassNotFoundException
   :message clojure.core$seq_QMARK___5406
   :at [java.net.URLClassLoader findClass URLClassLoader.java 382]}

r20:02:55

note that lowering the clojure version to 1.10.1 causes the error to go away

r20:02:06

thanks in advance!

p-himik20:02:47

Just in case you don't get an answer here - that statement about "working with 1.10.1, not working with 1.10.2" might make it worthy of a question at https://ask.clojure.org/

r20:02:19

thanks! i’ll post in there in a bit.

👍 3
Alex Miller (Clojure team)20:02:57

there's not that much different in 1.10.2 so that is surprising to me

r20:02:14

it’s surprising to me too! i just checked, and it actually even manifests on 1.10.2-alpha1

Alex Miller (Clojure team)20:02:01

it looks like methodical is a compiled uberjar and includes riddley, potemkin, etc inside it

Alex Miller (Clojure team)20:02:59

it's basically a hidden dep conflict in that you end up with two versions of those transitive deps on the classpath, one pre-compiled with a different version of the compiler

r20:02:18

ah! no wonder the exclusion had no effect

r20:02:54

nice catch 🙂

Alex Miller (Clojure team)20:02:28

same bug is filed multiple times on the methodical repo

r20:02:33

thanks so much mr. miller!

hiredman21:02:43

If excluding potemkin and ridley has no effect, it sounds like some other dependency is including their source or more likely aoted classfiles

hiredman21:02:53

Which is often bad news

hiredman21:02:47

(oops. missed already done I guess)

p-himik21:02:09

Would it be reasonable to detect and issue a warning when you're trying to exclude something that was uberjar'd?

hiredman21:02:55

it is tricky

hiredman21:02:11

impossible in the general case, maybe sort of workable in someway?

hiredman21:02:34

clojure has kind of two languages of dependencies

hiredman21:02:06

the language of clojure itself is expressed as clojure namespaces, references to classes by name, etc

hiredman21:02:33

but overlayed on top of that is the language of artifact dependencies (maven mostly)

hiredman21:02:09

the names in those two languages have no mechanical linkage

hiredman21:02:45

and exclusions are expressed in terms of maven names, and compiled clojure classfiles exist in terms of clojure names

p-himik21:02:03

And it's not possible to robustly get the URL from which a particular ns was loaded, right?

hiredman21:02:06

something that would give you similar kind of warning, and might be easier to implement, is a utility to look at your dependencies and warn you if any of them contain both classfiles and .clj files

hiredman21:02:02

you could maybe download the transitive dependency tree of your exclusions, build a list of namespaces that seem to be defined there, then look through your dependencies and issue a warning if they are defined there

p-himik21:02:06

Mmm, right, I see. Thanks!

Alex Miller (Clojure team)21:02:23

I believe this utility to examine your classpath exists

Alex Miller (Clojure team)21:02:38

it certainly exists for java, and I seem to recall someone made a clojure friendly one too

Alex Miller (Clojure team)21:02:13

but unfortunately I don't remember a link or context for it

borkdude21:02:21

I think I recently discussed a similar thing with @noisesmith. The way I solved a similar problem was just to do:

(io/resource "clojure/spec/alpha__init.class")
to look for an AOT-ed class of the lib I was expecting the source for and not a class. Then I suggested this could be hooked up to the output of
user=> (binding [clojure.core/*loading-verbosely* true] (require '[clojure.spec.alpha] :reload-all))
(clojure.core/load "/clojure/spec/alpha")
to search in the order of __init.class, .clj and .cljc and report all loaded "files" using the io/resource invocation and parse out the exact mvn versions or gitlib paths to give you an overview of what's loaded from where.

borkdude21:02:58

(clojure.spec.alpha is just a random example here, that wasn't the lib in question I had a conflict with then)

hiredman21:02:37

that kind of analysis is handy (an oracle that tells you which namespaces are defined in which jars), I started writing it the other day (for what feels like the 10th time) because I had the idea that some of our projects at work have unused dependencies, and getting rid of them would cut down on build artifact size (big dumb tree shaking)

borkdude21:02:46

But maybe clojure itself could emit this information during loading verbosely

hiredman21:02:55

for our purposes, while it is true that issues like this usually crop up because of aot dependencies, the aot bit really doesn't matter

hiredman21:02:37

you want it to be an error/warning when a clojure namespace froms a jar file you don't want it to come from

borkdude21:02:52

@U0NCTKEV8 there is also tools like this: https://github.com/SevereOverfl0w/vizns (based on static analysis, not on runtime data). This probably won't find the above problem though, since it relies on source for analysis, not AOT-ed stuff.

hiredman22:02:33

but yeah, vizns looks like what I was describing, not just namespace dependencies, but including maven artifact dependencies

r22:02:08

so, on a different tack—it appears that methodical is including potemkin (and its dependencies) because of potemkin’s definterface+

r22:02:56

it internally defines a java interface

r22:02:25

is there a way to work around needing to AOT compile that so the interface is present?

r22:02:33

or would that require a restructuring

borkdude22:02:39

That interface would be defined when you load those namespaces, so why is AOT needed?

r22:02:44

ah, i see what’s happening

r22:02:13

the interface namespace is never :required, it’s only :imported

hiredman22:02:46

it seems even somewhat likely that the lack of the :require caused them to aot, so they could import without requiring, the aot caused them issues where some code would use the aot'ed interfaces and some would use the newly created interfaces once the required code was loaded, which caused them to bring in potemkin to definterface+ which seems to have its only feature avoiding re-defining the interface

🎯 3
r23:02:34

thank you for the help and great conversation!