This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # announcements (7)
- # aws (5)
- # babashka (72)
- # beginners (43)
- # calva (12)
- # cider (9)
- # clara (3)
- # clj-kondo (12)
- # cljdoc (32)
- # cljs-dev (10)
- # cljsrn (1)
- # clojure (78)
- # clojure-dev (50)
- # clojure-europe (17)
- # clojure-gamedev (8)
- # clojure-nl (1)
- # clojure-spec (30)
- # clojure-uk (3)
- # clojurescript (52)
- # core-async (1)
- # cursive (5)
- # datomic (8)
- # emacs (58)
- # events (2)
- # fulcro (5)
- # graalvm (7)
- # holy-lambda (37)
- # honeysql (9)
- # jobs (5)
- # leiningen (3)
- # lsp (7)
- # lumo (2)
- # malli (3)
- # meander (13)
- # membrane-term (64)
- # missionary (19)
- # music (3)
- # nextjournal (8)
- # off-topic (29)
- # pathom (16)
- # polylith (14)
- # portal (16)
- # re-frame (2)
- # reagent (5)
- # sci (14)
- # shadow-cljs (20)
- # spacemacs (6)
- # sql (1)
- # tools-deps (58)
- # vim (14)
Would it be crazy to ask for additional compiler options to set
*compile-files* via java system properties at startup?
My current solution is a bit of a hack: I call
(.bindRoot Compiler/COMPILE_FILES true) on load of a particular namespace.
I am working in the context of distributed computing on HPC clusters. Most platforms that I am aware of use a JVM on each worker and expect all classes to be sent to each worker so that distributed tasks can be run. I use my hack to get class files from evaluations in my Clojure repl and then send them to a cluster of remote JVMs. It has been working well so far. For prior art, the Scala REPL always writes class files for each command and these are commonly sent to a remote JVMs.
The general problem statement is "Without the ability to write class files from the Clojure compiler, it is difficult to sync with remote JVMs."
if you're at the repl, I don't understand why you can't just dynamically bind those. why do you care about system properties?
writing classes is literally the only thing the compiler does, so I don't understand the problem
> if you're at the repl, I don't understand why you can't just dynamically bind those. why do you care about system properties? This is what we do today. It works for code compiled after the binding is set as long at there are no decencies on classes compiled before the binding. If system properties could be used to set the vars at startup, all classes created by the Clojure compiler would be portable. > writing classes is literally the only thing the compiler does, so I don't understand the problem https://stackoverflow.com/questions/69356231/get-or-write-class-files-for-the-classes-in-the-dynamicclassloader/69833025#69833025 was that the compiler creates classes in-memory and adds them to the DynamicClassLoader. If true, there are never any portable class files created. Is that correct?
whether to write them to .class files depends on
*compile-files* ... which you can bind when you need to
Here is an example:
(defn foo [x] x) => #'user/foo (binding [*compile-files* true] (eval '(defn bar [x] (foo x)))) => #'user/bar
*compile-path*will only have 1 class file,
user$bar.class. Based on my experiments, the bar class file seems to be unusable in a remote JVM because it references
foowhich I think will ask the class loader to load a class
user$foowhich won’t exist in the remote JVM.
The reason for my request to set the var at startup is that (I believe) it would allow me to keep the entire set of classes in sync across JVMs. I apologize if that I am thinking about this all wrong. I can see why this is an uncommon scenario.
I think you're starting to approach a better problem statement :)
compilation is a side effect of load, which is inherently namespace-oriented
you seem to want to compile functions to classes for remote portability
Correct! You absolutely trimmed the fat on my on my problem statement! For completeness, here is another example where a single form will result in multiple class files.
(defn i-must-be-portable [x] (let [me-too (fn [y] (+ x y))] (me-too x)))
in the prior case what do you want? the set of all classes needed to run a function (both bar and foo)?
My current solution is to bind the var and ask users to not call functions that were defined pre the binding of the var. It works, but feels wrong.
and you want an interactive environment, not one where foo is already on the remote machine
(that latter scenario is something we've been thinking about for a long time with rehydrating vars)
Yes. Everything works fine with an AOT compiled jar that is placed on all remote machines. Not so much in the interactive case.
several people have built things like this over the years, for cascalog/storm, for spark, etc
Ah, Spark is my use case as well. Do you recall which framework tackled this? I am familiar with most of them but I have never seen a solution to this.
FWIW, the Scala interface to Spark does it exactly as I am hoping to acomplish here: The Scala repl compiles all the incoming code from the user to physical class files and then broadcasts them to each worker JVM. The same could be done for Clojure if there was a way to set
*compile-files* before any functions get compiled.
Thanks. I’ve looked at Sparkling quite a bit but I must have missed it. I’ll look deeper.
I feel like there is another one too that I'm missing. certainly there were some older projects going way back, but I feel like there's something else
In addition to Sparkling, I have spent a bunch of time with https://github.com/sorenmacbeth/flambo and https://github.com/zero-one-group/geni. They either restrict deployment to AOT compiled uberjars (to interaction) or restrict the API to functions that are known upfront (no user defined functions).
@cgrand did something along these lines? Called ourobourus or some such. Gave a talk at a conj about it IIRC
Regardless, it sounds like you aren’t in favor of adding a way to set
*compile-files* at startup. Do you think there is value in putting together an http://ask.clojure.org on the topic for sake of record keeping? If so, I’m happy to write one.
the problem is much more interesting and there are probably other solutions that have nothing to do with system properties :)
Sure. I can imagine that this might be the only problem for which a possible solution is “get class files for everything”.
people have run into this same general problem with Hadoop, with database functions in Datomic, etc
On the other hand, it was generally confusing to me that there wasn’t a way to specify all the compiler vars on startup.
system properties are generally a pretty gross solution as they bake in the assumption that everything in jvm has the same goal and exactly one answer
Gotcha. That makes sense. If my project requires property A to be X and your requires A to be Y, we can’t compose.