Look at self-hosting clojurescript

An example of such a thing is https://clojurescript.io

@U04VDQDDY: http://clojurescript.io / replumb looks great; thanks for sharing the link. Is there a benchmark somewhere of replumb vs (cljs on jvm) performance ? In particular, I'm curious if (1) this is interpreted or (2) compiled. In the case of (1) interpreter slowdown and , in the case of (2), compile time.

Self-hosted is compiled

The self-hosted compiler is slower than the JVM compiler

I can't recall if actual perf numbers are recorded anywhere. YMMV

@U04VDQDDY: Follow up dumb question -- why does https://github.com/arichiardi/replumb not have updates for past 5 years? Has the CLJS community moved on to something else, or has this software somehow magically designed it's API in such a way to not need updates for the past 5 years ?

Well, the only thing that I would think to update with Replumb is the ClojureScript compiler... not much else to update in it. Similar thing is true for https://replete-repl.org which was one of the first (if not the first) self-hosted REPL. Once Replete was created 7 years ago (https://blog.fikesfarm.com/posts/2015-06-27-replete-a-standalone-ios-cljs-repl.html), its relatively simple UI has been stable since.

You can still see on that page that it is indeed used. Also note that GitHub search results don't include more than 2 lines from the files where the search query is found.

Can you give me a concrete example that it is used, for real?

I saw some examples here: https://github.com/clojure/clojure/blob/201a0dd9701e1a0ee3998431241388eb4a854ebf/src/clj/clojure/core_print.clj#L328

Right there in the search results.

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L4923

The compiler has to be able to serializing some what arbitrary data, to be stored in a class file, and then reconstructed when that class file is loaded or the bytecode in it is run, it has special handling for most of the built in data types in clojure, but falls back to print-dup when it encounters something unknown

In simple words, print-dup is solely for serialization and print-method is just for human readability. However, the result of print-method might be the same as print-dup sometimes. I used to serialize edn content with (pr-str) ( where by default *print-dup* is false), is this a valid way to serialize edn?

honestly the biggest difference between print-dup and print-method from usage that I've seen is that print-dup must be able to recover something equal to the original object by reading the structure assuming the code for it is loaded. print-method may produce something that requires data readers to be read.

(and the occasional unreadable object like with functions and java objects)

@U5NCUG8NR, Say if I define a custom type. Should I write my own print-dup function? Will there potentially any core lib that indirectly invokes my print-dup function? I am not quite sure how the print-dup function is used. Other than I do a ( binding [print-dup true] (pr something) myself and read it back.

So if you introduce your own print-dup impl the primary use for that is unsafe wire protocol (prefer clojure.edn and a data reader), and to allow people to have your new data structure be part of the source code emitted from a macro. At least to my current understanding.

And a key point here is this would be like having a vector in your source code, where the data structure is actually part of the syntax, not just constructed in the resulting code.

a print-dup method may also be important for your data type if you have a data-reader literal to allow you to have data structure literals like this in your source code

> a print-dup method may also be important for your data type if you have a data-reader literal to allow you to have data structure literals like this in your source code print-dup is on the producing side, while the data-reader literal is on the consuming side. For example, one can directly write #inst “some-time”, which totally bypasses print-dup. The confusion of me is “Where does print-dup gets used?“. Even clojure.edn/ is totally on the consuming side, it only contains the “read” and “read-string” function.

So I'm not 100% sure about this, but my understanding is that when you have a data reader and you compile code, then you have a literal object of the type it got read in as in your code. In order for an object to be stored in the produced class file it must be printed with print-dup . This means that types which have data readers should also have print-dup.

Well, it's pretty much read then eval, where prior to eval macro expansion is performed

I came across things like #= macro.

And also in the above thread (the one right above), @U0NCTKEV8 said some compiler treatment on print-dup. I am not sure how this is related to compiler. I thought it is more related serialize data to some file on disk and read it back, which has not much to do with the compiler. The compiler might only see some reader tag function to tell how to recreate some object, that

that’s it.

right, I didn't mention reader macros, which start with # and are expanded at read time

they include stuff like #{} for sets, #_ for comments, and #= for evaluation at compile time

so you can write something like (defn foo [x] (+ x #=(+ 1 2))) and it will evaluate (defn foo [x] (+ x 3))

There's also some bootstrapping involved where the reader relies on other functions being defined to fully work, which is why macros aren't implemented with syntax quote at the beginning of clojure.core but are later

I can understand the compiler transforms #=(+ 1 2) to 3. But what does #{1 2 3} be transformed to by the compiler?

Will #{1 2 3} be turned into some list form, e.g., (hash-set 1 2 3) (I doubt not), or some concrete hash-set object? If the latter, what happens for #{a 1 2} where a is unknown during the compilation time, e.g., a might be a parameter of a function.

> right, I didn’t mention reader macros, which start with # and are expanded at read time. @UK0810AQ2 I feel like for things #{a 1 2}, the evaluation happens at run time.

It isn't, the expansion is at read time. #= expands to eval, while a set, depending if all arguments are statically known, will compile to a static set or the hashset constructor

> {a 1 2} where a is unknown during the compilation time It becomes (hash-set a 1 2)

You can see it by doing:

(read-string "`#{~a 1 2}")

;> (clojure.core/apply clojure.core/hash-set (clojure.core/seq (clojure.core/concat (clojure.core/list 1) (clojure.core/list a) (clojure.core/list 2))))

https://clojuredocs.org/clojure.core/*read-eval* See note here, mainly the part starting with "As you can see, everything after the first symbol is left untouched."

Thanks. That’s quite mysterious.

The main use for this reader syntax is for print dup, where all values are already evaluated. Doing it this way means print dup implementations don't have to be careful when they contain symbols.

If you must refer to a value by a symbol like this, use #=(eval (...))

no, it's a real thread

channels are gc'ed when they are no longer referred to

if you're using go blocks, they are multiplexed over a fixed pool of 8 threads (by default) so you can have a zillion go blocks

there are no lightweight threads in the jvm (yet - see Project Loom)

OK -- go blocks will work. I'm processing an event feed and the events can be handled in parallel, but all update the same data structure (which uses refs to handle updates) -- was worried about overloading the OS if there are thousands of events in flight.

The bigger problem here is that you shouldn't block execution in a go block

I guess by the end of this it's more like a question about eval than about print-dup, but the question stands.

What happens when you (prn-str (rope [1 2])) instead?

it gives a rope with the numbers 1 and 2. Numbers don't need evaluation to have the correct value.

Sorry, I misunderstood the problem. Digging into your example, what's the value of (binding [*print-dup* true] (prn-str (rope ['a 'b]))) ?

If it's #rope[a b] you're good (I think 🤞):

user=> (defn rope [xs] xs)
#'user/rope
user=> (binding [*data-readers* {'rope rope}] (read-string "#rope[a b]"))
[a b]

Yeah, the problem is that (eval #rope [a b]) (with appropriate data readers) is #rope [a b] , it doesn't try to evaluate a or b

#babashka

probably the spot where you'll get the best help

yes thanks

I commonly use this thing:

(defn bash [cmd]
  (str/trim (:out (clojure.java.shell/sh "bash" "-c" cmd))))

(->> (bash "ls **/play.edn | sed 's|/play.edn||g'")
       (str/split-lines)
       (map (fn [id] {:id id})))

Not sure what "self hosted cljs" means. What exactly are you looking for?

Oh like http://replit.com but self hosted and just for clj/cljs?

https://github.com/babashka/sci

https://github.com/technomancy/leiningen/blob/master/doc/TUTORIAL.md#checkout-dependencies

IIRC you can't really do it, or at least not that easy at all. Clojure JAR is AOT-compiled and that compilation is tricky because there's spec which depends on Clojure which depends on spec. FWIW, Cursive is able to debug Clojure itself just fine, I've done plenty of sleuthing that way.

I hesitate to ask, but what aot problem?

@U0NCTKEV8 I ran compile on a single namespace, and curiously a non-aot'd namespace cannot refer to it anymore

compile basically does a force reload of the namespace you pass it

compilation is a side effect of code loading, so to compile a namespace compile has to load it again, and compile doesn't check the set of already loaded namespaces (like require does) which would stop it from compiling already loaded namespaces

I mean, I checked that the compiled class file exists for the namespace I ran compile on, but when restarting the clojure app the namespace depending on it cannot find it

ah, I assumed you were in the same repl where you can compile still

so you are getting a class not found error?

:cause No namespace: gambit.lib.async-buffers
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message Syntax error compiling at (gambit/common/enduro_utils.clj:1:1).
   :data #:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source gambit/common/enduro_utils.clj}
   :at [clojure.lang.Compiler load Compiler.java 7652]}
  {:type java.lang.Exception
   :message No namespace: gambit.lib.async-buffers
   :at [clojure.core$refer invokeStatic core.clj 4222]}]
 :trace
 [[clojure.core$refer invokeStatic core.clj 4222]
  [clojure.core$refer doInvoke core.clj 4205]
  [clojure.lang.RestFn applyTo RestFn.java 139]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$load_lib invokeStatic core.clj 5955]
  [clojure.core$load_lib doInvoke core.clj 5917]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$load_libs invokeStatic core.clj 5974]
  [clojure.core$load_libs doInvoke core.clj 5958]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$require invokeStatic core.clj 5996]
  [clojure.core$require doInvoke core.clj 5996]
  [clojure.lang.RestFn invoke RestFn.java 805]

it's a no namespace error inside refer

I would check to make sure you namespaces and file names all match up

I believe you get an error vaguely like that if the namespace a file has the ns decl at the top doesn't match the path (from the classpath root) of the file

Yeah i think they match up because when i delete the compiled class file it works

how are you calling compile? are you calling the function directly?

i'm not sure how else to compile except to run e.g. (compile 'gambit.lib.async-buffers)

well you could be using lein, or tools.build or maven even

what is the state of the repl where you call compile? do you have a user.clj?

yeah I run inside a comment block inside a user.clj

there's probably a ton of questions needed answering here lol

get rid of the user.clj

like move it off to the side with a different name, start a new repl, compile again and then see what happens

this is a giant project, and a lot of initial loading of everything happens in user.clj

any code the user.clj causes to be loaded will be part of the compilation enviroment when you call compile, which can do funky things, even mask problems with compilation

you really don't want that

okay I’ll try it, thanks

the ideal scenario for compilation is you have a sort of main namespace, and that requires everything it needs. so you call compile on that namespace, and it and all the namespaces it loads and so on get aot compiled when you do that

alright, i put the offending require statement in the main namespace, and cut out the user.clj calls, and it worked

basically the loading when you do the compiling matches the regular loading of the code

our main namespace is calling out to user/reset which uses tools.namespace to load all our namespaces for some reason

when they differ (different code loading orders, different sets of code being loaded) you often have problems

oof, sounds terrible

i guess it's a mechanism for reloading all the namespaces after a new checkout

reloading stuff while aot compiling is going to result in oddities too

thank you for clearing up that the problem might be with the way we're dynamically loading our namespaces, and with user.clj. i was hoping for a more clear view of what the compiler is doing, and now I’m more confused than when I started I think

the compiler is basically banging two rocks together to make fire

i hesitate to ask you to explain that lol

like, it is extremely primitive, clojure code is always compiled to java bytecode, when you aot compile it basically just sets a flag that says "while you are loading this code, generating bytecode, and then executing that bytecode, also write the bytecode to disk"

that I get, I’ve been walking through clojure's compiler.java a bit

the fact that the aot compilation process is also running the code as it is compiled makes it very easy to perturb

well actually I never recompiled the classfile for the namespace to make it work

so the problem is with the tools.namespace.repl/refresh or whatever not finding it

and because you do it from the comfort of a running clojure system, that running clojure system is part of the compilation environment for the code and will effect how it is compiled

but I thought tools.namespace was just a wrapper around require :reload

it is not

tools.namespace has a lot in it

I understand in theory that the compiler is stateful, but I thought that would be protected, unless all the namespace tables are fragile in a way I dont understand

it is all very fragile

I would definitely recommend your main namespace not depend on a user.clj and not call functions in that, and not call any functions that reload everything

part of this has to do with the transitive nature of compilation

so if you call compile on namespace A, the compiler will aot compile all code loaded when A is loaded, but if you have a user.clj that has already loaded everything, then A's dependencies won't be loaded again (assuming they are loaded via require) which means they won't get aot compiled, which can cause problems

main doesnt require user.clj as normal, but detects when we're in development mode then does a hard (require 'user)

would anything be different if I just renamed user.clj to foo.clj

if you renamed it to foo.clj and didn't load foo.clj, then yes

okay, so it's more the fact that it's being loaded and not some intrinsic effect of this being a specially treated user namespace

oh, you’re saying compile on a namespace whose dependencies may already be loaded may cause problems

and even within A, if A is already loaded, then you compile A, the compiled version of A could end up with some kind of dependency on the state of the already loaded version of A, which of course won't be there when you try to load directly from the class files

I see

so I should compile namespaces before loading them somehow

separate from the comfort of my already loaded app

yes

a tool like lein or tools.build or maven can assist with this kind of thing, but you can also bypass their assistance by having a user.clj in the wrong place, since they cannot stop clojure from automatically loading your user.clj

in theory then, a properly generated classfile based on a “clean” compiler state should be loaded fine when restarting my app, even with all the weirdness of how our namespaces are loaded via tnr/reset

reset likely does a scan for changed clojure files which may break if you are doing it unconditionally

well what confuses me is that trying to walk through this stacktrace, is that (require '[A :refer [foo]]) seems to try to run refer before it even gets to the point where the classfile can be loaded (in RT/load)

I am not sure what version of clojure you are using so I haven't tried to look at the line numbers

I’ve run out of time on this for today. Thank you so much brother for walking me through this

we’re on 1.10.3