This is the full backtrace:

markus@ubuntu2204:~/src/specter-demo$ npx shadow-cljs compile :bootstrap
shadow-cljs - config: /home/markus/src/specter-demo/shadow-cljs.edn
shadow-cljs - updating dependencies
shadow-cljs - dependencies updated
[:bootstrap] Compiling ...
------ ERROR -------------------------------------------------------------------
 File: /home/markus/src/specter-demo/specter/src/clj/com/rpl/specter/navs.cljc
failed to require macro-ns "com.rpl.specter", it was required by "com.rpl.specter.navs"
Error in phase :execution
ClassNotFoundException: com.rpl.specter.Util
	java.net.URLClassLoader.findClass (URLClassLoader.java:476)
	clojure.lang.DynamicClassLoader.findClass (DynamicClassLoader.java:69)
	java.lang.ClassLoader.loadClass (ClassLoader.java:589)
	clojure.lang.DynamicClassLoader.loadClass (DynamicClassLoader.java:77)
	java.lang.ClassLoader.loadClass (ClassLoader.java:522)
	java.lang.Class.forName0 (Class.java:-2)
	java.lang.Class.forName (Class.java:398)
	clojure.lang.RT.classForName (RT.java:2209)
	clojure.lang.RT.classForNameNonLoading (RT.java:2222)
	com.rpl.specter.impl/eval15625/loading--6789--auto----15626 (impl.cljc:1)
	com.rpl.specter.impl/eval15625 (impl.cljc:1)
	com.rpl.specter.impl/eval15625 (impl.cljc:1)
	clojure.lang.Compiler.eval (Compiler.java:7194)
	clojure.lang.Compiler.eval (Compiler.java:7183)
	clojure.lang.Compiler.load (Compiler.java:7653)
	clojure.lang.RT.loadResourceScript (RT.java:381)
	clojure.lang.RT.loadResourceScript (RT.java:372)
	clojure.lang.RT.load (RT.java:459)
	clojure.lang.RT.load (RT.java:424)
	clojure.core/load/fn--6908 (core.clj:6161)
	clojure.core/load (core.clj:6160)
	clojure.core/load (core.clj:6144)
	clojure.core/load-one (core.clj:5933)
	clojure.core/load-one (core.clj:5928)
	clojure.core/load-lib/fn--6850 (core.clj:5975)
	clojure.core/load-lib (core.clj:5974)
	clojure.core/load-lib (core.clj:5953)
	clojure.core/apply (core.clj:669)
	clojure.core/load-libs (core.clj:6016)
	clojure.core/load-libs (core.clj:6000)
	clojure.core/apply (core.clj:669)
	clojure.core/require (core.clj:6038)
	clojure.core/require (core.clj:6038)
	com.rpl.specter/eval15483/loading--6789--auto----15484 (specter.cljc:1)
	com.rpl.specter/eval15483 (specter.cljc:1)
	com.rpl.specter/eval15483 (specter.cljc:1)
	clojure.lang.Compiler.eval (Compiler.java:7194)
	clojure.lang.Compiler.eval (Compiler.java:7183)
	clojure.lang.Compiler.load (Compiler.java:7653)
	clojure.lang.RT.loadResourceScript (RT.java:381)
	clojure.lang.RT.loadResourceScript (RT.java:372)
	clojure.lang.RT.load (RT.java:459)
	clojure.lang.RT.load (RT.java:424)
	clojure.core/load/fn--6908 (core.clj:6161)
	clojure.core/load (core.clj:6160)
	clojure.core/load (core.clj:6144)
	clojure.core/load-one (core.clj:5933)
	clojure.core/load-one (core.clj:5928)
	clojure.core/load-lib/fn--6850 (core.clj:5975)
	clojure.core/load-lib (core.clj:5974)
	clojure.core/load-lib (core.clj:5953)
	clojure.core/apply (core.clj:669)
	clojure.core/load-libs (core.clj:6016)
	clojure.core/load-libs (core.clj:6000)
	clojure.core/apply (core.clj:669)
	clojure.core/require (core.clj:6038)
	clojure.core/require (core.clj:6038)
	shadow.build.macros/load-macros/fn--13100/fn--13111 (macros.clj:101)
	shadow.build.macros/load-macros/fn--13100 (macros.clj:100)
	shadow.build.macros/load-macros (macros.clj:94)
	shadow.build.macros/load-macros (macros.clj:85)
	shadow.build.compiler/post-analyze-ns (compiler.clj:49)
	shadow.build.compiler/post-analyze-ns (compiler.clj:46)
	shadow.build.compiler/post-analyze (compiler.clj:92)
	shadow.build.compiler/post-analyze (compiler.clj:89)
	shadow.build.compiler/analyze/fn--13241 (compiler.clj:265)
	shadow.build.compiler/analyze (compiler.clj:252)
	shadow.build.compiler/analyze (compiler.clj:211)
	shadow.build.compiler/analyze (compiler.clj:213)
	shadow.build.compiler/analyze (compiler.clj:211)
	shadow.build.compiler/default-analyze-cljs (compiler.clj:415)
	shadow.build.compiler/default-analyze-cljs (compiler.clj:404)
	clojure.core/partial/fn--5908 (core.clj:2642)
	shadow.build.compiler/do-analyze-cljs-string (compiler.clj:321)
	shadow.build.compiler/do-analyze-cljs-string (compiler.clj:278)
	shadow.build.compiler/analyze-cljs-string/fn--13326 (compiler.clj:518)
	shadow.build.compiler/analyze-cljs-string (compiler.clj:517)
	shadow.build.compiler/analyze-cljs-string (compiler.clj:515)
	shadow.build.compiler/do-compile-cljs-resource/fn--13354 (compiler.clj:637)
	shadow.build.compiler/do-compile-cljs-resource (compiler.clj:618)

This is the configuration for the :bootstrap target:

:bootstrap
  {:target     :bootstrap
   :output-dir "public/bootstrap"
   :exclude    #{cljs.js}
   :entries    [cljs.js com.rpl.specter]}

Unless I'm reading the source incorrectly, all usages of the Util class are inside a #?(:clj reader conditional: https://github.com/redplanetlabs/specter/blob/master/src/clj/com/rpl/specter/impl.cljc#L22 and https://github.com/redplanetlabs/specter/blob/master/src/clj/com/rpl/specter/impl.cljc#L91

I was under the impression that for the bootstrap build only #?(:cljs branches are taken.

bootstrap compiles cljc files twice. one for cljs, once for macros

this appears to be a java helper classes, so you need to write an equiv for JS

@thheller Ok, but both are compiled to JS by CLJS, and they take #?(:cljs branches, correct? All mentions of the helper class are in #?(:clj branches.

yes, it takes the :cljs branch when compiling for macros. when compiling normally it is taking for :clj branch for the macros, since that runs in clj

so I looked into self-hosted macros compilation and why deftime doesn't work

found this commit https://github.com/thheller/shadow-cljs/commit/ab82092788fac5f6fd4aece75663166eedeb73b6#diff-6fa7ba019eb2696a1633a048a1f9248d19c07e6c2f42d7cbfc7a3c2e75b1375eR154

which explains it. but it has been like this for 5 years. so either nobody ever used deftime with shadow-cljs

or nobody complained

I'm sort of hesitant to change it, since I'm not sure who'd be affected

I don't yet follow. What is going on that the :bootstrap target can complain about Java things if it should take only #?(:cljs branches? Is this an issue that's separate from the deftime question? About changing or not changing *ns*: "`macros/deftime` and macros/usetime to clearly demarcate regions of code that should be run in the macro-definition stage or in the macro-usage stage. (In Clojure there's no distinction; in pure Clojurescript it's easy: just wrap the first stage in #?(:clj ...) and the latter one in #?(:cljs ...); in self-hosted Clojurescript it's messy or everything gets evaluated twice; supporting the three at the same time is Macrovich's raison d'être.)" (from https://github.com/cgrand/macrovich#usage) I take this to mean that in other self-hosted environments deftime works, so maybe it's a good idea for shadow-cljs to be compatible in that regard to these other self-hosted environments.

:bootstrap must compile the namespace twice. the first time it is compiled like regular CLJS, just the normal namespace using CLJ for macro expansion. this will end up loading all the related Java code. on the second pass it is compiled again in macro mode, which is purely CLJS.

the entirely separate issue of deftime is related to the *ns* binding, which I guess macrovich expects to be bound to the $macros variant, which is not the case in shadow-cljs and never has been

it is not the case because otherwise namespaced keywords such as ::foo would have unexpected results

I have never verified what regular CLJS does in these cases since I frankly don't know how to do macro compilation for regular CLJS

but as I said many times before. just remove deftime. it is not needed and doesn't do anything useful here.

Part of my quest is to solve the problem, the other is to gain understanding 🙂

if you want to setup a reproduction with regular CLJS that proves the differences I'm happy to adjust

but without that I'm not changing something nobody had an issue with for 5 years

you seem to have a good understanding of stuff. just the wrong expectation of what deftime does. at its current stage it is broken with regards to how shadow-cljs handles it. so its either broken in shadow-cljs or macrovich

Just one more question now, of what use is a ClojureScript JVM compilation of a cljs namespace, that is, one using CLJ for macro expansion, in the context of the :bootstrap build, which is producing artifacts that should be consumed by self-hosted ClojureScript?

the entire point of :bootstrap is to *pre-compile* namespaces. this is not something regular CLJS self-hosted does

I'll see if I can set up a show case how stuff behaves differently in shadow-cljs self-hosted vs other environments.

Yeah ok, but isn't that slightly incorrect, given that without shadow-cljs's caching/pre-compiling, the self-hosted CLJS compiler would request the CLJS source for the namespaces, both :macro false and :macro true , and read both without taking any CLJ branches? That must result in different analysis results than doing it on the server side with CLJ-hosted CLJS.

Is it a trade-off, where the difference is just in uncommon edge-cases that haven't come up in five years, or is there something else that I'm missing that resolves these differences?

Here's an example of macro compilation in self-hosted cljs http://nbeloglazov.com/2016/03/07/getting-started-with-self-hosted-cljs-part-2.html

To do everything "correctly" (in my current understanding of what that is), you'd have to use the self-hosted CLJS compiler for the pre-compilation as well. Not sure how practical that is. Maybe a way forward would be to have the pre-compilation be optional and just have shadow-cljs provide or prepare the file system / class path for loading CLJS sources to be used by the self-hosted CLJS compiler.

I'm still missing why having Util is a problem suddenly? if its only in :clj branches everything is fine and I don't understand the context of the problem

pre-compilation is already optional. just don't use it if you don't need it

the entire :bootstrap target is fine with just compiling cljs.core. there is no need to precompile anything else

it is just there to things like maria.cloud (which is what I wrote all this for) load faster. it saves a couple seconds of initialization time to not have the client compile everything on page load

like 10sec+, which was a significant speedup

I do not use self-hosted much myself, so I'm not very clear on any of the details myself. I know it works and that a few people use it as is

I don't know anything using macrovich though

Oh, I can switch it off? I'll try that then and see how that behaves differently apart from the performance aspect.

well, you have to supply the compilation parts then on the client

So that's just self-hosted cljs without :bootstrap at all then? The part I'd like to use, that :bootstrap provides, is making the source for namespaces available. Just without the precompilation.

if you want custom compilation you write the code for that custom compilation

bootstrap does exactly what it does and not more

Well, not so much custom, just that it should all be self-hosted CLJS 🙂

that is custom

the specify :entries [cljs.core] in :bootstrap and do the rest yourself

there is no need to specify anything else. it is just there as a convenience.

self-hosted is a very complex topic and it is impossible to cover all aspects simply

things are going to vary greatly depending what you are building this for

ie. running node means the sources and code are loaded entirely differently than in the browser

pre-compilation is provided as a convenient but OPTIONAL "service"

there are some other things that pre-dated :target :bootstrap that tried to do the same thing

https://github.com/braintripping/cljs-live

and some others I can't remember the names of

maybe they can provide some useful background

Alright, I learned a lot, thanks! This is my take-away: :bootstrap compiles namespaces in normal mode (i.e. :macro false) with the CLJ ClojureScript compiler. It'll correspondingly compile any required macro namespaces with Clojure and use these to expand macros. It'll also compile any required namespaces in macro mode with the CLJ ClojureScript compiler, producing JS output. (Here there's no difference between using the CLJ ClojureScript compiler or a JS-hosted CLJS ClojureScript compiler.) Effectively, if a namespace is required as both a normal and a macro namespace, it'll be compiled three times: • a) once explicitly by :bootstrap with the CLJ ClojureScript compiler in non-macro mode as ClojureScript to produce output JS and analysis for the browser-side CLJS compiler • b) once somewhat implicitly by the CLJ ClojureScript compiler with the Clojure compiler in macro mode as Clojure, to do macro expansion for a) • c) once explicitly by :bootstrap with the CLJ ClojureScript compiler in macro-mode as ClojureScript to produce output JS and analysis for the browser-side CLJS compiler Is this correct?

> It'll correspondingly compile any required macro namespaces

no. it'll not compile them at all. it'll (require 'that.ns) in CLJ only. the compiler never looks at them.

and we need to be explicit about .cljc in this case. cljc is always multiple files in one, which makes it more difficult to understand

lets takes a regular .cljs file with macros in .clj

for self hosted/boostrap the cljs file is compiled once and the clj file is compiled once. for normal compilation only the cljs file is looked at and compiled by the cljs compiler. the clj file is loaded by clojure and a black box as far as the cljs compiler is concerned

the same is true for cljc just that there is only one file

OK, let me type that out to account for that.

With the following :bootstrap configuration

{:entries [cljs.js my.separate]
 :macros  [my.separate]}

(ns my.separate
  (:require-macros [my.separate]))

;; main code omitted

(ns my.separate)

;; macro implementations and supporting code omitted

(ns ...
  (:require [my.separate]))

{:entries [cljs.js my.combined]
 :macros  [my.combined]}

(ns my.combined
  #?(:cljs (:require-macros [my.combined])))

;; macro implementations and supporting code omitted
;; main code omitted

(ns ...
  (:require [my.combined]))

no

> 2. During this process, the CLJ ClojureScript compiler requests ...

no, as I explained above ... the clojurescript compiler does absolutely nothing with those files

except require them.

also cljs/clj files don't take any #? branches. thats CLJC only

also the macro compilation happens after all cljs compilation is completed

> my/combined.cljc was compiled three times: once as ClojureScript in non-macro mode, once as Clojure in macro-mode, and once as ClojureScript in macro-mode.

again .. 2 times by the cljs compiler

the third time it is loaded by clojure as a regular clojure file. it is not touched or compiled in any way by the cljs compiler

this is true for cljs and cljc files

Alright, thanks! Back from holidays now, continuing this inquiry 🙂 Here's the corrected text: With the following :bootstrap configuration

{:entries [cljs.js my.separate]
 :macros  [my.separate]}

(ns my.separate
  (:require-macros [my.separate])

;; main code omitted

(ns my.separate)

;; macro implementations and supporting code omitted

(ns ...
  (:require [my.separate]))

{:entries [cljs.js my.combined]
 :macros  [my.combined]}

(ns my.combined
  #?(:cljs (:require-macros [my.combined])))

;; macro implementations and supporting code omitted
;; main code omitted

(ns ...
  (:require [my.combined]))

in all this summary I'd remove "It is compiled as Clojure." as it just confuses what is going on IMHO

no and yes to the last two questions

yes, the .clj file is compiled to produce JS output

the macro can emit what it wants in case of .clj too. Using reader conditionals for that is not the only way

and FWIW you do not need :macros [my.combined] in the build config. it is infered that this is require'd via the :require-macros. It is only needed for namespaces that are not otherwise required

Alright, thanks!