Fork me on GitHub
#shadow-cljs
<
2017-10-04
>
thheller07:10:24

@mhuebert do you know if the CLJS compiler will read macro files with :clj or :cljs conditionals?

thheller07:10:44

only references I can find is :cljs but that does not seem correct to me?

thheller08:10:33

@mhuebert mind if I take your site an example for why :foreign-libs are bad? 🙂

thheller08:10:03

you do have 4 seperate react versions in your page at https://www.maria.cloud/intro

thheller08:10:43

2 for iframe, 2 for main

mhuebert09:10:53

@thheller :-). I don't mind at all. I noticed the duplicates a couple days ago and was one reason why I was trying to switch to your js system. (The duplicates between the two frames is another story though. One frame has to be advanced compiled, the other can't be, and they load from different domains for security reasons. I would need to load React separately, from the same domain in both frames, & only once. The only way I know how to do that is deps.clj with an empty js file 'providing' react with a :global-exports option. I have encountered so many troubles with this stuff)

mhuebert09:10:39

@thheller that depends on if you are compiling from the self hosted compiler or not

thheller09:10:45

I think I have this self-host stuff almost figured out

thheller09:10:26

yeah not saying that my stuff would fix all your issues, just that :foreign-libs is bad 😉

mhuebert09:10:12

I believe that if we AOT compile macros then it must take the :clj branch because it is java doing the compiling, but if you run macros from self-host it takes the :cljs branch

thheller09:10:14

but at least in my version you can tell the compiler :js-options {:resolve {"react" {:target :global :global "React"}}} in config 😉

mhuebert09:10:58

@thheller: that is exactly what I want to be able to specify with one of these fake empty js files

mhuebert09:10:56

Plus it is a huge pain now that cljs will just magically decide whether to take my js from what i specify in deps.clj or go read from node_modules

thheller09:10:16

indeed, thats why my stuff is completely different. 🙂

mhuebert09:10:24

Sometimes to fix a build I have to 'yarn remove <pkg>'

thheller09:10:04

(ns demo.selfhost
  (:require [cljs.js :as self]))

(defn compile-it []
  (js/console.log
    (self/compile-str
      (self/empty-state)
      "(ns my.user) (map inc [1 2 3])"
      ""
      {:eval self/js-eval}
      identity)))

thheller09:10:12

I just compiled this and it worked

thheller09:10:42

only weird thing I need to figure out is loading order since cljs.js has this weird hack for goog.require

thheller09:10:38

do you have a bootstrap method for your stuff?

thheller09:10:56

ie. before compiling anything load a bunch of stuff async?

thheller09:10:19

seems like that would be a better place to load cljs.core$macros and others

thheller09:10:36

doing it via goog.require seems bad

thheller09:10:50

or do you just not have any other macros?

thheller09:10:07

I saw yourcljs-live stuff which seems to do that kind of thing

mhuebert09:10:21

I AOT-compile everything into bundles which I load asynchronously yes

mhuebert09:10:38

Yes exactly, I think it's in the 'compile' namespace

mhuebert09:10:51

I had to do an ugly hack around goog.require too

thheller09:10:35

my current thinking is to do a normal compile for your “build”, then do a second pass that just adds everything you want for self host

thheller09:10:00

{:build-id :selfhost
        :target :browser

        :output-dir "out/demo-selfhost/public/js"
        :asset-path "/js"

        :build-options
        {:self-host
         {:module :base
          :macros [cljs.core
                   cljs.spec.alpha]}}

        :modules
        {:base
         {:entries [demo.selfhost]}}}

mhuebert09:10:22

So in that case macros are compiled using js / :cljs branch

thheller09:10:57

the question then becomes bootstrapping the whole thing

thheller09:10:19

loading everything the way closure wants to is too slow

thheller09:10:31

which is why you build cljs-live I think

thheller09:10:20

problem I have currently is that cljs.js does not really require cljs.core$macros when loading

mhuebert09:10:26

Yes, there is just one network request per bundle, organized as json file with classpath-like 'paths' as keys

thheller09:10:32

only when actually using the compile/eval

mhuebert09:10:52

That's not bad tho?

mhuebert09:10:35

Eg with Maria I want things to load on-demand because otherwise start time is too slow

thheller09:10:41

the issue is load order

thheller09:10:17

well doesn’t matter much, I just need to patch the deps a bit

thheller09:10:37

the issue is that goog.require usually is a no-op

thheller09:10:59

ie. the goog loader will ensure that the deps were actually loaded BEFORE loading the actual file

thheller09:10:20

but since cljs.js does not formally declare the dep on cljs.core$macros

thheller09:10:35

it will trigger the goog.require out-of-order

mhuebert09:10:40

*Also analysis caches are necessary for anything one wants to use in self-host

mhuebert09:10:07

Hmm. I think I manually load macros before anything else

thheller09:10:21

yeah I saw, but thats just dumping the env data to a file. thats easy.

mhuebert09:10:28

(Not all macros, just core)

thheller09:10:39

how does your build config look currently?

thheller09:10:08

is it just a normal build and then a bunch of tooling to provide the self-host things you need?

thheller09:10:51

say if I just modify shadow-cljs to properly compile macro namespaces you could work with that?

mhuebert09:10:41

by build config, do you mean like compiler options?

thheller09:10:07

lein-cljsbuild config or whatever you are using to build the live.js

thheller09:10:39

do you do something special to the build itself

thheller09:10:49

or is all that done by a library outside the build

thheller09:10:09

AFAICT you need a way to compile macros and get the analyzer data

mhuebert09:10:10

here is my cljs-live config where i package stuff to use for self-host: https://github.com/mhuebert/maria/blob/master/editor/live-deps.clj#L8

mhuebert09:10:58

so, if you added this ability to AOT compile macros, I would (attempt to) modify cljs-live to use shadow-cljs, and it would remove one step (manually doing a 2nd pass with self-host compiler).

thheller09:10:11

right so its a post-process step

mhuebert09:10:42

yes. so I need a way to get access to analysis caches,

thheller09:10:01

so what I can do for you: a) compile a list of macro namespaces, dump them in a location of your choosing

thheller09:10:14

b) also dump analysis cache of certain namespaces

thheller09:10:22

c) provide dependency information so you know when to load what

mhuebert09:10:36

can it dump all analysis caches (w/o specifying which)

mhuebert09:10:51

that sounds good

thheller09:10:20

sure it can dump everything

thheller09:10:05

transit is fine correct?

mhuebert09:10:12

is it faster to spit edn?

thheller09:10:32

spit is almost the same. read is quite a lot faster with transit

mhuebert09:10:58

ok then yes transit’s good

thheller09:10:23

hmm how does the self host compiler load resources?

thheller09:10:30

does it load resources?

thheller09:10:01

the compile process in shadow-cljs is quite different so I actually don’t know how the cljs compiler does it 😛

thheller09:10:18

ie. if you want to compile-str (ns my.demo (:require [something.foo :as x])) but something.foo is not yet loaded. what happens?

mhuebert09:10:49

you have to provide your own load function

mhuebert09:10:31

i am doing some manual tracking of loads in that namespace

mhuebert09:10:10

cljs-live bundles include a “name-to-path” key which maps namespaces to file-paths

thheller09:10:12

are there any docs on this?

thheller09:10:27

kinda hard to tell whats going on

mhuebert09:10:26

there is the readme but i am not sure it goes into that level of detail: https://github.com/braintripping/cljs-live

mhuebert09:10:36

so i think load-fn is always called when you (require) something. it is up to load-fn to (a) know if it has already been loaded or not, and if not, (b) provide a source file and analysis cache. the source file can be compiled js or uncompiled clj*

thheller09:10:40

ok, I have a plan 😉

mhuebert09:10:17

load-fn gets namespaces as input and must map that to source. so we need to know which names map to which “files”

thheller09:10:36

AFAICT you need 2 seperate builds

thheller09:10:48

one that ensures that your “app” itself can be loaded properly

thheller09:10:25

one that “bootstrap” the compiler with macro namespaces, analyzer data, etc

thheller09:10:52

build a then loads b

thheller09:10:53

yeah I think that should work

mhuebert10:10:11

and an important thing, certainly in my case but I think generally, is that these secondary builds, you might want to have a number of them and be able to load them independently / at a time of your choosing. Because they are quite large. cljs.core analysis cache + cljs.core$macros w/ cache is 900kb. the maria bundle is 1.5mb, clojure spec + deps is 1.8mb

thheller10:10:15

you have done most of the work already

thheller10:10:42

yeah I have an idea for that

thheller10:10:30

yeah keeping it separate solves a whole bunch of issues

mhuebert10:10:27

it is important that the self-host compiler / bootstrap loader does not re-evaluate stuff that was already loaded by the “app” build

thheller10:10:11

yeah, thats easy though

mhuebert10:10:48

with the right information at hand, yes 🙂

thheller10:10:43

Information: I have it all! 😉

thheller11:10:56

@mhuebert so far I create a “bootstrap.index.json” (transit) which contains infos like this

thheller11:10:10

{:type :cljs,
  :provides #{cljs.compiler},
  :requires
  #{cljs.tools.reader goog.string cljs.core goog cljs.env
    goog.string.StringBuffer clojure.set cljs.analyzer cljs.source-map
    clojure.string},
  :resource-name "cljs/compiler.cljc",
  :output-name "cljs.compiler.js"}
 {:type :cljs,
  :provides #{cljs.core$macros},
  :requires
  #{cljs.compiler cljs.core cljs.env clojure.set cljs.analyzer
    clojure.string clojure.walk},
  :resource-name "cljs/core$macros.cljc",
  :output-name "cljs.core$macros.js"}
 {:type :cljs,
  :provides #{cljs.analyzer.api},
  :requires #{cljs.core goog cljs.env cljs.analyzer},
  :resource-name "cljs/analyzer/api.cljc",
  :output-name "cljs.analyzer.api.js"}
 {:type :cljs,
  :provides #{cljs.spec.gen.alpha},
  :requires #{cljs.core goog},
  :resource-name "cljs/spec/gen/alpha.cljs",
  :output-name "cljs.spec.gen.alpha.js"}
 {:type :cljs,
  :provides #{cljs.spec.alpha$macros},
  :requires
  #{cljs.core goog cljs.analyzer.api cljs.env cljs.analyzer
    clojure.string clojure.walk cljs.spec.gen.alpha},
  :resource-name "cljs/spec/alpha$macros.cljc",
  :output-name "cljs.spec.alpha$macros.js"}

thheller11:10:47

the idea is to load this index on startup, build a runtime index out of the :requires/:provides

thheller11:10:20

then either load the compiled :output-name files in :load or the :resource-name file and compile that

thheller11:10:26

does that sound useful?

thheller11:10:48

+ the compiler will add a list of namespaces that were already loaded by the build

thheller11:10:12

so you can keep track of it easily

mhuebert11:10:33

by ‘the build’ you mean the 1st/app build

thheller11:10:25

'{:build-id :selfhost
        :target :browser

        :output-dir "out/demo-selfhost/public/js"
        :asset-path "/js"

        :bootstrap-options
        {:entries []
         :macros [cljs.spec.alpha]}

        :modules
        {:base
         {:entries [demo.selfhost]}}}

thheller11:10:30

this is how it looks for now

thheller11:10:44

in :bootstrap-options you declare which namespaces should be precompiled

thheller11:10:02

so if you want the user to be able to use reagent but you don’t use it yourself

thheller11:10:11

you just add :entries [reagent.core]

thheller11:10:17

same for :macros

thheller11:10:27

should be about the same result as your cljs-live

mhuebert11:10:40

does it follow macro dependencies? ie. if you require namespace x which requires x$macros

thheller11:10:41

not sure if it needs to pre-compile everything or just use :entries to copy the requires sources so cljs.js can compile it

thheller11:10:49

not yet but it will

mhuebert11:10:02

compiling is super slow in the browser, if things aren’t precompiled it can be multiple-second wait times

mhuebert11:10:05

on a fast computer

thheller11:10:21

ah ok so pre-compile should be the default

mhuebert11:10:29

in my opinion yes

thheller11:10:01

I think I have the basics now, I’ll try to put together something that actually works

mhuebert11:10:23

i remember encountering subtle issues with macros. normally if i wanted to use a namespace, i wanted to include the macros it uses as well. but some macros are not self-host compatible, so i added :entry/exclude and :entry/no-follow.

thheller11:10:20

hmm yeah didn’t deal with that yet

mhuebert11:10:42

in your example - we specify explicitly which macro namespaces we want, and then shadow would ensure that those macro namespaces & their dependencies are loaded

thheller11:10:42

no, not loaded. available.

thheller11:10:01

it just copies everything required into a special folder

thheller11:10:43

(ns demo.selfhost
  (:require [cljs.js :as cljs]
            [shadow.bootstrap :as boot]))

(boot/init
  (fn []
    (cljs/compile-str
      (cljs/empty-state)
      "(ns my.user) (map inc [1 2 3])"
      ""
      {:eval cljs/js-eval
       :load boot/load}
      identity)))

thheller11:10:47

looks something like this now

thheller11:10:00

the boot/init function basically is supposed the index from above

mhuebert11:10:17

so boot/load will download files as necessary from the special folder

thheller11:10:16

init is done async so the cljs.core analyzer data and macros can be loaded lazily

thheller11:10:58

still working out the details but thats the plan

mhuebert11:10:23

would it be reasonable to ask that the original source + compiled source are both copied into this dir

mhuebert11:10:53

my own use case is source lookups from the editor. but since bootstrap is used commonly for tooling i probably wouldn’t be the only one

thheller11:10:46

yes all of it is copied

thheller11:10:49

source maps as well

mhuebert11:10:06

i am liking this a lot. for Maria I could put one big long list of things into :bootstrap-options and the only constant cost is a larger bootstrap.index.json. then users can require any of those namespaces and they will be loaded on demand.

mhuebert12:10:32

is it necessary for the app that runs shadow.bootstrap/init (& /load) to be compiled together with all the stuff in :bootstrap-options, or could it ‘consume’ any bootstrap.index.json file, provided with a prefix path of where to load the files from?

thheller12:10:19

well I haven’t actually written the bootstrap init fn

thheller12:10:28

but no you could use whatever bootstrap mechanism you like

thheller12:10:10

just need a small reference fn to show how its supposed to work

thheller12:10:21

its really just about properly discovering where to find stuff

mhuebert12:10:07

and the app that uses bootstrap/init or bootstrap/load knowing what has already been loaded (by itself)

thheller12:10:30

do you have a :browser shadow-cljs build handy?

mhuebert12:10:50

mm not at the moment

thheller12:10:29

ok, let me finish this so I can show the actual example 😉

thheller12:10:09

[zilence@zpro ~/code/shadow-cljs/out/demo-selfhost/public/js/bootstrap]$ find .
.
./ana
./ana/cljs.analyzer.api.cljc.ana.transit.json
./ana/cljs.analyzer.cljc.ana.transit.json
./ana/cljs.compiler.cljc.ana.transit.json
./ana/cljs.core$macros.cljc.ana.transit.json
./ana/cljs.core.cljs.ana.transit.json
./ana/cljs.env.cljc.ana.transit.json
./ana/cljs.reader.cljs.ana.transit.json
./ana/cljs.source_map.base64.cljs.ana.transit.json
./ana/cljs.source_map.base64_vlq.cljs.ana.transit.json
./ana/cljs.source_map.cljs.ana.transit.json
./ana/cljs.spec.alpha$macros.cljc.ana.transit.json
./ana/cljs.spec.gen.alpha.cljs.ana.transit.json
./ana/cljs.tagged_literals.cljc.ana.transit.json
./ana/cljs.tools.reader.cljs.ana.transit.json
./ana/cljs.tools.reader.edn.cljs.ana.transit.json
./ana/cljs.tools.reader.impl.commons.cljs.ana.transit.json
./ana/cljs.tools.reader.impl.errors.cljs.ana.transit.json
./ana/cljs.tools.reader.impl.inspect.cljs.ana.transit.json
./ana/cljs.tools.reader.impl.utils.cljs.ana.transit.json
./ana/cljs.tools.reader.reader_types.cljs.ana.transit.json
./ana/clojure.set.cljs.ana.transit.json
./ana/clojure.string.cljs.ana.transit.json
./ana/clojure.walk.cljs.ana.transit.json
./index.transit.json
./js
./js/cljs.analyzer.api.js
./js/cljs.analyzer.js
./js/cljs.compiler.js
./js/cljs.core$macros.js
./js/cljs.core.js
./js/cljs.env.js
./js/cljs.reader.js
./js/cljs.source_map.base64.js
./js/cljs.source_map.base64_vlq.js
./js/cljs.source_map.js
./js/cljs.spec.alpha$macros.js
./js/cljs.spec.gen.alpha.js
./js/cljs.tagged_literals.js
./js/cljs.tools.reader.edn.js
./js/cljs.tools.reader.impl.commons.js
./js/cljs.tools.reader.impl.errors.js
./js/cljs.tools.reader.impl.inspect.js
./js/cljs.tools.reader.impl.utils.js
./js/cljs.tools.reader.js
./js/cljs.tools.reader.reader_types.js
./js/clojure.set.js
./js/clojure.string.js
./js/clojure.walk.js
./js/goog.array.array.js
./js/goog.asserts.asserts.js
./js/goog.base.js
./js/goog.debug.error.js
./js/goog.dom.nodetype.js
./js/goog.functions.functions.js
./js/goog.iter.iter.js
./js/goog.math.integer.js
./js/goog.math.long.js
./js/goog.math.math.js
./js/goog.object.object.js
./js/goog.reflect.reflect.js
./js/goog.string.string.js
./js/goog.string.stringbuffer.js
./js/goog.structs.map.js
./js/goog.structs.structs.js
./js/goog.uri.uri.js
./js/goog.uri.utils.js
./src
./src/cljs.analyzer.api.cljc
./src/cljs.analyzer.cljc
./src/cljs.compiler.cljc
./src/cljs.core$macros.cljc
./src/cljs.core.cljs
./src/cljs.env.cljc
./src/cljs.reader.cljs
./src/cljs.source_map.base64.cljs
./src/cljs.source_map.base64_vlq.cljs
./src/cljs.source_map.cljs
./src/cljs.spec.alpha$macros.cljc
./src/cljs.spec.gen.alpha.cljs
./src/cljs.tagged_literals.cljc
./src/cljs.tools.reader.cljs
./src/cljs.tools.reader.edn.cljs
./src/cljs.tools.reader.impl.commons.cljs
./src/cljs.tools.reader.impl.errors.cljs
./src/cljs.tools.reader.impl.inspect.cljs
./src/cljs.tools.reader.impl.utils.cljs
./src/cljs.tools.reader.reader_types.cljs
./src/clojure.set.cljs
./src/clojure.string.cljs
./src/clojure.walk.cljs
./src/goog.array.array.js
./src/goog.asserts.asserts.js
./src/goog.base.js
./src/goog.debug.error.js
./src/goog.dom.nodetype.js
./src/goog.functions.functions.js
./src/goog.iter.iter.js
./src/goog.math.integer.js
./src/goog.math.long.js
./src/goog.math.math.js
./src/goog.object.object.js
./src/goog.reflect.reflect.js
./src/goog.string.string.js
./src/goog.string.stringbuffer.js
./src/goog.structs.map.js
./src/goog.structs.structs.js
./src/goog.uri.uri.js
./src/goog.uri.utils.js

thheller12:10:19

eek thats long

thheller12:10:48

well, you get the idea /src are the actual sources. /js is the compiled code, /ana is analyzer data

thheller12:10:00

all in :output-dir + “boostrap”

thheller12:10:23

so if you app is in /js/foo.js the index will be in /js/bootstrap/index.transit.json

thheller12:10:37

probably a bit too many files, not sure if you need the goog files

thheller12:10:45

(since they are already loaded)

mhuebert12:10:49

not all of them

mhuebert12:10:26

eg. say you include quil.core under :bootstrap-options :entries, but you don’t have quil.core in your app itself

mhuebert12:10:42

and then quil.core requires some goog namespaces that aren’t required anywhere else in your app

thheller12:10:49

I can test which files are loaded by the build and skip those

thheller12:10:56

for now its fine to copy everything

thheller12:10:34

index should be small enough

mhuebert12:10:48

better to start with all the info in the index

thheller12:10:03

1.5kb gzip’d for the above

mhuebert12:10:36

the nice thing if you include everything is that this becomes decoupled from the app

mhuebert12:10:07

so different apps using shadow.boot/init can consume these dependencies remotely

thheller12:10:05

yeah that could work .. could be problematic though

thheller12:10:27

it might have issues already since shadow-cljs may sometimes compile things slightly different

thheller12:10:52

well .. it basically only applies to JS deps

thheller12:10:59

no idea how that would work anyways 😉

thheller12:10:22

lets finish this first then deal with the issues

mhuebert12:10:25

right, because js deps is no longer a matter of just including source files

mhuebert12:10:42

does shadow do its own parsing of ns dependencies?

mhuebert12:10:05

ok. IIRC the cljs.analyzer/parse-ns didn’t follow macro dependencies (because it was never including them in cljs builds) so i had to make my own modified version for that.

thheller12:10:38

shadow.build.ns-form is the ns parser based on clojure.spec

thheller12:10:51

not using parse-ns at all, ever.

thheller12:10:06

woho twbs/bootstrap should finally become usuable https://github.com/twbs/bootstrap/issues/24167 😉

thheller12:10:51

so much bootstrap goodness today 😉

thheller13:10:13

@mhuebert I just pushed the first part that should cover the most basic examples

thheller13:10:55

if you want to try master the new :bootstrap-options config entry controls everything

thheller13:10:11

:bootstrap-options
        {:entries [reagent.core]
         :macros [cljs.spec.alpha]}

thheller13:10:50

it does not yet properly follow the macros so it will only compile the macros you specify

thheller13:10:57

ie. it would not include reagent macros

thheller13:10:49

also missing is client bootstrap. maybe you can adjust yours

thheller13:10:16

shadow.bootstrap.loaded-libs is an atom which is a set of symbols matching all goog.provided names that have been loaded

thheller13:10:51

the :load fn should add to that set whenever it loads something

thheller13:10:07

gotta take care of some stuff now, will do the reference init/load later

mhuebert13:10:15

noob question, i’ve been using shadow-cljs via npm, what would the simplest way to run master be?

thheller13:10:11

clone master and run lein install

thheller13:10:31

make sure the installed version matches the npm version

thheller13:10:37

should be if you are on the latest

mhuebert13:10:40

INFO: duplicate resource cljs/js.cljs on classpath, using jar:file:/Users/MattPro/.m2/repository/org/clojure/clojurescript/1.9.946/clojurescript-1.9.946.jar!/cljs/js.cljs over jar:file:/Users/MattPro/.m2/repository/thheller/shadow-cljs/2.0.7/shadow-cljs-2.0.7.jar!/cljs/js.cljs

mhuebert13:10:30

now, The required namespace "react" is not available, it was required by "re_view/render_loop.cljs".

mhuebert13:10:07

this is because react is a symbol and not a string

mhuebert13:10:47

does shadow-cljs support /checkouts looks like yes

mhuebert14:10:21

this still throws from cljs.js:

live.js:680 goog.require could not find: cljs.core$macros
goog.logToConsole_ @ live.js:680
live.js:714 Uncaught Error: goog.require could not find: cljs.core$macros
    at Object.goog.require (live.js:714)
    at cljs.js.js:15

mhuebert14:10:29

guessing this is because of that conflict?

thheller16:10:41

I’m back … ah hehe damn.

thheller16:10:50

yeah I had to make 2 small changes to cljs.js

mhuebert16:10:31

is there a way to prefer shadow’s copy over clojure’s? (it would appear that shadow’s is being ignored?)

thheller16:10:17

simplest way for now is to copy it to one of your :source-paths

thheller16:10:04

thats why I put mine there but forgot that .jar loading rules load clojurescript before shadow-cljs so that version wins over mine

mhuebert16:10:36

ie. copy the file itself

thheller16:10:08

yeah use that for now

thheller16:10:40

wanted to work it out first before I submit patches to cljs

thheller16:10:03

I believe :dump-core toggle already exists

thheller16:10:01

simplest way to test probably is inide the shadow-cljs checkout itself

thheller16:10:45

otherwise AOT will get in your way frequently

mhuebert16:10:42

if I have this in shadow-cljs.edn:

:js-options   {:resolve {"react"      {:target :global
                                        :global "React"}
                          "react-dom"  {:target :global
                                        :global "ReactDOM"}
                          "codemirror" {:target :global
                                        :global "CodeMirror"}}}
then should i be able to just put react, react-dom and codemirror in separate script tags and (:require [codemirror :as CM])?

thheller16:10:18

(:require ["codemirror" :as CM])

thheller16:10:40

symbol is allowed but I prefer a string for JS deps

thheller16:10:46

makes it easier to separate the two

mhuebert16:10:01

i prefer it, i am just trying to get something that can work with both shadow-cljs and cljsbuild

mhuebert16:10:30

i am getting this error The required namespace "codemirror" is not available,

thheller16:10:46

technically you can just ignore all that and just access the globals

thheller16:10:17

is :js-options part of your build?

thheller16:10:24

its per build not global

thheller16:10:53

having said that I should totally support it as global too

mhuebert16:10:50

Uncaught TypeError: Cannot read property 'provide' of undefined in `shadow.js.provide(“module$codemirror”, function(require,module,exports) { module.exports=(CodeMirror); });`

thheller17:10:45

not weird, bug

thheller17:10:30

manually add shadow.js to the build for now

thheller17:10:37

:module {:entries ...}

mhuebert17:10:02

& require it from a namespace i suppose

thheller17:10:46

just put it before your code

thheller17:10:11

:modules {:main {:entries [shadow.js your.ns]}} should ensure its loaded before the codemirror include

thheller17:10:08

you are really brave for testing it on your actual project 😛

thheller17:10:45

I struggle with my test-build specifically chosen for this without any other stuff going on 🙂

mhuebert17:10:27

that worked

mhuebert17:10:22

ok, one last thing. one of my dependencies (which i control) has a javascript file that i want to include. currently i do that via deps.cljs/foreign-lib. any idea how i can get that to work?

mhuebert17:10:41

codemirror et al is loading fine now 🙂.

thheller17:10:03

in the future it will just be (:require ["/path/to/file" :as x]) with relative path support

thheller17:10:42

currently its (:require ["some-name" :as x]) and then in :resolve {"some-name" {:target :file :file "path/to/file.js"}}

mhuebert17:10:11

hmm. so, the file is in another project; do I put a shadow-cljs.edn file in there?

thheller17:10:29

nah its all build config

mhuebert17:10:31

also this reinforces the idea that :js-options would frequently be a global thing

mhuebert17:10:58

can :file be anything on the classpath?

thheller17:10:58

not really global but global-ish

thheller17:10:18

no, need to add :target :resource which checks on the classpath

thheller17:10:28

sec I can add that now

thheller17:10:11

hmm crap no I can’t 😛

thheller17:10:28

hmm I did not plan for that

mhuebert17:10:59

being able to include arbitrary js file x as a dep is something i’ve done a few times

thheller17:10:59

the JS deps stuff is all based on files, since npm uses all files and doesn’t have a classpath

thheller17:10:28

yeah the issue is that the .js files are allowed to require others

thheller17:10:56

say you include a.js and that does require("./b")

thheller17:10:03

I need to package both

thheller17:10:32

what kind of JS file is it?

mhuebert17:10:05

i have a prosemirror package which has cljs code which is an interface on top of a slew of ProseMirror stuff which i have bundled up with webpack

mhuebert17:10:35

so I have a deps.cljs:

{:foreign-libs [{:file     "js/pm.pack.js"
                 :provides ["pack.prosemirror"]}
                {:file     "js/pmMarkdown.pack.js"
                 :requires ["pack.prosemirror" "cljsjs.markdown-it"]
                 :provides ["pack.prosemirror-markdown"]}]
 :npm-deps {:prosemirror-markdown "^0.22.0"}}

thheller17:10:00

how do you bundle it togehter in webpack?

mhuebert17:10:45

now at some point I can get rid of the in-between step and just require this stuff directly from npm, however, i haven’t yet & am fairly certain that at least the non-shadow cljs stuff can’t support all the modules required by prosemirror

mhuebert17:10:08

babel-loader

mhuebert17:10:09

the file is just a bunch of es6 exports:

export {EditorView, Decoration, DecorationSet} from "prosemirror-view"
export {EditorState, Selection, SelectionRange, TextSelection, NodeSelection, AllSelection, Transaction, Plugin, PluginKey} from "prosemirror-state"
export {keymap} from "prosemirror-keymap"

export {findWrapping, liftTarget, canSplit, canJoin, ReplaceAroundStep, ReplaceStep} from "prosemirror-transform"
export {wrapInList, splitListItem, liftListItem, sinkListItem} from "prosemirror-schema-list"
export {InputRule, wrappingInputRule, textblockTypeInputRule, inputRules, undoInputRule, allInputRules} from "prosemirror-inputrules"
export {Schema, Node, Mark, ResolvedPos, NodeRange, Fragment, Slice, MarkType, NodeType} from "prosemirror-model"

import * as commandsObj from "prosemirror-commands"
export const commands = commandsObj;

import * as historyObj from "prosemirror-history"
export const history = historyObj;

import * as modelObj from "prosemirror-model"
export const model = modelObj;

thheller17:10:20

yeah, shadow can do that 😉

mhuebert17:10:17

it’s just a lot to rewrite right now 🙂. maybe i can just manually copy the compiled files and link with script tags for now

mhuebert17:10:40

later i have to go through and change every single reference to these things

thheller17:10:59

(:require ["prosemirror-transform" :refer (findWrapping, liftTarget, canSplit, canJoin, ReplaceAroundStep, ReplaceStep)])

thheller17:10:20

but yeah for now its probably easier to just copy the file into the project

mhuebert17:10:21

yeah, it will be much better than it is now

mhuebert17:10:29

it’s really awkward and messy as-is

thheller17:10:32

I did not yet account for js deps on the classpath

mhuebert17:10:57

i don’t think any js deps on classpath expect for require to work

thheller17:10:58

well js deps written in goog compatible style

mhuebert17:10:25

AFAIK cljs just copies them straight into the bundle

thheller17:10:50

yeah but I wanted require/`import` to work

thheller17:10:16

and it was too much work to build that on top of URL so its built on top of File

thheller17:10:22

but classpath jars are urls

thheller17:10:33

anyways … copy it for now

thheller17:10:46

then :resolve {"some-name" {:target :file :file "path/to/file.js"}}

thheller17:10:19

or include via script and :global

mhuebert17:10:45

ok we are making progress 🙂

mhuebert17:10:51

Cannot read property 'findInternedVar' of null at line 1 when i eval an expression

thheller17:10:17

thats because the bootstrap/init function doesn’t to anything yet

thheller17:10:24

the macros are missing basically

thheller17:10:49

you can fake it and load it via script tag

mhuebert17:10:20

oh i am not using the bootstrap/* functions yet

mhuebert17:10:30

this is using my cljs-live bundles

thheller17:10:50

yeah you somehow need to load js/bootstrap/js/cljs.core$macros.js

thheller17:10:18

I took out the require that did that automatically

thheller17:10:25

because I wanted it to be part of bootstrap

thheller17:10:01

bootstrap fn is almost done

mhuebert17:10:23

i haven’t set :bootstrap-options at all yet

thheller17:10:04

then do it 😉

mhuebert17:10:57

shadow is so fast

mhuebert18:10:46

so I am not using any of the bootstrap loader code yet

mhuebert18:10:59

i just put the core macros file in a script tag

mhuebert18:10:22

but it works with my existing bundles/loader

mhuebert18:10:44

I just had to create a namespace like so:

(ns shadow-init.init
  (:require shadow.js
            shadow.bootstrap))

mhuebert18:10:13

by not including the macros in the build, it’s ~900kb lighter. of course we have to load that anyway, but that can happen after the page is visible & first things have started to work.

thheller18:10:38

I just pushed the barebones init fn I have in mind

thheller18:10:59

it does load the correct files, just need to hook them up into the proper places

thheller18:10:12

it just evals the macro js outright, not sure thats correct 😛

thheller18:10:38

but it delays the macro and cljs.core analyzer load

thheller18:10:13

so it should be 600kb for cljs.core analyzer data, 800kb for cljs.core$macros.js

thheller18:10:58

it could be smarter and collect everything it needs to load by looking at the index data

thheller18:10:08

then load them in parallel or something

thheller18:10:21

cljs.core is easy since its just one file

thheller18:10:36

but I suppose its possible to load 10+ files in order in that way

thheller18:10:01

try is by cloning master, then run shadow-cljs watch bootstrap

thheller18:10:06

in the project dir

thheller18:10:33

I need to bump some deps … too many checkout deps 😛

thheller18:10:09

now its bumped

mhuebert18:10:34

ok i’ll be back in an hour or so

thheller18:10:19

hehe I had no idea transit-cljs still have no uri? warning fix

thheller18:10:30

now that I’m using it I constantly get the warning

mhuebert19:10:43

yeah, i checked yesterday and that’s the latest version

mhuebert19:10:01

maybe with http2 it is not a big deal to be fetching each dependency independently, since we have the index and can fetch all transitive deps in parallel.

mhuebert19:10:20

a naïve load-fn loads each dep, parses requires, then loads the next, takes forever

thheller19:10:09

this stuff really is less documented than shadow-cljs

thheller19:10:39

I can’t figure out what load-fn is supposed to do exactly

thheller19:10:54

ie. call the callback with some data?

thheller19:10:00

it is documented… just not where I was looking for it 😛

mhuebert19:10:08

a map of :cache, :source, :lang

thheller19:10:14

(defonce
  ^{:doc "Each runtime environment provides a different way to load a library.
  Whatever function *load-fn* is bound to will be passed two arguments - a
  map and a callback function: The map will have the following keys:

  :name   - the name of the library (a symbol)
  :macros - modifier signaling a macros namespace load
  :path   - munged relative library path (a string)

  It is up to the implementor to correctly resolve the corresponding .cljs,
  .cljc, or .js resource (the order must be respected). If :macros is true
  resolution should only consider .clj or .cljc resources (the order must be
  respected). Upon resolution the callback should be invoked with a map
  containing the following keys:

  :lang       - the language, :clj or :js
  :source     - the source of the library (a string)
  :file       - optional, the file path, it will be added to AST's :file keyword
                (but not in :meta)
  :cache      - optional, if a :clj namespace has been precompiled to :js, can
                give an analysis cache for faster loads.
  :source-map - optional, if a :clj namespace has been precompiled to :js, can
                give a V3 source map JSON

  If the resource could not be resolved, the callback should be invoked with
  nil."
    :dynamic true}
  *load-fn*
  (fn [m cb]
    (throw (js/Error. "No *load-fn* set"))))

thheller19:10:40

I was searching ain cljs.compiler and cljs.analyzer 😛

mhuebert19:10:27

it wasn’t documented much when i started cljs-live

mhuebert19:10:09

do you use cljs’s source-mapping functions during compile?

thheller19:10:22

so if I understand this correctly

thheller19:10:33

assuming that nothing is precompiled and only the sources are available

thheller19:10:18

load-fn should load the source of something, parse it to load the deps, then load all deps (recursively), then finally call the callback

thheller19:10:47

ah wait .. if you have the source I suppose you can call eval-str which will call back into the load-fn

thheller19:10:28

I’m so glad that everything is sync in shadow-cljs, doing stuff like this async hurts my brain

thheller19:10:45

but since pre-compiled sources exist most of the work must still be done by us 😛

thheller19:10:05

index to the rescue

mhuebert19:10:41

the append-source-map function includes the original source along with the source map, which all needs to be encoded/decoded/shipped. i have source-maps disabled for cljs-live, if we enable them here it would be worth seeing what the effect on speed and file size is with the source file left out

thheller19:10:11

didn’t do source maps yet

thheller19:10:25

well the normal build has source maps

mhuebert19:10:32

so, I think in that case load-fn does not have to parse deps, it can just return the thing for itself, but will then be called repeatedly as more calls to require() are evaluated in the source

thheller19:10:33

just not the bootstrap stuff

mhuebert19:10:59

you don’t have to eval inside load-fn

mhuebert19:10:30

you just return that map, then cljs takes it from there. a file won’t actually be evaluated until its deps have been.

mhuebert19:10:51

yeah there is absolutely no urgency on source maps, just wanted to mention that small thing

thheller19:10:06

(defn compile-it []
  (cljs/compile-str
    boot/compile-state-ref
    "(ns my.user (:require [reagent.core :as r])) (map inc [1 2 3])"
    ""
    {:eval cljs/js-eval
     :load boot/load}
    print-result))

thheller19:10:17

this is my test

thheller19:10:33

boot/load is called once, with reagent.core

mhuebert20:10:01

and it is returning the compiled source?

thheller20:10:18

ah nvm, set :type not :lang

thheller20:10:27

hmm no still only called once

thheller20:10:41

makes sense though

thheller20:10:00

it can’t extract deps from compiled js

mhuebert20:10:00

is the ana cache being returned as well?

mhuebert20:10:16

but the compiled js would contain goog.require statements

thheller20:10:58

yeah, buts thats not a viable way to load code

thheller20:10:40

ah you extract the goog.require in cljs-live and load it right?

thheller20:10:47

saw something like that

mhuebert20:10:21

only for js files, i extract goog.provide

mhuebert20:10:03

that part is a bit of a hack:

(cond-> bundle
          (or (string/starts-with? v "goog")
              (string/starts-with? k "goog")
              (and (string/ends-with? k ".js")
                   ;; for a JS file, see if a goog.provide statement shows up
                   ;; in teh first 300 characters.
                   (some-> v (> (.indexOf (subs v 0 300) "goog.provide") -1))))


          ;;
          ;; parse google provide statements to enable dependency resolution
          ;; for arbitrary google closure modules.
          ;; this is a slow operation so we don't want to do it on all files.

          (update "name-to-path" merge (let [provides (parse-goog-provides v)]
                                         (when (empty? provides)
                                           (log k ": " provides "\n"))
                                         (apply hash-map (interleave provides (repeat k))))))

thheller20:10:36

yes but you eval those files?

thheller20:10:05

ah you patch goog.require and load there

mhuebert20:10:09

i think the ana cache is how it reads deps for cljs* files

mhuebert20:10:13

that parsing of goog.provide statements happens just when i download/load a ‘bundle’, nothing is eval’d there yet

thheller20:10:21

ok. I’m just stupid … shouldn’t work so late, too many mistakes 🙂

mhuebert20:10:32

🙂 the bright side is that these problems are far__ better solved with full access to the build/compile system, eg. shadow, than trying to peck at it from the outside like cljs-live

thheller20:10:01

or not … it doesn’t seem to load deps

thheller20:10:55

load gets called once

thheller20:10:23

I return the {:lang :js :source js-code :cache ana-data}

thheller20:10:04

let me try returning the actual .cljs source

thheller20:10:14

it will probably attempt to load deps if I compile it

thheller20:10:37

(defn load [{:keys [name path macros] :as rc} cb]
  ;; check index build by init
  ;; find all dependencies
  ;; load js and ana data via xhr
  ;; maybe eval?
  ;; call cb

  ;; FIXME: needs to ensure that deps are loaded first
  (go (let [ana (<! (load-analyzer-data name))
            js (<! (load-js name))]
        (js/console.log "boot/load" name macros ana (count js))
        (cb {:lang :js :source js :cache ana}))
      ))

thheller20:10:15

gets called once with reagent.core

thheller20:10:00

but it doesn’t even seem to eval the js

mhuebert20:10:41

i put this in for :eval during debugging:

(fn [{:keys [source cache lang name]}]
  (println "Eval" {:name   name
                   :lang   lang
                   :cache  (some-> cache (str) (subs 0 20))
                   :source (some-> source (subs 0 150))})
  (js/eval source))

mhuebert20:10:51

to see when/if that is called

thheller20:10:36

yeah doesn’t get called

thheller20:10:47

(defn compile-it []
  (cljs/compile-str
    boot/compile-state-ref
    "(ns my.user (:require [reagent.core :as r])) (map inc [1 2 3])"
    ""
    {:eval
     (fn [{:keys [source cache lang name]}]
       (js/console.log "Eval" name lang {:cache (some-> cache (str) (subs 0 20))
                                         :source (some-> source (subs 0 150))})
       (js/eval source))
     :load boot/load}
    print-result))

thheller20:10:04

boot/load gets called once with reagent.core

thheller20:10:37

compile does finish since it doesn’t actually use the code

mhuebert20:10:28

what is the shadow-cljs.edn you are using w/ that example?

thheller20:10:45

clone master, shadow-cljs watch bootstrap, src/dev/demo/selfhost.cljs, open

thheller20:10:19

pushed my current state

mhuebert20:10:54

The required namespace "react" is not available, it was required by "reagent/core.cljs".

mhuebert20:10:50

oh wait, for some reason version 2.0.6 is running

thheller20:10:34

or npm install react

thheller20:10:12

I’m trying to pre-compile reagent for self-host using my JS deps

mhuebert20:10:06

ok. would be great if that error message could indicate if the missing thing is a javascript namespace

mhuebert20:10:19

(which is possible because of the string convention?)

thheller20:10:27

hmm do you have reagent 0.8 alpha?

thheller20:10:44

Its supposed to say The required JS dependency "%s" is not available, ...

thheller20:10:41

I should go to bed …

thheller20:10:53

I require reagent 0.8

mhuebert20:10:15

it’s building now

thheller20:10:14

seriously .. how does anyone understand how the self host compile works

mhuebert20:10:03

i will see if i can figure out why it’s not calling eval.

thheller20:10:53

I’ve been staring at this for an hour or so

thheller20:10:56

no idea what it does

mhuebert20:10:39

we are calling compile-str, that’s why it won’t eval

mhuebert20:10:21

change to eval-str and we get a nice loop i think

thheller20:10:03

ah nice infinite loop 😉

thheller20:10:09

loading reagent.core over and over

mhuebert20:10:26

:macros is true, but we keep returning reagent.core, which then requests reagent.core$macros again?

thheller20:10:53

ah I didn’t catch that

mhuebert20:10:16

next thing is $macros aren’t included

thheller20:10:16

ok I got that loaded now

thheller20:10:04

but now it tries to load cljs.core endlessly. damn I wish I could just say “its taken care of, just continue compilation”

mhuebert20:10:44

you can do that by returning a blank map, {:lang :js :source “”}

thheller20:10:42

haha I didn’t think of that

mhuebert20:10:02

maybe this gets into the thing, where we need to know which namespaces are loaded in the app__, ie. javascript exists on the window.namespace.variable, and return a blank map for those

mhuebert20:10:43

*blank :source, but still include :cache

mhuebert21:10:38

true strife awaits those who eval an existing namespace twice. more than one late night because of that.

thheller21:10:22

I tried adding everything to cljs.js/*loaded*

mhuebert21:10:41

i haven’t had time to look at the latest cljs resolve stuff

mhuebert21:10:19

it looks like that should work.

mhuebert21:10:47

which stuff were you adding to cljs.js/*loaded*?

thheller21:10:38

(defn set-loaded [namespaces]
  (let [loaded (into #{} (map symbol) namespaces)]
    (swap! loaded-ref set/union loaded)
    (swap! cljs/*loaded* set/union loaded)))

thheller21:10:05

thats all namespaces the app uses

thheller21:10:20

(swap! cljs/*loaded* conj ns)

thheller21:10:37

if I add to the load fn is seems to work

thheller21:10:50

hmm progress

mhuebert21:10:36

didn’t see, where were the names that the app uses?

thheller21:10:24

take a look at out/demo-selfhost/public/js/base.js

mhuebert21:10:24

oh nevermind, i am reading the comment

thheller21:10:42

last line goog.require('shadow.module.base.append')

thheller21:10:02

which is cat out/demo-selfhost/public/js/cljs-runtime/shadow.module.base.append.js

mhuebert21:10:20

when i log *loaded* it is empty

thheller21:10:34

ok, I pushed latest state. seems to work properly now.

thheller21:10:47

until it gets to create-react-class node dep

thheller21:10:10

which can’t work since I do something totally different than what the self-host compiler knows and expects

thheller21:10:43

I’ll see if I can do something the using the :source "" trick

mhuebert21:10:18

did you push to github already?

thheller21:10:01

not sure why it attempts to load 404 (Not Found)

thheller21:10:21

dammit .. because I’m …

thheller21:10:40

that file is not CLJS so it has no analyzer data 😛

mhuebert21:10:57

🙂 step by step

mhuebert21:10:18

it’s so great that you can change build settings and not have to restart the build tool

thheller21:10:11

ok one step closer

thheller21:10:21

but as expected it fails to properly load JS deps

thheller21:10:33

it does load the entries but not the deps

thheller21:10:04

but I can probably silently load those before giving control back to cljs.js

thheller21:10:28

but really this is getting out of hand, I should just take the shortcut 🙂

mhuebert21:10:22

silent loading, when you have full knowledge of the deps tree, I think should be the path forward for all types once we get the basics working

mhuebert21:10:32

otherwise we can’t load deps over the network in parallel

thheller21:10:54

given the index I can find all deps without a network request

mhuebert21:10:09

i mean downloading the cache and source files

thheller21:10:12

then load them all, eval in order, callback

mhuebert21:10:20

yes exactly

mhuebert21:10:42

for cljs as well as js

mhuebert21:10:20

so eg. the call to :load-fn with reagent.core will result in calculating all the transitive deps, downloading in parallel, eval’ing + putting into analysis cache, then returning; or, calculating + downloading all transitive deps, put in cache somewhere, then return to cljs, which will then recurse the :load-fn through all the transitive deps, but not downloading them sequentially

mhuebert21:10:09

i’m calling it a night.

thheller21:10:23

yeah I should too

mhuebert23:10:26

seq, chunked-seq?, chunk-first, count & others undefined

mhuebert23:10:26

in the shadow-cljs bootstrap demo, there were also :ns errors because the macros ns wasn’t being loaded in init, I tried adding that:

(let [idx (build-index data)
                ana-core (<! (load-analyzer-data 'cljs.core))
                ana-core$macros (<! (load-analyzer-data 'cljs.core$macros))]
            (cljs/load-analysis-cache! compile-state-ref 'cljs.core ana-core)
            (cljs/load-analysis-cache! compile-state-ref 'cljs.core$macros ana-core$macros)
            (js/eval (<! (load-macro-js 'cljs.core$macros)))
            (init-cb)
            )

mhuebert23:10:38

that removed the :ns errors but there are still undeclared vars.

mhuebert23:10:01

then I remembered that the first thing we do in Maria, before evaluating anything, is run `(require ‘[cljs.core :include-macros true]). Then our :load function takes care of all that ^^ stuff, without us doing it manually. https://github.com/mhuebert/maria/blob/master/editor/src/maria/eval.cljs#L66

mhuebert23:10:06

definitely heading to sleep now!