ah, right macros have analyzer data too

@mhuebert I finished the proper resource loading and that definitely works. can’t quite work out why it complains about those missing vars

I suspect I generate macros incorrectly

WARNING: Use of undeclared Var cljs.core$macros/seq at line 2 is the warning I get

but it should use cljs.core/seq as its not a macro but a fn

either incorrect macros or not setting some compiler state properly

analyzer data for everything is definitely loaded

Macro loading can be weird. I think even self host macros can be in .clj files, as long as no java code is called

I fixed all the macro stuff

they are defintely all loaded (including analyzer data)

Hmm

they just generate weird code

must be missing something I’m supposed to set

Is it happening at read time, syntax quote symbol resolution?

maybe? haven’t tracked down the dynamic vars

cljs.js/patch-alias-map not quite sure what that does yet

;; when compiling a macros namespace that requires itself, we need
              ;; to resolve calls to `my-ns.core/foo` to `my-ns.core$macros/foo`
              ;; to avoid undeclared variable warnings - António Monteiro

cljs/analyzer.cljc

that sounds like the problem

I’m a bit lost. Gotta do some other work stuff. will try more later

ah.. I see. when shadow is running the analyzer, compiling macros for self-host, is it taking the :clj path? haven’t looked if the 2nd pass is running in clojure or clojurescript I see, bootstrap.clj runs in clj-mode, so that reader conditional expr is ignored.

re FIXME: actually load it all, not just core - loading analyzer caches on init - not sure I understand the intention exactly, but FWIW I think loading/decoding ana caches is something better done at the time the relevant source files are required vs. all at once. (because this can also be slow, these ana caches can be huge.) so for example, the ana cache for cljs.spec.alpha would be downloaded and loaded at the same time as cljs.spec.alpha is required by source code evaluated by the self-host compiler (even if cljs.spec.alpha is already in the app build)

boot/load does not load namespaces that are already loaded

that also means it does not load analyzer data for those

so basically boot/init needs to do one load for the “missing” analyzer data

run the bootstrap build from master, it should log quite a bit about whats happening

it does load everything correctly (I think)

I’m just not setting some analyzer data I’m supposed to set

https://www.maria.cloud/intro

does this expose the compiler state atom somewhere?

or do you have a bootstrap example handy with just CLJS tooling?

the analyzer data looks correct to me but I have no idea what cljs.js/patch-alias-map does and cljs calls it all over the place

I never do, so thats my prime suspect currently

yes, its at maria.eval/c-state

https://dev.maria.cloud/gist/a7eb0b08359e4d50ca920b67564f2cd8?eval=true

yeah I understand what’s happening & have it running locally, have been toying with cljs.analyzer

ah you have a data visualizer, cool

it is a work in progress, not perfect 🙂. but you can click the left-brackets to collapse

one suggestion: for maps, attempt to sort the keys

oh, good idea

I do it in devtools console stuff

what I meant is that instead of a no-op, load should load the ‘missing’ analyzer data only when a namespace is required

makes a huge difference when debugging maps

that way you don’t have to do it all in the beginning

so the load function, for namespaces that are already loaded, returns only the cache

or a blank js as source

in addition to the cache

hmm yeah that could work as well

and then sets the namespace as ‘fully’ loaded, so the next time it runs, it is ignored

need to rewrite the loader a bit but possible

I’ve played with these settings back and forth. in a reasonably sized project this stuff really matters

delaying as much as possible makes sense

;; FIXME: this should probably do something async
          ;; otherwise it will block the entire time 60 or so files
          ;; are eval'd or transit parsed

already left a note there to turn the load into something that is async

yeah

so the browser gets a chance to breathe

in the demo it already loads 60 files in one go

well the download is parallel and async

but the eval isn’t

I believe Planck uses a lazy-map for cljs.analyzer/namespaces

so the ana cache is decoded hte first time it is read

need to finish this work stuff first

bbl

ok. i am setting up another project with shadow-cljs. a few of us are using maria at clojurebridge this weekend so it needs to be as stable as possible 🙂

could these warnings/resolutions be related to clojure.core/cljs.core aliasing? (probably not, more likely related to self-require..)

that took way longer than expected 😛

back from fighting some horrible css

🙂 i spent half the afternoon fighting with webpack trying to get sass and postcss autoprefixing to play together

webpack would be a delight. I have this legacy system some designer built that uses compass from like 100 years ago

hehe. haven’t heard anyone mention compass in a long time

too much compass weirdness to use sass directly

one day I’ll replace that sucker 😉

getting an example project set up now but taking a break for dinner

I have narrowed it down

it must be the macro generation

the macro file is full of

new cljs.core.Symbol("cljs.core$macros","seq","cljs.core$macros/seq",-1873195024,null) which should probably be cljs.core/seq

ah

so, I just set-up a test macro ns with a simple dummy macro

(ns demo.macro)

(defmacro foo [& body]
  `(seq ~@body))

that then contained new cljs.core.Symbol("demo.macro$macros","seq","demo.macro$macros/seq",-1873195024,null)

rewrote that manually and it works

now I need to figure out what controls that 😛

woho I might have it

https://github.com/reagent-project/reagent/blob/master/examples/simple/src/simpleexample/core.cljs

works, with react via my JS deps 🙂

only one issue, reagent doesn’t declare that it uses clojure.string. gotta get that fixed.

hmm weird. it has proper requires in master

nvm the demo doesn’t have the proper require 😛

spinning up a demo project, getting

[:browser] Build failure:
The required namespace "cljs.env.macros" is not available, it was required by "cljs/js$macros.cljc".

thats good news!

hmm strange. now I’m scared to relaunch my process 😛

sorry not that project

my own project

ah did you delete target/shadow-cljs before trying? it might just be AOT issues

ah no

didn’t do that

still need to fix some issues but I’m amazed that reagent just works

yeah that is fantastic

how does reagent require react?

ah i see, just [react :as react]

so shadow is pulling that from node_modules

yes

no :foreign-lib involved

code evaled in the browser probably will not be able to access those JS dependencies

i’ve put mine here, https://github.com/mhuebert/shadow-eval.git

still getting the same error

I’ll check it out

might be the missing cljs/js.cljs override?

oh wait

yes, i remembered to put it in the project but not in the entry

sorry

do you use yarn or npm?

hmm

yarn usually

hmm no the override is correct

always yarn add shadow-cljs to the project itself

i am not using node_modules here for anything though

ok

I’m surprised it compiles at all

since it won’t find any of the foreign libs

from a yarn global add ...

oh

the global install is just for convenience

so you can use shadow-cljs CLI command with yarn run

if its part of the project, it will always use the version from the project

and ensure that the project has the proper dependencies available

anyways .. not sure what the problem is

digging deeper

oh

i had clojurescript in the dependencies that was project.clj

ah wait

:entries [shadow-eval.core]

this is not required, as the app will load it anways

it just needs “extra” entries that the app doesn’t load on startup

ie. support files

it should still find the macros though

there might be an issue with self-host using self-host 😛

wait, what do you mean about shadow-eval.core

doesn’t every build need at least one entry point?

err every module

oooh

you might be right about this self-host using self-host

taking it out fixes it

here is my cljs-live entry for maria.user:

{:name            maria.user
                  :entry           #{maria.user}
                  :provided        #{maria.frames.live-frame}
                  :entry/no-follow #{maria.eval
                                     cljs-live.eval
                                     cljs.js
                                     cljs.compiler
                                     maria.editors.code
                                     maria.repl-specials
                                     cljs.core.match
                                     maria.views.repl-specials
                                     maria.live.analyze
                                     maria.live.source-lookups}
                  :entry/exclude   #{cljs.pprint}}

right I didn’t add the no-follow support yet

i forgot about that.

basically the :bootstrap-options :entries just controls which namespaces are allowed to be used by self-host eval

and that can’t use self-host itself it seems … would probably make my head explode if that worked 😛

yeah. so the :entry/no-follow excludes a thing, and does not look at its transitive deps at all (it does not specifically exclude all of its transitive deps, because they could be used elsewhere), there are some namespaces (like all those listed above) that I just found to be problematic with self-host

cljs.js is in there 😉

precisely 🙂

so I can probably add a :bootstrap-options {:exclude #{cljs.js other syms} to control that

to prevent it from compiling those

so the app fails with a bunch of React is not defined

that is because its still trying to use the React global

eg. (def ^:dynamic *create-element* js/React.createElement) in re-view-hiccup.hiccup

thats not great.

hmm I should add an option that lets expose some of my js deps as globals

to ease the transition

i ended up doing that because cljs compiler things have been changing so frequently, and the :foreign-libs was so unpredictable

finally gave up

yeah, totally understandable

^^ an option to expose js deps as globals would be super helpful i think, i was trying to figure out how to make all my libraries possible to compile with shadow-cljs and also cljsbuild et al

wait ha

you already configured everything to use the global

just didn’t include it 😉

yep

but

no nevermind. for now i will just include script tags.

if you want something that is compatible with both tools

you should probably use ["react" :as react]

BUT that needs updated cljsjs.react, I’m not sure those are available yet

do you have a script tag I can test with?

seems to work

doesn’t work quite right, init doesn’t seem to complete

there are some other errors on that page, just a sec

cleaning it up

do i still need to include :macros in bootstrap options

no

only if you want macros that aren’t otherwise included

ah, the load fn just silently stops if a namespace is not available

thats not great 😛

if you pull the demo, it works now

put in the value-viewer

yes, works. 🙂

did you fix the thing with for loops?

yes

🙂

(defonce _
         (boot/init #(swap! state assoc :ready true)))

now, if i run (require '[re-view.core :as v :refer [defview]]) it says :invalid-refer, but if I run (require-macros ...) first, it loads

I’ll fix that so you can call init more than once without issue

ok

(`defview` being a macro)

right, thats the analyzer data issue

this one ;; FIXME: actually load it all, not just core

since the shadow-eval.core namespace already uses re-view.core it doesn’t properly load the analyzer data for it

at least I think so

hmm. so currently we set cljs/*loaded* (or whatever) with the names of preloaded namespaces, so boot/load isn’t called at all for them

yes, but it should only skip loading the JS. it should still load the analyzer data

but if we don’t put all of those into *loaded*, but instead just keep that knowledge to ourselves, then when we see them for the first time in boot/load we return the cache + blank js

but that evals the JS

it only evals what we return, the blank helper js

no I control the eval of the deps

hm

and always return the blank helper?

yes

I load the JS and analyzer data, put it into the correct places, then callback

it just stops too early in cases where the JS is loaded but the analyzer data is not

should be an easy fix though

thought for a moment I could drop init entirely

but this is triggered by load and not init

but I can’t since cljs.js may not always call the load fn

i think cljs.js will call the load fn if we don’t put all the app namespaces into *loaded*

then we can get rid of init

(i think)

well, still need init for the index

can load the index in load if its not loaded

ah right

hmm nope

if I just take out the init and then do something like (+ 1 2) I get an error

[ 
:div "Error: " "#error {:message \"Could not eval [test]\", :data {:tag :cljs/analysis-error}, :cause #error {:message \"Cannot read property 'findInternedVar' of null at line 1 \", :data {:file nil, :line 1, :column 1, :tag :cljs/analysis-error}, :cause #object[TypeError TypeError: Cannot read property 'findInternedVar' of null]}}"
 ] 

so definitely need init

without init do we need (require '[cljs.core :include-macros true])

I think init is safer to ensure that everything is actually ready

should be simple fix anyways, just really need to grab some food 🙂

is there a reason to use both loaded-ref and cljs/*loaded*?

re, yes. will matter later.

ok. i made a quick example https://github.com/thheller/shadow-cljs/compare/master...mhuebert:patch-1

ignore the fact that loaded-ref was also taken out, i didn’t know its purpose

this compiles/runs for me

(the bootstrap demo)

ech i’ll put loaded-ref back in there

here is a much cleaner view: https://gist.github.com/mhuebert/4eaf8eef3f704f7905545e3530c3e519/revisions

it should never load the :js for a pre-loaded ns, but otherwise the compiler doesn’t need to know they exist until they are required

pushed the fix

the loaded-ref is there because of code splitting, well thats the plan

will make sense if I actually enable it 😉

right now there will be a problem if you use 2 modules and shadow.bootstrap ends up in the second module

the first will blindly call shadow.bootstrap.set_loaded without having declared a proper dependency on it

so it will fail

not important yet

but the fix should take care of all loading issues

ok taking a look

it might load a bit too much now in init

since it loads analyzer data for everything loaded by the app

but that could be delayed until load

ah not its all correct

cljs.core$macros depends on all that stuff, not the app

where is it loading all of that?

hmm something doesn’t quite work

did you see the gist above? the demo totally worked without doing any extra loading in the beginning, just making sure not to load :js for the pre-loads

it worked, but it was missing analyzer data for things like clojure.string

how so?

boot/load should be called for clojure.string, analyzer data loaded but no js evaled

you didn’t take out this check https://gist.github.com/mhuebert/4eaf8eef3f704f7905545e3530c3e519/revisions#diff-4ebb24b82089d1f69766f74ec1eed471R88

so if something was loaded it never ended up in the list

its correct now though

that check stays

but the ns doesn’t get put into loaded-ref until loaded

but (require '[re-view.core :as v]) doesn’t seem to do anything

right .. *loaded*

sec

right … now I get it

the only reason i am focusing on this is because i used to have cljs-live behave like this and load everything up-front, and with a project that goes beyond tiny example, it can be really slow

don’t worry .. it will not be slow, doesn’t matter which size

🙂

once I make load async properly

but yes I get your point

analyzer data is now checked against the actual compiler state not the loaded-ref (or *loaded*)

https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/bootstrap.cljs#L148

should be far more accurate

would it make sense if I let you pass in the compiler state to init/load?

yes

i think so

yeah seems wrong to declare that global

you can have multiple compiler states in an app, perform namespace surgery on namespaces, reset them, etc. so checking the compiler state for whether or not to load an analysis cache makes perfect sense

I’d have to do quite a refactor if I had to ‘move’ maria’s compiler state, it is referenced from so many places

ok (require '[re-view.core :as v :refer [defview]]) works now

pushed the fix

does a bit more work but should ensure that everthing is properly loaded

hmm, I think there are still some funky things with syntax quote

these are all in syntax quoted forms where i would expect them to resolve to cljs.core

actually that one, self i think is a function parameter

this is the macro being called, in the cells.cell$macros namespace:

(defmacro cell
  "Returns an anonymous cell. ..."
  ([expr]
   `(~'cells.cell/cell nil ~expr))
  ([key expr]
   `(let ~lib-bindings
      (~'cells.cell/cell* ~(cell-name key) (fn [~'self] ~expr)))))

i’ve pushed the current state

(to shadow-eval)

heading to sleep now

oh…… AOT issue

🙂

it works!

hehe cool

whats that? quil?

a thing i wrote

https://dev.maria.cloud/cells

well i wrote the cells lib, friends wrote the shapes library and collaborated on that intro doc 🙂

just pushed a bit of cleanup, both boot/init and boot/load now take the compiler-state as their first argument

also did the first async run queue

excellent

but its a bit too naive for our purposes

if code was written on paper, i would be preparing for a festive burning of the cljs-live repo 😉

hehe

this maria.cloud stuff is seriously cool

I would hold off on the burning, not quite sure we covered everything yet

still missing the excludes

although not sure they are actually needed

how would you approach it w/o them?

eg. the reason that you can do (require [maria.eval :as e]) and inspect the compiler state is because I’m able to exclude cljs.js. I could attempt to reorganize the whole app around dividing lines based on which namespaces happen to not freak out in self-host, but that could get rather tedious

hmm right

yeah thats a good use case

there are quite a few macro namespaces, for instance, which will never work in self-host (or not unless the authors make a serious effort), but namespaces that use those macros can nevertheless be used in self-host without problem (provided we can exclude those problematic ones)

eg. you can have a namespace which uses core.async, and run those functions (compiled) in self-host no problem, because macroexpansion already happened, despite core.async not working in self-host. (mfikes has a fork that does, but many libs don’t use it)

*core.async macros

hmm yeah but inside self-host you can’t use go

you can use the functions of cljs.core.async but not actually do anything without go

right. you can’t use the macro, but you can use a function that was defined using go, because it has already been expanded/compiled.

yes

(maybe some day this will become obselete: https://github.com/mfikes/andare)

ok, I’ll add :exlude taking a set of namespace symbols for macros that won’t be included

yeah problem with andare is that it uses the same namespaces

so it overrides cljs.core.async

can’t really do that

should have used other namespaces

https://github.com/mhuebert/shadow-eval/blob/master/src/shadow_eval/core.cljs#L38

that seems tedious, should I add an option to init that can load stuff?

i was wondering about that

probably would make sense

ok, i am crashing. good night!

gn8 🙂

(shadow.bootstrap/init compile-state-ref {:load-on-init '[shadow-eval.user]} init-cb)

damn forgot about excludes again 😛

after sleep