Fork me on GitHub
#shadow-cljs
<
2017-10-09
>
thheller08:10:12

@mhuebert I don’t think it will be possible to compile cljs.js with the bootstrap build

thheller08:10:21

its the bootstrap within bootstrap thing again

thheller08:10:10

I think I found the problem, the “host” loads cells.cell, the bootstrap then incorrectly assumes that the cells.cell$macros are loaded as well

thheller08:10:41

doh nope … I just forgot to put the analyzer info about macros into the index

thheller09:10:38

btw to make your testing easier: I suggest using lein

thheller09:10:42

:lein true
 :nrepl {:port 8703}

thheller09:10:48

add that do your shadow-cljs.edn

thheller09:10:20

then use shadow-cljs as a checkout dependency

thheller09:10:20

the nrepl port usually is random, adding one in the config makes it easier to use

thheller09:10:40

then run shadow-cljs server once and leave running

thheller09:10:01

shadow-cljs watch bootstrap browser then as usual

thheller09:10:17

using lein via shadow-cljs allows using checkouts and bypasses all AOT headaches

thheller09:10:45

the shadow-cljs command really is not well suited for testing master

thheller09:10:48

with the pushed fixed to master everything works

mhuebert09:10:12

So for cljs.js this is really just about the exclude option

mhuebert09:10:24

That we talked about the other day

thheller09:10:28

do you have an example where it is actually needed?

thheller09:10:17

I think you just added cljs.js in an attempt to fix the problem?

thheller09:10:30

or does it really end up somewhere in there naturally?

mhuebert09:10:01

I added cljs.js just to replicate the issue i have with Maria.user

mhuebert09:10:17

Now I have to look.. the repl-specials namespace is what is pulling that in. Maybe it is unnecessary

thheller09:10:46

geez why is that project split into 50 projects

thheller09:10:50

do you hate yourself? 🙂

thheller09:10:59

:source-paths ["src"
                 "test"
                 "checkouts/re_view/src"
                 "checkouts/re_view_hiccup/src"
                 "checkouts/re_view_routing/src"
                 "checkouts/re_view_prosemirror/src"
                 "checkouts/re_db/src"
                 "checkouts/cljs_live/src"
                 "checkouts/shapes"
                 "checkouts/friendly"
                 "../../lark/cells/src"
                 "../../lark/commands/src"
                 "../../lark/structure/src"
                 "../../lark/editors/src"
                 "../../lark/tree/src"]

mhuebert09:10:23

I currently have the compiler state and the eval functions all in maria.eval. I could potentially move the compiler state into another namespace

thheller09:10:11

adding the :excludes option really is not a big deal, afaict it only needs to ignore macros anyways

mhuebert09:10:46

Haha. I was having trouble getting figwheel to reload in changes in checkouts and so tried putting that all in source paths

thheller09:10:50

just want to make sure that we don’t accidentally break everything by running bootstrap inside bootstrap

thheller09:10:43

for checkouts you don’t really need to add the :source-paths, lein will pick them up

mhuebert09:10:44

Yeah. Let me try a little refactoring to allow Maria.user access to compiler state without cljs.js

thheller09:10:19

do you use cursive?

mhuebert09:10:46

^autocorrect had turned ‘figwheel’ into ‘together’ and couldn’t fix on my phone

mhuebert09:10:57

“I was having trouble getting figwheel to reload in changes in checkouts and so tried putting that all in source paths”

mhuebert09:10:52

I don’t even know if it worked, I should clean that up

mhuebert10:10:28

so with :lein true, should i just keep the dependencies in project.clj

thheller10:10:50

yes, :source-paths and :dependencies from lein will be used in that case

mhuebert10:10:04

and it will ignore the clojurescript/clojure deps

mhuebert10:10:09

or do i take those out, i got a bunch of warnings about cljs.core

mhuebert10:10:25

i thought cursive needed clojure(script) to be in project.clj to resolve things

thheller10:10:49

yeah I’m using cursive myself

mhuebert10:10:10

ok shadow-eval works for me now too simple_smile

thheller10:10:40

all :lein true means is that the JVM will be launched via lein run instead of java

thheller10:10:02

which warnings about cljs.core?

thheller10:10:00

you can also set :lein {:profile "+cljs"} which will then use the :cljs profile when launching

thheller10:10:00

so you can create a :cljs profile in project.clj and put some dependencies there is needed

thheller10:10:10

I do that for shadow-cljs itself

thheller10:10:58

ok I looked at maria.eval, seem like you really need the exclude option 😉

mhuebert10:10:27

so i thought i did, but what if I just put compiler-state into a different namespace.

mhuebert10:10:38

I think that’s all i really need for introspection

thheller10:10:00

you can always access the code by just using js directly

mhuebert10:10:06

so maria.live.state will just define a blank atom, then maria.eval will reset the atom to (empty-env) and use it

thheller10:10:27

say if you eval js/maria.eval.c_state

mhuebert10:10:43

yeah that would work too

thheller10:10:21

but maria.user.loaders also accesses eval

thheller10:10:45

is that code the user is supposed to be able to use?

mhuebert10:10:03

(load-gist ...) is a convenience function

mhuebert10:10:21

right so being able to exclude cljs.js allows us to enable eval in self-host, because we can expose a namespace that references cljs.js without following it

thheller10:10:54

its ok to follow since the “host” will always have those loaded

thheller10:10:17

just need to stop the compiler from trying to compile the macros

mhuebert10:10:18

if :lein true is set, can i still have :source-paths that are only in shadow-cljs.edn

thheller10:10:32

no, it is completely ignored in that case

thheller10:10:41

lein will manage the classpath, which means checkouts work

thheller10:10:03

shadow-cljs itself doesn’t support checkouts

thheller10:10:03

:excludes #{cljs.js} would ignore ALL macros of that namespace

thheller10:10:40

or JUST that macro ns?

thheller10:10:37

added :exclude in master

thheller10:10:03

(ns shadow-eval.user
  (:require re-view-hiccup.core
            [cells.cell :refer [cell defcell]]
            [cells.lib :as cell :refer [interval timeout fetch geo-location with-view wait]]
            [shapes.core :as shapes :refer [listen
                                            circle square rectangle triangle path text image
                                            position opacity rotate scale
                                            colorize stroke no-stroke fill no-fill
                                            color-names rgb hsl rescale
                                            layer beside above
                                            fish            ; for functional geometry demo
                                            ;; are these internal only? -jar
                                            assure-shape-seq shape-bounds bounds shape->vector]]
            [re-view.core]
            [cljs.js]))

thheller10:10:05

this compiled

thheller10:10:01

hmm it might be loading too many macros now though

mhuebert10:10:11

yeah, if there is a namespace that you know isn’t wanted in self-host, it might be useful to essentially pretend it doesn’t exist as far as dependencies go.

mhuebert10:10:24

also it will just be downloading and loading a lot of unnecessary analyzer data

thheller10:10:04

if that namespace is loaded by the “host” it won’t be downloaded

thheller10:10:31

the :bootstrap build still needs to compile everything to ensure everything is correct

thheller10:10:54

it seems to work alright as far as I can tell

thheller10:10:17

(:require-macros [cljs.core.async.macros :as m]) this compiled with the proper :exclude 😉

thheller10:10:45

just can’t use it but thats expected

thheller10:10:43

thats what I mean by too many macros though

thheller10:10:50

I don’t think the user will ever need those?

thheller10:10:53

might be worth adding to :exclude (without the $macros)

thheller10:10:19

maybe they are actually required, I can’t really tell

thheller10:10:13

ah ok found one issue

thheller10:10:39

which explains the cljs.env.macros issue

thheller10:10:07

[ :as io] this is not required, not sure why its thee

thheller10:10:57

when compiling as a macro namespace

thheller10:10:08

I still resolve :require as CLJS

thheller10:10:28

so it tries to load cljs.env.macros as CLJS, which it can’t find since its a .clj file

thheller10:10:51

maybe should change it so :require in macro files actually resolves as a macro

thheller10:10:18

would be fixed if it was macros.cljc and the io require removed in CLJS

thheller10:10:51

anyways .. exclude is fine

mhuebert10:10:52

to me this is a bit mind bending:

Even though the macros are defined in ClojureScript, they are defined in *.clj files. You can, if you wish, also define macros in *.cljc files, but when they are processed, the :cljs branch of reader conditionals will be used.

mhuebert11:10:22

we have started a conversation about macros in maria: https://github.com/mhuebert/maria/issues/151

mhuebert11:10:57

so if we exclude cljs.js, does it serve any purpose to include transitive macro dependencies of it?

thheller11:10:41

it depends if the USER is actually going to use those macros directly

mhuebert11:10:06

right. i think if they are, then you could include them as a direct dependency in the user namespace

thheller11:10:06

you can use it just fine inside maria, just everything eval’d by the user can’t use them directly

mhuebert11:10:12

but if they are a transitive dep of an excluded namespace, ignore them

thheller11:10:26

yeah I think that should be fine

thheller11:10:42

macros are far more complicated than I thought in self host

mhuebert11:10:52

that way you don’t have to specifically exclude all the transitive macro deps of cljs.js in addition to excluding cljs.js

thheller11:10:33

so really the only thing missing is [ :as io] which we can’t import anyways

thheller11:10:09

dunno the whole cljs.js namespace seems a bit hacky, it shouldn’t even attempt things like requiring cljs.core$macros or the cljs.core analyzer data

thheller11:10:50

dumping the cljs.core analzyer data into the code just seems like a bad idea to begin with

mhuebert12:10:18

ok, so with current, Maria can now compile and eval with a shadow-cljs build 🙂. of the basic tests I run (https://www.maria.cloud/gist/8d7dd5bb0294b9623af0bc7c607f46d2?eval=true), only one fails, and that is (require '[cljs.js :as cljs]). I forgot that I even do that. I’ve tried putting cljs.env.macros into the :exclude but it still throws The required namespace "cljs.env.macros" is not available, it was required by "cljs/js$macros.cljc". on compile. I’m guessing this relates to your comment above: > it tries to load cljs.env.macros as CLJS, which it can’t find since its a .clj file

thheller12:10:39

@mhuebert yeah, I need to look more into what CLJS does in this case

thheller12:10:55

when compiling a macro namespace I treat :require as a CLJS require, ie. it looks for .clj(s|c) files

thheller12:10:28

I’m not sure what is fully correct

mhuebert12:10:37

i’m also not sure

mhuebert12:10:46

going out for food

thheller13:10:23

btw @mhuebert one thing you might want to consider for maria: do the read yourself to split user input properly

thheller13:10:43

eg. if I input 1 2 3 4 5 I only get one result

thheller13:10:03

if you read and split it, you could get all 5

thheller13:10:19

and if you read you can check for defmacro and do the namespace dance automatically

thheller13:10:35

right now there only seem to be *-str functions in cljs.js though

mhuebert13:10:43

@thheller: we actually do that already in cljs-live.eval

mhuebert14:10:39

I am thinking about that namespace dance. It would be nice if evaluating defmacro in a non-macros namespace would put that into :macros in the analyser state, and then 'get-expander' would also look in cljs namespaces for macros. Since defmacro isn't currently used for anything at all in cljs namespaces AFAIK, this wouldn't conflict with any existing code, only enable better macros in self-host

mhuebert14:10:52

This is just thinking aloud, how crazy is it to think of shadow-cljs supporting compiling cljs namespaces that use defmacro? I have no idea how your builds work. The current way of doing things is complicated and an understandable but unfortunate artifact of decisions made years ago

mhuebert14:10:54

We definitely can't reasonably teach macros in Maria without doing this somehow, so incompatibility with existing clojurescript seems inevitable

thheller14:10:54

I can’t really add support for that to shadow-cljs itself since its the JVM compiler, I avoid everything self-hosted when I can.

thheller14:10:20

adding support for your use-case is different since its still just a normal CLJS compile that just adds a few things on top

thheller14:10:40

but I’m never actually running anything of the self-hosted code myself

mhuebert14:10:23

I see. So this would mean using the self-host compiler itself as part of the build

thheller14:10:41

but adding defmacro to self-hosted really should not be that hard, you just can’t use the default compile-str or other *-str fns

thheller14:10:14

in shadow-cljs macros are eval’d in the JVM

thheller14:10:39

in theory it is easy to just eval defmacro forms in the JVM instead of generating JS

thheller14:10:58

it however gets very complicated when the macros want to call non-clojure.core code

thheller14:10:21

(def foo []) (defmacro bar [] (let [x (foo)] '(whatever ~x))

thheller14:10:47

how do I syntax quote 😛

thheller14:10:32

macros are allowed to use functions normally, but they wouldn’t exist if there is only special case handling for defmacro

thheller14:10:03

I thought a lot about this before since I wanted to make defmacro easier

thheller14:10:23

but really in self-hosted mode you do not have the issue of 2 separate runtimes

thheller14:10:33

so it should be possible

thheller15:10:57

is there a trick in maria to see the actual JS that gets eval’d by a block?

mhuebert15:10:40

Yes, try command-p and search for js

mhuebert15:10:51

I forget the shortcut

mhuebert15:10:33

That was a nice improvement (refer of macros as normal)

mhuebert15:10:09

So the only way to get real clojure-style macros is to eval them in a JS environment

mhuebert15:10:31

I did this with cljs-live, used Planck

thheller15:10:54

not to eval “them” in a JS env

thheller15:10:59

eval everything in a JS env

thheller15:10:29

but even then I’m not quite sure how to get around the namespace issue

mhuebert15:10:28

so command-shift-J for view js source. i was on my phone before

mhuebert15:10:08

right, it would have to eval everything in a JS env

mhuebert15:10:28

which namespace issue?

thheller15:10:02

sec testing something, it might be pretty easy

thheller16:10:42

can you see changes I made?

thheller16:10:53

I’m not logged in or anything

thheller16:10:19

but it seems to work 😛

thheller16:10:03

(in-ns 'cljs.core)

(defn get-expander* [sym env]
  (when-not (or (some? (gets env :locals sym)) ; locals hide macros
                (and (excluded? env sym) (not (used? env sym))))
    (let [nstr (namespace sym)]
      (cond
        (some? nstr)
        (let [ns (get-expander-ns env nstr)]
          (when (some? ns)
            (.findInternedVar ^clojure.lang.Namespace ns (symbol (name sym)))))

        (some? (gets env :ns :rename-macros sym))
        (let [qualified-symbol (gets env :ns :rename-macros sym)
              nsym (symbol (namespace qualified-symbol))
              sym (symbol (name qualified-symbol))]
          (.findInternedVar ^clojure.lang.Namespace (find-macros-ns nsym) sym))

        :else
        (let [cur-ns (gets env :ns :name)]
          (or (when (get-in @env/*compiler* [::namespaces cur-ns :defs sym :macro])
                (.findInternedVar (find-ns cur-ns) sym))
              (let [nsym (gets env :ns :use-macros sym)]
                (if (and (some? nsym) (symbol? nsym))
                  (.findInternedVar ^clojure.lang.Namespace
                  (find-macros-ns nsym) sym)
                  (.findInternedVar ^clojure.lang.Namespace
                  (find-macros-ns CLJS_CORE_MACROS_SYM) sym)))))))))

(in-ns 'maria.user)

thheller16:10:23

I just hacked the get-expander* fn :else case to also check in the current ns for macros

thheller16:10:38

had to remove the reader conditionals that are there usually

thheller16:10:15

ah you can’t see my changes

thheller16:10:23

sec let me gist the whole thing

mhuebert16:10:58

if you sign in, then you can ‘Duplicate’

mhuebert16:10:04

and it will save to your own gist

mhuebert16:10:36

you can also Command-P > ‘Print to console’, it will spit the whole gist to the console as a string

mhuebert16:10:45

I tried something like this yesterday but didn’t figure out get-expander* well enough to work

thheller16:10:38

the get-expander* hack would be something you do when bootstrapping

thheller16:10:51

but with that macros in the current ns just work

mhuebert16:10:53

this is a great little patch 😀

thheller16:10:18

I understand why its not something core does by default

thheller16:10:30

but for projects like maria it would totally make sense to support this

mhuebert16:10:02

and because these are in the same namespace, macros can reference other things in the same namespace from within syntax quote, and it just works

mhuebert16:10:03

with the shadow bootstrap build, is :dump-core disabled? ie. avoids spitting the cljs core compiler state into cljs.js?

thheller16:10:12

yes always disabled, you can’t even enable it 😉

thheller16:10:15

fun to hot patch the analyzer like that 😛

mhuebert16:10:10

that’s what is so great about a live environment

mhuebert16:10:04

was there a way to enable adding timestamps/hashes to the bootstrap outputs?

thheller16:10:17

already done, shadow-cljs release bootstrap

mhuebert16:10:36

ahhh release

thheller16:10:05

didn’t seem to make much sense for dev

mhuebert16:10:41

a namespace that is :exclude‘d, I think I’d want to just load its analysis cache

mhuebert16:10:06

be back later

thheller16:10:43

what if you just (doc cljs.js/eval-str) without the require? I’m currently only excluding the cljs.js macro ns from compilation

thheller16:10:03

I do not remove the require for it

mhuebert17:10:21

that works. I see, so the analysis cache is already loaded.

mhuebert17:10:35

so you could have it referred into precompiled namespaces, but not (require ...) it or put it in an (ns ..) form

thheller17:10:07

but that could be fixed as well

mhuebert17:10:40

i think that would be useful, ensure that we can still use precompiled namespaces even if they have unfriendly macro deps?

mhuebert17:10:49

then all of the self-host-related code that i’m aware of us using with Maria will work

mhuebert17:10:38

next step will be for me to sort out the javascript deps + externs for the outer frame (an :advanced build, eval happens in a iframe isolated from auth tokens etc.) but that should be more straightforward 🙂

thheller17:10:22

@mhuebert master should have a fix for the macro deps, didn’t test it but should work 😛

thheller17:10:29

it only removes the macro require info from the index

thheller17:10:50

might be useful to emit all excludes into the index itself but just pretending that it doesn’t require the macro should be fine

thheller19:10:48

@mhuebert btw I wanted to build a UI for shadow-cljs for quite some time now. I might steal some ideas from Maria for the quick-start mode 😉

thheller19:10:41

let users start playing with code immediately and then have some kind of UI widget guide them through generating a proper build config

mhuebert20:10:05

Sounds like a good idea

mhuebert20:10:29

let me know if you’d like help or feedback

mhuebert20:10:42

@thheller I’ve pulled from master and attempted to (require '[cljs.js :as cljs]), still getting the same:

ns cljs.js$macros not available
Could not require cljs.js

thheller20:10:30

ok, I’m about finished with this other thing. will take a look after.

thheller20:10:23

(require '[cljs.js :as cljs]) this works

thheller20:10:43

did you try in https://github.com/mhuebert/shadow-eval? it works in my simple example but that doesn’t have to mean much.

thheller20:10:45

(ns my.foo (:require [cljs.js :as cljs]))
   (js/console.log cljs/eval-str)
also works

mhuebert20:10:21

still seeing the long string of cljs.core warnings in shadow-eval

mhuebert20:10:49

and ns cljs.js$macros not available

mhuebert20:10:59

shadow-cljs still on 2.0.8?

thheller20:10:56

just bumped to 2.0.9

thheller20:10:09

ah right, totally forgot about those cljs.core warnings

thheller20:10:20

never saw them before, no idea whats going on

mhuebert20:10:51

trying with 2.0.9, same errors

mhuebert20:10:56

i have to crash and have some meetings tomorrow so i won’t get back to this till later

thheller20:10:18

ok, I’ll try to reproduce. might just be related to too much version mockery and bad AOT artifacts

mhuebert20:10:59

you have cljs.js directly in entries as well as in exclude, i only have it as a transitive dep of another entry. shouldn’t make a difference i don’t think

thheller20:10:29

yeah :exclude should correctly be called :exclude-macros as its only for excluding macros

thheller20:10:55

but no if something else requires cljs.js thats basically the same as :entries [cljs.js]

thheller21:10:54

I’m not seeing those warnings

thheller21:10:19

in the shadow-eval project (or my own)

thheller21:10:39

maybe try deleting target/shadow-cljs?