Fork me on GitHub
#sci
<
2021-11-19
>
pmooser09:11:24

Can sci valuate a defmacro form in clojurescript directly?

pmooser09:11:22

I think I'm going to see if it is practical to use sci instead of the bootstrap build we're currently using in my project. Initial tests are looking good but I have more work to do.

borkdude09:11:25

one other benefit is that SCI can also work with advanced compilation, so the rest of your libs will be small

pmooser09:11:48

Yeah, that's definitely one of my reasons as well.

pmooser09:11:01

The biggest challenge for me isn't really sci's fault, but it's that you can't enumerate cljs namespaces.

pmooser09:11:33

So I need to come up with a solution for that on my own and maintain a list of namespaces (and then walk them, to give sci access to the vars).

borkdude09:11:48

you can probably generate this with a macro right?

pmooser09:11:09

Hmm I'm not sure I understand what you mean ... how would a macro help?

borkdude10:11:08

@pmooser A macro is invoked in the JVM, you can use that phase to enumerate namespaces and then generate the CLJS code for the SCI config

pmooser14:11:18

That's an interesting idea except I'm not sure exactly at what phase all of the cljs namespaces will be visible to the clj macro (since it's not like they exist as actual clj namespaces).

pmooser14:11:44

Maybe there's something very simple I'm missing.

borkdude14:11:13

ClojureScript has an analyzer API you can use from Clojure

borkdude14:11:39

It contains all-ns too

pmooser14:11:48

Ah ok - that is a good hint. I will give it a look - I've never dug into it, and know very little about it. But thank you so much!

borkdude14:11:53

and this analyzer api you can use from a macro

borkdude14:11:17

Having said that, I often just enumerate things manually because that always works ;)

pmooser14:11:58

Ok last question - to experiment, should I just be able to require that cljs.analyzer.api and experiment in a clj namespace ?

borkdude14:11:17

good question. I haven't used this myself a lot, but I will try it out now :)

pmooser14:11:10

Don't take up your time doing this, please - I can experiment and read the code and stuff. And as you say, manually enumerating isn't a bad option. Or I could probably make my own ns macro with some side-effects that does the enumeration for me, if I only care about my own namespaces.

borkdude15:11:46

No worries, I'm curious about this myself too.

borkdude15:11:58

When I add a macro in .clj file, e.g. I added one in sci.core:

(require '[cljs.analyzer.api :as api])
#?(:clj
   (defmacro foo []
     (println (api/all-ns))
     nil))

borkdude15:11:14

And then in a CLJS repl:

cljs.user=> (require '[sci.core :refer-macros [foo]])
nil
cljs.user=> (foo)

borkdude15:11:30

I get a ton of namespaces:

(sci.impl.opts cljs.env.macros sci.impl.macros cljs.tools.reader.impl.commons sci.impl.types cljs.tools.reader.edn

pmooser15:11:03

Ok perfect. So I guess we can rely on the analyzer in our cljs macros and all the namespaces have already been analyzed or whatever. Thank you, borkdude.

borkdude15:11:35

To be specific: this is in normal CLJS and the macro is running in the JVM, calling the cljs.analyzer.api in the JVM. Then all-ns enumerates all CLJS namespaces create so far, after having already loaded/compiled those.

borkdude15:11:25

So if I do this:

#?(:clj
   (do
     (require '[cljs.analyzer.api :as api])
     (defmacro foo []
       (prn (api/ns-publics (first (api/all-ns))))
       nil)
     ))

borkdude15:11:56

I will get back a map of all public CLJS "vars" of the first namespace:

{default-classes {:name sci.impl.opts/default-classes, :file "/Users/borkdude/Dropbox/dev/clojure/babashka/sci/src/sci/impl/opts.cljc", :line 47, :column 1, :end-line 47, :end-column 21, :meta {:file "/Users/borkdude/Dropbox/dev/clojure/babashka/sci/src/sci/impl/opts.cljc", :line 47, :column 6, :end-line 47, :end-column 21}, :tag cljs.core/IMap},

borkdude15:11:20

so this way you can get the names of all the public vars of all the namespaces at compile time

borkdude15:11:27

and then use them to generate CLJS code

borkdude15:11:34

I should probably document this

pmooser15:11:15

This is perfect - I think it's a great trick potentially. When you said before that all-ns enumerates all namespaces created "so far", I'm not exactly sure what that means - maybe just so far in terms of that point in the compilation or macroexpansion of all the namespaces?

borkdude15:11:56

I expect that when you don't load any namespaces, it doesn't go out and read stuff from disk.

borkdude15:11:14

So you first have to load namespaces and then call all-ns to see what namespaces exist

borkdude15:11:33

Similar to how all-ns in clojure works

pmooser15:11:32

Yes, makes sense. I find cljs macros so confusing (compared to clj). I'm still puzzling out certain issues, but this will certainly work! Thank you again.

borkdude15:11:35

yes, it's confusing because they run at compile time in the JVM but then target CLJS code

pmooser15:11:45

Yea but I mean like ... ok, here's a macro:

(defmacro cljs-namespaces
  []
  (let [namespaces (map ns-name (ana/all-ns))]
    `(list ~@namespaces)))

pmooser15:11:55

That returns a list of strings, where I'd expect symbols.

pmooser15:11:21

So I always find things a bit more "weird" even than in normal macros, which are sometimes substantially weird to begin with.

pmooser15:11:15

Anyway - you've been a great help, so thank you and have a nice weekend!

borkdude15:11:29

ns-name always returns a string

borkdude15:11:38

all-ns already returns a list of symbols

pmooser15:11:07

I misunderstood the docs.

borkdude15:11:23

ns-name accidentally works here

pmooser15:11:29

Yeah, I just need to quote the symbols, because in the macro the symbol gets expanded to the actual namespace object itself.

borkdude15:11:38

since it calls .-name on the object and doing this on a symbol returns a string

pmooser15:11:53

I misunderstood the doc string for ns-name, since it ends with "a symbol", but it clearly means the ns argument is a symbol. And yeah, it all makes sense.

pmooser15:11:15

(Like most things in software, when it seems like something crazy is happening, it's usually my fault)

borkdude15:11:58

very recognizable ;)

borkdude15:11:20

I think I've got something:

cljs.user=> (def ns-map (copy-ns clojure.edn (sci.core/create-ns 'clojure.edn nil)))
#'cljs.user/ns-map
cljs.user=> (def read-string (get ns-map 'read-string))
#'cljs.user/read-string
cljs.user=> (read-string "{:a 1}")
{:a 1}

pmooser15:11:34

Hmm! Very nice.

pmooser15:11:03

I'm still struggling a bit in cljs macro universe, but I know this will all eventually work.

borkdude15:11:20

This is the impl:

(defn copy-ns-fn [ns-publics-map sci-ns]
  (reduce (fn [ns-map [var-name var]]
            (let [m #?(:clj (meta var)
                       :cljs (:meta var))
                  no-doc (:no-doc m)
                  doc (:doc m)
                  arglists (:arglists m)]
              (if no-doc ns-map
                  (assoc ns-map var-name
                         (new-var (symbol var-name) #?(:clj @var
                                                       :cljs (:val var))
                                  (cond-> {:ns sci-ns
                                           :name (:name m)}
                                    (:macro m) (assoc :macro true)
                                    doc (assoc :doc doc)
                                    arglists (assoc :arglists arglists)))))))
          {}
          ns-publics-map))

#?(:clj
   (defmacro copy-ns [ns-sym sci-ns]
     (macros/? :clj `(copy-ns-fn (ns-publics ~ns-sym) ~sci-ns )
               :cljs (let [publics-map (cljs.analyzer.api/ns-publics ns-sym)
                           publics-map (zipmap (map (fn [k]
                                                      (list 'quote k))
                                                    (keys publics-map))
                                               (map (fn [m]
                                                      {:name (list 'quote (:name m))
                                                       :val (:name m)
                                                       :meta (select-keys (:meta m) [:arglists
                                                                                     :no-doc
                                                                                     :doc])})
                                                    (vals publics-map)))]
                       ;; (prn publics-map)
                       `(copy-ns-fn ~publics-map ~sci-ns)))))

borkdude15:11:31

It works for both clojure and clojurescript

borkdude15:11:15

I'll add it to sci.core ...

borkdude15:11:06

or maybe sci.experimental? or a doc page first ;)

borkdude15:11:10

let's do a doc page first ;)

pmooser15:11:21

Thank you for that ! Sure, maybe being cautious is best.

pmooser15:11:35

I'll give it a try on my side as well while trying to banish the bootstrap build from my application.

pmooser15:11:38

I've never seen macros/? before but I can imagine what it does.

borkdude16:11:49

It comes from macrovich which I vendored in sci

pmooser16:11:58

Very neat.

borkdude16:11:00

it allows you to target environments during compilation time

pmooser16:11:12

Ok, gotta run to the market. Thank you again!

borkdude17:11:27

@pmooser I added copy-ns to the sci.core namespace now.

(defmacro copy-ns
    "Returns map of names to SCI vars as a result of copying public
  Clojure vars from ns-sym (a quoted symbol). Attaches sci-ns (result
  of sci/create-ns) to meta. Copies :name, :macro :doc, :no-doc
  and :argslists metadata."
    ([ns-sym sci-ns] `(copy-ns ~ns-sym ~sci-ns nil))
    ([ns-sym sci-ns _opts]

borkdude17:11:14

It still requires you to provide the name of the namespace.

(copy-ns 'clojure.edn (sci/create-ns 'clojure.edn))

borkdude17:11:21

It should work in both CLJ and CLJS.

borkdude17:11:04

Added `copy-ns` to the sci.core namespace now.

(defmacro copy-ns
    "Returns map of names to SCI vars as a result of copying public
  Clojure vars from ns-sym (a quoted symbol). Attaches sci-ns (result
  of sci/create-ns) to meta. Copies :name, :macro :doc, :no-doc
  and :argslists metadata."
    ([ns-sym sci-ns] `(copy-ns ~ns-sym ~sci-ns nil))
    ([ns-sym sci-ns _opts]
Usage:
(copy-ns 'clojure.edn (sci/create-ns 'clojure.edn))
It's a macro to provide the same interface on CLJS as in CLJ.

mkvlr18:11:09

does this work for macros as well?

mkvlr18:11:29

ah yes it does, should read properly first. Thanks for adding it!

borkdude18:11:23

it works for macros in CLJ, but not for macros in CLJS since they are (normally, not-self-hosted) defined in the JVM environment, so SCI/CLJS cannot see them.

borkdude18:11:48

at least, I think so

borkdude19:11:28

So this is quite interesting:

(ns foo.bar
  #?(:cljs (:require-macros [foo.bar :refer [foo]])))

(defmacro foo []
  `(println :hello))
(ns foo.foo
  (:require [foo.bar] :reload
            [sci.core :as sci]))

(def sci-ns (sci/copy-ns 'foo.bar (sci/create-ns 'foo.bar)))
$ clj -Sdeps '{:deps {org.clojure/clojurescript {:mvn/version "1.10.879"}}}' -M -m cljs.main -re node
ClojureScript 1.10.879
cljs.user=> (require '[foo.foo] :reload)
nil
cljs.user=> foo.foo/sci-ns
{foo #'foo}
cljs.user=> ((get foo.foo/sci-ns 'foo))
(cljs.core/println :hello)
cljs.user=> (sci/binding [sci/print-fn *print-fn*] (sci/eval-string "(foo.bar/foo)" {:namespaces {'foo.bar foo.foo/sci-ns}}))
:hello
nil

borkdude19:11:00

So it even seems to work for macros if they don't do weird compile time things in the JVM

borkdude19:11:00

Oh, if you put those in reader conditionals then they aren't copied :)

(defmacro foo []
  #?(:clj (prn (System/getProperty "user.dir")))
  `(println :hello))

borkdude21:11:13

I changed the syntax. it's now `

(sci/copy-ns foo.bar (sci/create-ns 'foo.bar))
for reasons. I also introduces options; {:include [foo] :exclude [bar]}

borkdude21:11:27

for including or excluding trouble-some vars

phronmophobic21:11:37

Is there a reason why it only copies the :macro ,`:doc` ,and :arglists metadata keys?

borkdude21:11:50

because those are usually the only ones I'm interested in... do you have a reason to copy more?

borkdude21:11:55

line and column and file information may be misleading if you copy that

borkdude21:11:22

(btw, I enjoyed the defn episode.. you rock!)

😊 1
phronmophobic21:11:47

I use it for membrane in the mobiletest project

phronmophobic21:11:01

since I use metadata for the component library

phronmophobic21:11:31

It's not a huge deal, but it was somewhat of a surprise.

borkdude21:11:35

we can make metadata configurable as well

borkdude21:11:45

what was the surprise?

phronmophobic21:11:58

that the metadata was missing

borkdude21:11:16

it depends how you copied those vars right?

borkdude21:11:27

did you use sci/copy-var?

phronmophobic21:11:45

theoretically, I could consider replacing my implementation with the builtin one, but not without the metadata

borkdude21:11:00

yes, but this was after your surprise. I'm trying to learn what you used at the moment of your surprise

phronmophobic21:11:05

I used this example code:

(reduce (fn [ns-map [var-name var]]
            (let [m (meta var)
                  no-doc (:no-doc m)
                  doc (:doc m)
                  arglists (:arglists m)]
              (if no-doc ns-map
                  (assoc ns-map var-name
                         (sci/new-var (symbol var-name) @var
                                      (cond-> {:ns fns
                                               :name (:name m)}
                                        (:macro m) (assoc :macro true)
                                        doc (assoc :doc doc)
                                        arglists (assoc :arglists arglists)))))))
          {}
          (ns-publics 'foobar))

borkdude21:11:57

ok, that's the code I based the new sci/copy-ns on

borkdude21:11:23

we can make a configurable metadata function which default to identity, dunno

phronmophobic21:11:47

yea, I was checking out the implementation and recognized it! 😄

borkdude21:11:50

one consideration is that in some envs you don't want to copy the docstrings for example

borkdude21:11:58

for smaller JS bundles

borkdude21:11:04

so it has to be configurable anyway

phronmophobic21:11:06

yea, that makes sense

phronmophobic21:11:55

bundle size for common sci targets like graalvm and cljs both make sense to exclude unnecessary/misleading metadata

phronmophobic21:11:19

I might also suggest a 1-arity call to copy-ns that defaults to (sci/create-ns ns-sym) as the sci-ns

borkdude21:11:40

yeah, I also thought of that. would it be ok if you could pass a set of metadata keys or an option to select all? since this is a macro the options are kinda static

borkdude21:11:48

it's a macro because of CLJS basically

borkdude21:11:33

I need to be able to exclude certain vars at compile time in the macro in CLJS

borkdude21:11:37

for problematic macro things

borkdude21:11:45

I guess if you choose all metadata you can postprocess the vars as well (could expose alter-meta! in the sci API)

borkdude21:11:16

Oh I guess it already works with regular alter-meta!

user=> (meta (doto (sci/new-var 'foo) (alter-meta! assoc :foo 1)))
{:foo 1}

👍 1
phronmophobic21:11:31

it doesn't look like my original ns->ns-map got checked into git

borkdude23:11:10

@smith.adriane

(defmacro copy-ns
    "Returns map of names to SCI vars as a result of copying public
  Clojure vars from ns-sym (a symbol). Attaches sci-ns (result
  of sci/create-ns) to meta. Copies :name, :macro :doc, :no-doc
  and :argslists metadata.
  Options:
  - :include: a seqable of names to include from the namespace. Defaults to all.
  - :exclude: a seqable of names to exclude from the namespace. Defaults to none.
  - :copy-meta: a seqable of keywords to copy from the original var meta.
    Use :all instead of a seqable to copy all. Defaults to [:doc :arglists :macro].
  - :exclude-when-meta: seqable of keywords; vars with meta matching these keys are excluded.
    Defaults to [:no-doc :skip-wiki]"

phronmophobic23:11:33

I probably won't get a chance to look at it soon, but looks :thumbsup:

borkdude23:11:26

just pushed to master. let me know if this works, I'll be back tomorrow-ish