Fork me on GitHub
#shadow-cljs
<
2023-01-27
>
dvingo00:01:42

Hi, I have a bit of an out there request/idea.. So malli supports emitting clj-kondo configs for function schemas (https://github.com/metosin/malli#clj-kondo) which will write the config to the proper kondo location on the filesystem. I would like to get this to work for clojurescript function schemas as well. The issue is that the schemas can include Vars which must be resolved at runtime by a JS vm, which also means we don't (directly) have access to the filesystem (this is for browser targets). Right now the not so good solution is to just give the user a console.log of the kondo config and ask them to paste the contents into the kondo config file. So... I'm wondering if there is a way to use the shadow.cljs.devtools.client API (or another, I'm not sure) in the browser combined with a custom websocket event handler that will pass the kondo config, which is just data, to the shadow clojure process which can then write it to the filesystem. The goal would be to include this as an optional addon for malli users so they can get kondo static analysis in clojurescript files. Thanks!

thheller07:01:28

you can technically use shadow.remote to talk to the runtime directly

thheller07:01:51

so you could create an external tool that also connects to shadow.remote and then gets the data

thheller07:01:43

not many docs on this subject though

thheller07:01:52

right now only websocket is supported

thheller07:01:17

but every remote interaction shadow-cljs does is done over shadow.remote and its all accessible to anyone, this includes REPL, hot-reload, etc

dvingo16:01:27

very nice, thanks for the info. I'll try to figure out how to go about it based on the shadow codebase.

thheller17:01:14

just like :clj-eval there is :cljs-eval, just need to find the correct :to via :request-clients https://github.com/thheller/shadow-cljs/blob/master/src/dev/shadow/remote_example_ws.clj#L155

thheller17:01:59

so you could either do it on top of eval

thheller17:01:27

or implement some client side code, but given that it'll be mostly macro driven I guess eval is required either way

thheller17:01:19

if you know the runtime id you can also just use shadow.cljs.devtools.api/cljs-eval

dvingo17:01:12

amazing, thanks for the example - I was going down the route of trying to use runtime.shared/add-extension with a new op like :write-malli-config. I don't think there's any macros involved. After your suggestion to do instrumentation at runtime I refactored it to do so https://github.com/metosin/malli/blob/master/src/malli/instrument.cljs . so at runtime all the schema data is available and a simple data transform is run on it and then printed: https://github.com/metosin/malli/blob/master/src/malli/clj_kondo.cljc#L208 So my hope was I could do something like

(shadow.remote.runtime.shared/call @runtime-ref 
  {:op :write-malli-config :data kondo-config})
and then have a listener on the clj side to simply write to file

dvingo18:01:14

ok so I have it working with the method I was thinking, I just guessed at a few pieces, but the communication is happening clj side, loaded by including this ns in a shadow build hook:

(defmethod worker/do-relay-msg ::write-kondo-config
  [worker-state msg]
  (println "Got kondo config" msg)
  worker-state)
cljs side:
(shared/call
    (deref client-shared/runtime-ref)
    {:op ::write-kondo-config
     :to env/worker-client-id
     :data (get-kondo-config)}
    {})

thheller19:01:56

(shadow.cljs.devtools.api/cljs-eval :the-build "(whatever.ns/get-kondo-config)") and then write the result to disk?

dvingo19:01:38

oh man, that's so cool, ha - yea that's perfect! I can put that in a compile finish hook and I think that's it! Thanks Thomas

dvingo21:01:50

it's working fairly well, but there is an off-by-one problem/non-determinism at play with hot-reload. The hook I have runs on :flush and then invokes cljs-eval but sometimes the data returned contains the old values from the previous compilation. So I think the cljs-eval is happening before the latest code is loaded. Ideally I wouldn't have to update the cljs side of things and could do this all from the build hook, so was wondering if there is a way to determine if the latest code was loaded in the client and not only written to disk.

thheller09:01:33

well the server side implementation does not synchronize with the client side. it just sends messages

dvingo19:01:14

got it, I just used a simple counter on the client side and busy loop until it gets increment on the jvm side to know when the code is loaded. Thanks again for the help!

thheller20:01:55

I'm still not sure what you are doing, do you have code to share?

dvingo20:01:16

oh haha, yea this is what I got working: https://github.com/metosin/malli/pull/829

dvingo16:01:02

Malli has a feature that takes a function schema and converts it to equivalent clj-kondo config and writes that to the kondo config directory. We want the same capability in cljs but the constraints are: • we need to evaluate the cljs code because the schemas may contains Vars which can only be resolved at runtime • once we have the output of the schemas-> clj-kondo config we need to persist it to the filesystem so given those constraints I realized that shadow-cljs is already doing this communication and can we utilize that somehow? and that's how I landed on this solution Hope that helps clarify what the goal is.

thheller17:01:43

and where or how do you generate it for CLJ?

dvingo17:01:19

in CLJ: • all schemas are collected and then transformed to kondo config and then saved to disk that happens here: https://github.com/metosin/malli/blob/cfbc8eaa05bedb1e7e7b132d52bbf2e61b143f92/src/malli/dev.clj#L35

thheller18:01:27

so this is done once on start?

dvingo18:01:35

and then again via that watch on the https://github.com/metosin/malli/blob/cfbc8eaa05bedb1e7e7b132d52bbf2e61b143f92/src/malli/dev.clj#L32 so as you dev and change any function schemas the new kondo config is output

thheller18:01:41

as far I can tell you already have the preload ns calling instrument! right?

thheller18:01:21

then your previous attempt might be better?

thheller18:01:53

ie. the (defmethod worker/do-relay-msg ::write-kondo-config [worker-state msg]... )

thheller18:01:28

and in the preload ns (defn ^:dev/after-load kondo-transfer! [] (whatever-to-send-message-to-relay))

thheller18:01:46

then you don't need a hook at all

dvingo18:01:22

and that do-relay-msg would be in the malli clojure codebase?

thheller18:01:21

could be yeah. in the preload ns you could ensure its loaded via :require-macros [malli.shadow-helper]

dvingo18:01:20

I see - I think that would work! from the user's perspective they would then only need to add the preload. And users not using shadow, would be unaffected by this

thheller18:01:51

or could just remove that part of the code

thheller18:01:20

I also have a late-cljs feature in a branch for a couple months now

thheller18:01:31

which basically just lets you annotate a ns to be compiled last

thheller18:01:38

which would be perfect for this 😛

thheller18:01:53

since you could just make the preload part of mall, without needing to add all the requires

dvingo18:01:16

yesss! that was my next question - how to make this a library only concern and not need the user to deal with it

dvingo18:01:18

if (defn ^:dev/after-load kondo-transfer! [] (whatever-to-send-message-to-relay)) this lives in the library in a ns that cannot include the user's code, will it always run after the user's code?

thheller18:01:10

well it runs after all code is loaded which is enough right?

thheller18:01:03

but not sure if the instrument! call is relevant?

thheller18:01:20

nvm. that also executes before after-load

dvingo18:01:16

from a user's point of view they're using ! in an after-load hook and that is where the latest schemas are picked up after hot reload. So this kondo preload would have to run after that, which I'm not sure about the order of multiple after-load hooks

thheller18:01:51

I thought they are using the preload ns only?

thheller18:01:16

the order is somewhat unreliable, it should be in dependency order but isn't guaranteed to be

dvingo18:01:49

"the preload ns only" meaning the one in the library?

dvingo18:01:57

and not their own?

thheller18:01:36

why is (md/start!) not in the preload?

dvingo18:01:48

the idea is that a user will call ! however they like - could be in a preload but doesn't have to be - it also takes an options map which we want to make visible to the user

thheller18:01:49

I don't really know how any of this works, so I'm just guessing what parts do

thheller18:01:09

I assume the goal should be to keep malli.dev.cljs out of the build entirely for release

thheller18:01:17

so best way to achieve that would be the preload?

thheller18:01:39

(ns com.my-org.client.preload
  {:dev/always true}
  (:require
    [] ;; must require all namespace here that potentially get instrumented
    [my.app.util]
    [malli.dev.cljs :as md]
    [malli.dev.shadow-hooks]
    [malli.instrument :as mi]))

(mi/instrument!)
(md/start!)

thheller18:01:42

something like that?

dvingo18:01:44

yea agreed there - I think we're saying there is a preload ns the library provides which will deal with the kondo transmission, but the user will have to provide their own separate preload to include the namespaces in their app to ensure the preload runs last

thheller18:01:06

yes, right now

dvingo18:01:42

just need the start:

(ns com.my-org.client.preload
  {:dev/always true}
  (:require
    [] ;; must require all namespace here that potentially get instrumented
    [my.app.util]
    [malli.dev.cljs :as md]
    [malli.dev.shadow-hooks]
    [malli.instrument :as mi]))

(defn {:dev/after-load} dev []
   (md/start!))

thheller18:01:44

(ns malli.dev.shadow-instrument
  {:shadow.build/late-cljs true
   :dev/always true}
  (:require
    [malli.dev.cljs :as md]
    [malli.dev.shadow-hooks]
    [malli.instrument :as mi]))

(mi/instrument!)
(md/start!)

thheller18:01:56

if I ever add the late-cljs part 😉

thheller18:01:39

although for your case that isn't even relevant much

dvingo18:01:45

but looking at this I think another option is to tell the user to just invoke this call:

(ns com.my-org.client.preload
  {:dev/always true}
  (:require
    [] ;; must require all namespace here that potentially get instrumented
    [my.app.util]
    [malli.dev.cljs :as md]
    [malli.dev.shadow-hooks]
    [malli.instrument :as mi]))

(defn {:dev/after-load} dev []
    (md/start!)
    (malli.shadow-kondo/emit!))

thheller18:01:50

since you don't do stuff in macros anymore right?

dvingo18:01:02

the schema collection is still in a macro

dvingo18:01:11

which does need to get the latest code

thheller18:01:23

ok so the instrument! is the macro?

dvingo18:01:04

emits something like:

(malli/collect [long list of all namespaces])
(malli/instrument!)

dvingo18:01:09

the instrument is all in js

thheller18:01:33

that is confusing. what emits the collect?

dvingo18:01:40

so the user-facing api is just !

thheller18:01:18

so the start is the macro

thheller18:01:26

why is the instrument! call in the preload then?

thheller18:01:32

or is that just old docs?

dvingo18:01:01

probably old docs - is that in their now?

thheller18:01:27

I looked at the PR link you sent

dvingo18:01:49

hmm, let me double check

dvingo18:01:21

ooh yea sorry I overlooked that, that is incorrect

thheller18:01:02

yeah, I'd recommend to move the start! into the preload

dvingo18:01:13

right that's what that should say

dvingo18:01:35

and inside a function with dev/ater-load right?

thheller18:01:47

(ns user.app.shadow-instrument-preload
  {:dev/always true}
  (:require
    []
    [malli.dev.cljs :as md]
    [malli.dev.shadow-hooks]
    ))

(md/start!)

thheller18:01:06

the :dev/after-load hook lives in .shadow-hooks

thheller18:01:14

not required for the user to define it

thheller18:01:35

user code otherwise doesn't contain anything

thheller18:01:49

unless I'm missing something that should be enough

dvingo18:01:11

i think so - if this runs after their latest code changes that should work

thheller18:01:26

thats what the [] will ensure

thheller18:01:00

if thats the :init-fn ns it will ensure that the preload is actually compiled after those

thheller18:01:11

preload is a bad name, doesn't mean its actually preloaded just injected into the build

thheller18:01:03

also none of this is needed then

dvingo18:01:05

got it - ok yea that should work then. and then the malli side can be a preload like you suggested

dvingo18:01:21

yea I updated the docs with that PR to suggest the preload strategy first

thheller18:01:29

and fwiw could have shortened without all the CLJ code

dvingo18:01:13

yea, if you know a way around that that'd be awesome to know

thheller18:01:17

{:main
 {:target :browser
  :output-dir "resources/public/js/main"
  :asset-path "js/main"
  :dev {:modules {:main {:init-fn com.my-org.client.entry/init}}}
  :release {:modules {:main {:init-fn com.my-org.client.release-entry/init}}
            :build-options {:ns-aliases
                            {malli.dev.cljs malli.dev.cljs-noop
                             com.my-org.client.malli-registry com.my-org.client.malli-registry-release}}}
  :closure-defines {malli.registry/type "custom"}
 ,,,

dvingo18:01:52

didn't know you can change the init-fn for release

thheller18:01:42

can override anything, only trouble is vectors since they just append not replace

thheller18:01:57

but init-fn just replaces

thheller18:01:20

but this entire switching becomes unnecessary with the preload calling start

thheller18:01:36

since none of the dev code is included by the user code, only the build config

dvingo18:01:42

exactly - thanks for the info on that

dvingo18:01:44

this is really helpful. I'll try out the setup you suggested using the multi-method and clean up those docs. I really appreciate the help on this.

Sam Ritchie19:01:46

is there a way to trigger an npm install for cljs dependencies from shadow.cljs.devtools.api/watch? or maybe from (shadow.cljs.devtools.server/start!)?

Sam Ritchie21:01:43

(npm-deps/main {} {})
got the job done!