Fork me on GitHub
#shadow-cljs
<
2018-11-14
>
lsnape14:11:39

Hi, is there a way I can get a list of the source files that were compiled in the build-state from within a :flush build hook?

thheller14:11:04

sure the :build-sources key has the resource-ids in dependency order

thheller14:11:33

each id will have (get-in build-state [:sources resource-id]) and [:output resource-id] for the source and generated output

lsnape14:11:56

Cool, thanks!

lsnape14:11:12

And from that I can get the files that were touched and recompiled in that incremental compile step?

thheller14:11:21

you can also check :shadow.build/build-info which has the full summary

thheller14:11:00

including a :compiles #{rc-a rc-b} set for each resource that was actually compiled

thheller14:11:32

can also use (shadow.build/resources-compiled-recently build-state)

thheller14:11:49

recently referring to compiled in last compile increment (when using watch)

thheller14:11:17

which also shows how you'd extra that info yourself

lsnape14:11:39

Amazing, sounds like just what I need. Trying it out now..

thheller14:11:20

:shadow.build/build-info is what the websocket uses to decide which code to reload and stuff so that might be most useful

lsnape14:11:36

Yep I can see (get-in build-state [:shadow.build/info :compiled]) contains exactly what I need.

lsnape15:11:50

Another related question: if I wanted to modify the source files before compilation, will writing to the source files directly be sufficient? Or do I need to update the strings in the build state?

thheller15:11:07

modify in what way? most reliable option is to just modify the files directly on disk

thheller15:11:23

otherwise the caching logic might get in your way

lsnape15:11:49

I want to change references to assets in my source files, so that they are fingerprinted.

lsnape15:11:47

e.g. [:img {:src "${img/ponys.png}"}] becomes [:img {:src "${img/ponys-HASH.png}"}]

thheller15:11:57

IMHO that should not be done by modifying the source files directly

thheller15:11:21

[:img {:src (helper/get-img-src "img/ponys.png")}]

thheller15:11:35

and then have the helper/get-img-src fn look it up from the JSON object

lsnape15:11:43

And get-img-src references a built manifest file?

thheller15:11:44

that is either present in the page set by your HTML

thheller15:11:53

or prepend generated in a hook

lsnape15:11:53

I agree that is a better solution. I've already written a library that does it the 'search and replace' way, so was planning to reuse that.

thheller16:11:32

you can modifiy sources in a :compile-prepare hook but getting correct caching is tricky enough as it is

lsnape16:11:01

Yeah, I had a feeling that I might run into trouble!

thheller16:11:57

I have plans to integrate something like this fingerprinting for assets directly but didn't get to it yet

lsnape16:11:05

I'm just thinking about how I could make the img->hashed-img-path map available to the implementation of get-img-src.

thheller16:11:34

but my plan involved this lookup map I suggested

lsnape16:11:10

yes, well I don't think it's too much work

thheller16:11:12

a) you just append it manually after compilation

thheller16:11:24

b) you do it in a hook in :compile-finish (or :flush)

thheller16:11:05

doing it via hooks is gonna be trickier since you need to tinker with some internals

lsnape16:11:08

Sure, but the compiled JS needs the updated map? IIRC if we write a file then we'll need a macro to slurp the contents of that file?

lsnape16:11:26

And that file needs compiling.

thheller16:11:48

(defn get-img-src [path] (or (goog.object/get js/your$lookup$map path) path))

thheller16:11:44

echo "\nvar your$lookup$map = {"img/ponys.png": "img/ponys-HASH.png"};" >> main.js

lsnape16:11:44

I see. Presumably we don't have to shell out to echo though?!

thheller16:11:58

yeah no shell if you do it on a hook

thheller16:11:15

appending to the final JS is easy no matter which method you use

thheller16:11:16

the idea is to just use the path directly if it was not overridden with a hashed value

thheller16:11:24

works nicely in dev

thheller16:11:35

and optimizes easily in prod

lsnape16:11:59

Cool, that makes sense. And where in the build-state can I prepend to the top of main.js?

lsnape16:11:31

Yeah, initially I was planning to use mode to either skip or fingerprint the assets.

lsnape16:11:24

There are some edge cases to take care of. e.g. changing an asset involves a recompilation of the source file that references it.

lsnape16:11:43

Actually, you have something in place already to do that for CSS files, don't you?

thheller16:11:45

no it doesn't

thheller16:11:13

or asset as in something inlined by a macro?

thheller16:11:29

or why would you recompile if an asset changes?

lsnape16:11:15

I mean, if I changed ponys.png will the browser receive a message to reload that asset?

thheller16:11:40

no image reloading does not exist (yet)

thheller16:11:49

kinda tough to do

lsnape16:11:12

Well typically we don't want to hash assets in dev mode anyway, so it's kind of a moot point.

thheller16:11:14

yeah in dev you just use the normal file paths

thheller16:11:54

just make get-img-src return the original either via a :closure-define or some logic that checks if the path was overridden

thheller16:11:05

(which it won't be in dev)

lsnape16:11:26

I'll try and knock something up. And whereabouts can I prepend the lookup map to the main.js file?

lsnape16:11:52

Yeah, we don't want to generate a junk hash-map of identities in dev mode!

thheller16:11:35

option a) :flush will be called after the file has been written

thheller16:11:40

so you just append on disk

thheller16:11:45

b) do it in :configure by basically setting :modules {:main {:prepend-js "var lookup$map = <json>;\n"}}}

thheller16:11:51

as if you had that in the build config

thheller16:11:25

(update-in build-state [:shadow.build.modules/config <module-id> :prepend-js] str "var lookup$map = " (json/write-str {"foo" "bar"}))

thheller16:11:27

something like that

lsnape16:11:33

Yeah I got ya

thheller16:11:38

can use :prepend-js or :append-js doesn't really matter

thheller16:11:52

or :prepend :append

thheller16:11:13

:prepend-js will go through :advanced so the closure compiler may be smart enough to directly inline the strings

thheller16:11:28

optimizing out the get-img-src lookup completely

lsnape16:11:30

The latter sounds preferable.

thheller16:11:08

potentially yes

lsnape16:11:46

One advantage of the dumb regex search-and-replace approach is the same mechanism works for assets referenced in index.html and CSS files. Luckily for us we generate the page frame in hiccup and CSS in garden. Still, we need to generate a manifest file for use on the server.

thheller16:11:44

sure that works too

thheller16:11:04

in :compile-prepare you can modify the source

thheller16:11:52

just probably need to disable caching since it otherwise won't invalidate properly

richiardiandrea18:11:01

@thheller it seems we have another problem like yesterday in our build D:\home\.shadow-cljs\builds\az\dev\out\cljs-runtime\goog.debug.error.js

richiardiandrea18:11:12

az is the name of target :azure-app

richiardiandrea18:11:33

it seems like it is again trying to skip a folder

richiardiandrea18:11:43

and read from the parent of our project

thheller18:11:46

which path are you using in the config?

thheller18:11:57

just take out ./ and it should work

richiardiandrea18:11:57

:app-dir "azure/build"

richiardiandrea18:11:17

could it be somewhere in the code?

thheller18:11:32

check whats in the output file

richiardiandrea18:11:34

(checking in the meantime)

thheller18:11:22

beginning to regret trying to the the output dir clean

richiardiandrea18:11:07

we can tweak no prob...let's see where the problem is 😄

richiardiandrea18:11:54

I am checking the output file but I don't see anything like the SHADOW-IMPORT like yesterday

thheller18:11:32

hmm? are you checking the correct file?

richiardiandrea19:11:44

@thheller so this is what my colleague sees:

var SHADOW_IMPORT_PATH = __dirname + '/../../../.shadow-cljs/builds/az/dev/out/cljs-runtime';
if (__dirname == '.') { SHADOW_IMPORT_PATH = "/Users/user/github-ep/laputa/commit-event-fn/.shadow-cljs/builds/az/dev/out/cljs-runtime"; }
global.$CLJS = global;

richiardiandrea19:11:34

like yesterday, the first points outside the project folder

richiardiandrea19:11:43

but this time we don't have ./

richiardiandrea19:11:59

I see in the code this happens:

{:exports fn-map
                                    ;; FIXME: can't have a {:cljs some.ns/fn} function when using cljs dir
                                    :output-to (str app-dir "/cljs/shared.js")}`

richiardiandrea19:11:52

so that looks good to me

thheller21:11:49

@richiardiandrea you posted a d:\home windows location and the code snippet above is clearly not windows

thheller21:11:57

so I'm guessing the code wasn't compiled on that machine

thheller21:11:13

dev builds are only meant to run on the machine they were compiled on

richiardiandrea21:11:14

uhm ok thanks, there must be something going on here

richiardiandrea21:11:34

the snippet above came from a deployed lambda, this latest from a laptop

richiardiandrea21:11:46

something is not right, I will get a better understanding first

richiardiandrea21:11:51

then get back to you

thheller21:11:59

you should only ever deploy release compiled files

thheller21:11:04

never compile or watch

thheller21:11:31

if its trying to access anything in cljs-runtime it was not a release build

richiardiandrea21:11:49

yep it seems like something like deploy a compiled thing happened

richiardiandrea21:11:13

@thheller it looks like the culprit on my colleague's MacOsX machines is:

<- Cache read: cljs/core.cljs (151 ms)
-> Compile CLJS: cljs/core.cljs
------ ERROR -------------------------------------------------------------------
 File: jar:file:/Users/user/.m2/repository/org/clojure/clojurescript/1.10.439/clojurescript-1.10.439.jar!/cljs/core.cljs
NullPointerException:
    cljs.compiler/emit/fn--3144 (compiler.cljc:189)
    clojure.lang.Atom.swap (Atom.java:37)
    clojure.core/swap! (core.clj:2344)
    clojure.core/swap! (core.clj:2337)
    cljs.compiler/emit (compiler.cljc:187)
    cljs.compiler/emit (compiler.cljc:182)
    shadow.build.compiler/eval12942/fn--12943 (compiler.clj:238)
    clojure.lang.MultiFn.invoke (MultiFn.java:234)
    shadow.build.compiler/default-compile-cljs/fn--12969 (compiler.clj:289)
    shadow.build.compiler/default-compile-cljs (compiler.clj:288)
    shadow.build.compiler/default-compile-cljs (compiler.clj:274)
    clojure.core/partial/fn--5739 (core.clj:2617)
    shadow.build.compiler/do-compile-cljs-string/fn--12930 (compiler.clj:227)

richiardiandrea21:11:29

it compiles fine on linux though

thheller21:11:27

thats a version conflict

thheller21:11:45

1.10.439 requires 2.7.X

thheller21:11:16

the errors above indicates old shadow-cljs version with new CLJS

richiardiandrea21:11:36

ok now that makes sense

richiardiandrea22:11:01

my colleague did not do yarn install after updating to 2.7.3

thheller22:11:25

that doesn't matter when using deps.edn? (or lein)

richiardiandrea22:11:39

we are using yarn shadow-cljs for launching things but I am interesting in knowing more about deps.edn case

thheller22:11:08

I assume you are using deps.edn because the version conflict above can't happen with pure shadow-cljs.edn

thheller22:11:04

it enforces the CLJS version used based on the version of shadow-cljs in package.json

thheller22:11:24

if you use deps.edn that can't be enforced so whatever you have in there will be used

richiardiandrea22:11:43

oh I see what you mean

haywood22:11:44

oh yea, do I not need to declare it in package.json if I'm adding shadow to my deps? had been cargo culting that without knowing if I needed it

richiardiandrea22:11:52

yeah ClojureScript is taken from deps.edn

richiardiandrea22:11:11

so you are saying I should get rid of that in deps.edn basically

thheller22:11:32

@haywood yes. it is recommended so your project doesn't rely on some global install in the system

haywood22:11:01

oh, but I guess if I remove it from package.json, my npm run scripts will stop working

thheller22:11:43

@richiardiandrea no. I guess its fine. forgot that shadow-cljs injects itself into deps.edn on startup

thheller22:11:42

@haywood I'm confused. you SHOULD have it in your project. do not remove it.

thheller22:11:26

@richiardiandrea I was just wrong. it actually does matter that the proper version is installed via yarn/npm

👍 4
richiardiandrea22:11:04

oh ok good to know thanks

thheller22:11:51

just keep it all in shadow-cljs.edn so you don't get such issues 😉

richiardiandrea22:11:29

I know we should move away from using yarn shadow-cljs but haven't had the time to figure out the replacement

richiardiandrea22:11:03

ideally we should be able to do clojure -A:shadow watch as a replacement

thheller22:11:34

["-m" "shadow.cljs.devtools.cli"] for whatever the args key way

thheller22:11:39

:main-opts?

thheller22:11:01

but that won't properly handle the server mode so thats not really recommended

richiardiandrea22:11:21

uhm ok then good that you've just told me that 😄

thheller22:11:53

the npm package detects if a server is running and talks to that directly if it is

thheller22:11:10

the clj script will just start a new jvm every time

richiardiandrea22:11:48

oh no we don't want that 😄

thheller22:11:25

could maybe add a clojure version that does that check but it would be way slower and launcher a full JVM just to connect to a remote socket is a bit overkill

richiardiandrea22:11:18

yeah actually for one-off tasks in CI we don't want a shared JVM anyways

kniarzyn23:11:58

HI. I am using shadow-cljs with re-frame. What can I do to prevent reseting app-db after each file chenge?

thheller23:11:09

it should never reset unless you do it?

kniarzyn23:11:25

Yes. It keeps reseting after each file change. Don't know where to look for cause.

thheller23:11:15

well lets start with your setup. do you have any :after-load or :before-load hooks configured?

kniarzyn23:11:34

I have autoload

kniarzyn23:11:50

sets to true

thheller23:11:02

that defaults to true so doesn't matter

thheller23:11:51

what does the browser console say when you change a file?

thheller23:11:06

it should print at least one load JS message

thheller23:11:11

there are a couple example repos here and plenty of them use re-frame https://github.com/thheller/shadow-cljs#examples

thheller23:11:42

maybe that can help comparing the configs

kniarzyn23:11:52

shadow-cljs: load JS herman/pages/observations.cljs browser.cljs:25 shadow-cljs: load JS herman/views.cljs browser.cljs:25 shadow-cljs: call herman.core/mount-root browser.cljs:25 shadow-cljs: call herman.core/init

thheller23:11:50

alright. so what does herman.core/mount-root or herman.core/init do? are you sure both should be called on reload?

kniarzyn23:11:38

No. I think herman.core/init should not be run

thheller23:11:08

you probably have :dev/after-load metadata on it?

kniarzyn23:11:47

Only this ^:export

thheller23:11:08

then maybe :after-load herman.core/init in your build config?

thheller23:11:43

must be set somewhere otherwise shadow-cljs would not call it

kniarzyn23:11:29

You are great. Thank you.

kniarzyn23:11:18

It is working now as I want.

👍 8