Fork me on GitHub
#shadow-cljs
<
2021-12-03
>
residentsummer11:12:03

Hi! I was doing some setup for local development to minimize the chances of accidentally releasing a build with wrong configuration and encountered a strange behaviour. I've tried to override :closure-defines from my build hook with values it reads from local file, but updates were not propagating into the build. Here are the snippets: app/dev.clj

(defn load-dev-vars
  {:shadow.build/stage :configure}
  [{:keys [:shadow.build/mode] :as build-state} relative-path]
  (let [conf-path (File. (:project-dir build-state) relative-path)]
      (if (and (= :dev mode) (.exists conf-path))
        (let [overrides (edn/read-string (slurp conf-path))]
          (println "Applying vars override:" overrides)
          (update-in build-state [:shadow.build/config :closure-defines]
                     merge overrides))
        build-state)))
shadow-cljs.edn
{...
 :builds
 {:front
  {:target :browser
   :modules {:main {:init-fn app.system/go}}
   ;; Dev vs Prod build vars. Override via extra config.
   :build-hooks [(app.dev/load-dev-vars "local-defines.edn")]
   :closure-defines {app.config/api-root "/api"}}}}
Inspecting build-state showed that overrides were applied as expected, but build contained original values... Looking at the shadow-cljs source I found that at :configure stage defines are injected into other part of build-state - :complier-options. Changing the path from [:shadow.build/config :closure-defines] to [:compiler-options :closure-defines] indeed fixed the issue. Why not just add my defines into :dev :closure-defines in shadow-cljs.edn - I don't want to commit local configuration into version control and/or stash the changes on every pull. With an additional file, added into git and docker ignores, it's much easier to manage. My question: is it expected behavior or just code ordering issue in configure stage of shadow-cljs? Maybe it was not supposed that users would change build configuration in hooks?..

thheller15:12:52

at the point the :configure hook runs the build config is already applied and as such modifying it has no effect

thheller15:12:30

you can update [:compiler-options :closure-defines] directly instead

residentsummer15:12:34

Hello. Yeah, thanks, I've figured that out already 🙂

residentsummer15:12:07

Just wondered if it's a bug or I was not supposed to change build config in hooks.

thheller15:12:56

I answered that. Intended to be like that

Franklin14:12:59

Hello I'm getting this error when I try to call postMessage on a web worker

subs.cljs:94 Uncaught ReferenceError: document is not defined
    at subs.cljs:94
    at worker.js:1457
    at Array.forEach (<anonymous>)
    at Object.env.load (worker.js:1453)
    at worker.js:1489
(anonymous) @ subs.cljs:94
(anonymous) @ worker.js:1457
env.load @ worker.js:1453
(anonymous) @ worker.js:1489

Franklin14:12:21

the webworker itself looks as follows

(ns quagga.components.dataview.worker)

(defn init []
  (set!
   js/self.onmessage
   (fn [^js e]
     (js/console.log "message received" (.. e -data)))))

Franklin14:12:51

I've added it to the modules like this

:worker
          {:init-fn quagga.components.dataview.worker/init
           :depends-on #{:shared}
           :web-worker true}

Franklin14:12:21

I've also added it to the html with a script tag of type "text/js-worker"

Franklin14:12:58

I'm calling postMessage as follows on the repl

(let [worker (js/Worker. "/js/quagga/worker.js")] (.. worker (postMessage "hello world")))

Franklin14:12:43

not sure what I'm missing

p-himik14:12:35

What comprises the module :shared? Given the stacktrace, there's some subs.cljs file in there that tries to reference js/document, which does not exist in web workers.

Franklin15:12:32

shared actually has nothing, it's defined like this

:shared {:entries []},

Franklin15:12:07

I think it's because all my other modules depend on shared

Franklin15:12:14

I tried to make the web worker module depend on :depends-on #{} but then it says

ExceptionInfo: two modules without deps, please specify which one is the default

p-himik15:12:47

Oh, interesting. Maybe I misunderstand something then. Shadow-cljs docs say: > You can leave the `:entries` of the `:shared` module empty to let the compiler figure out which namespaces are shared between the other modules. But that subs.cljs still comes from somewhere even though your web worker ns clearly doesn't depend on it. @U05224H0W Could it be a bug?

thheller15:12:58

are there other modules that may cause stuff to be moved into :shared that the worker doesn't like?

thheller15:12:16

can't say without your full config

Franklin15:12:40

:devtools
         {:preloads [day8.re-frame-10x.preload],
          :watch-dir "resources/public/css/"},
       :modules
         {:home
            {:depends-on #{:shared}, :entries [quagga.components.home.init]},
          :organization
            {:depends-on #{:shared},
             :entries [quagga.components.organization.init]},
          :settings
            {:depends-on #{:shared},
             :entries [quagga.components.organization.settings.init]},
          :shared {:entries []},
          :tabbed-dataview
            {:depends-on #{:shared},
             :entries [quagga.components.dataview.init]},
          :user-management
            {:depends-on #{:shared},
             :entries [quagga.components.organization.users.init]}
          :worker
          {:init-fn quagga.components.dataview.worker/init
           :depends-on #{:shared}
           :web-worker true}},
       :output-dir "resources/public/js/quagga/",

Franklin15:12:17

there probably are other modules bringing in code that's trying to use js/document

thheller15:12:22

yeah so so since you only have :shared anything that two of the other modules may use will get moved to :shared

thheller15:12:34

end as such also end up getting loaded by the worker

thheller15:12:45

make an extra :base module or so

thheller15:12:02

that the :shared and :worker depend on but have all the others only depend on :shared

thheller15:12:09

both with :entries [] is fine

p-himik15:12:39

Ah, I see. At least to me, the wording in the documentation made it seem like :shared has namespaces that are shared by all other modules that depend on :shared. Thanks!

thheller16:12:26

yeah a namespace can only ever be in one module so if say something from :home and :settings share a required namespace the only place it can go is :shared.

p-himik17:12:01

> a namespace can only ever be in one module Now that you said it out loud, it became obvious to me as well. :)

Franklin18:12:33

unfortunately, I'm still getting the same error with this setup

:modules
         {:home
            {:depends-on #{:shared}, :entries [quagga.components.home.init]},
          :organization
            {:depends-on #{:shared},
             :entries [quagga.components.organization.init]},
          :settings
            {:depends-on #{:shared},
             :entries [quagga.components.organization.settings.init]},
          :shared {:entries []
                   :depends-on #{:base}},
          :base {:entries []}
          :tabbed-dataview
            {:depends-on #{:shared},
             :entries [quagga.components.dataview.init]},
          :user-management
            {:depends-on #{:shared},
             :entries [quagga.components.organization.users.init]}
          :worker
          {:init-fn quagga.components.dataview.worker/init
           :depends-on #{:base}
           :web-worker true}}

thheller19:12:41

are you sure none of the namespaces used by the quagga.components.dataview.worker ns end up including the problem ns?

thheller19:12:19

generate a build report, it'll tell you why something is where it is on mouse over

Franklin05:12:36

there's a line in worker.js that looks like this

SHADOW_ENV.load({}, ["goog.debug.error.js",..."day8.re_frame_10x.inlined_deps.re_frame.v1v1v2.re_frame.subs.js","shadow.cljs.devtools.client.browser.js","shadow.module.base.append.js","quagga.components.dataview.worker.js","shadow.module.worker.append.js"]);
(I have eliminated some code) could this mean that re-frame-10x might be adding some undesired js to the worker.js somehow?

Franklin05:12:37

I also suspect that dev tools are excluded from the build report somehow

Franklin05:12:09

so, I commented out :devtools from my app build

;;  :devtools
      ;;    {:preloads [day8.re-frame-10x.preload],
      ;;     :watch-dir "resources/public/css/"},

Franklin05:12:17

and that got rid of the error

Franklin05:12:50

I wonder if there's a way to disable dev tools for specific modules

thheller06:12:01

ah its is the preloads yeah. you can move the preloads to the :shared module. just :shared {:entries [] :depends-on #{:base} :preloads [day8.re-frame-10x.preload]}

👍 1
Adam Helins16:12:22

At work, we noticed that bumping Shadow to 2.16.6 resulted in a 50% increase in size of our advanced build. The increase seem to have been introduced with 2.15.13 which uses a newer version of the CLJS compiler with some various Closure updates. Did anyone notice an increase in size as well?

thheller16:12:38

@adam678 what does the build report look like before/after?

dergutemoritz16:12:03

I'm trying to build them but it fails like this:

Hook [0 shadow.cljs.build-report/hook] failed in stage :flush
NoSuchFileException: .shadow-cljs/builds/psap/release/out/main.js
Does that ring any bell?

dergutemoritz16:12:01

(also hi to my original hometown and thanks for making shadow-cljs 👋😄)

dergutemoritz16:12:57

ah lemme try the standalone build

thheller16:12:16

not a :browser build?

thheller16:12:49

build reports only work with :browser currently

dergutemoritz16:12:49

yup, it's :node-library

dergutemoritz16:12:55

ah, too bad 😞

dergutemoritz16:12:08

Indeed, the standalone run failed, as well

dergutemoritz16:12:35

I think I was able to convert it into a :browser 💪 Still need to confirm whether the regression is reproducible this way, too. Stay tuned.

thheller16:12:22

try setting :js-options {:js-provider :require}. it won't actually run in the browser that way but the build report won't suddenly contain all the bundled npm depenendeices

👍 1
dergutemoritz17:12:02

OK nice, the size increase still reproduces that way

dergutemoritz17:12:11

And we got some build reports now

dergutemoritz17:12:19

Looks like the main culprit is the application code itself

dergutemoritz17:12:57

Its build output pretty much doubled in size

dergutemoritz17:12:17

Maybe more inlining going on or something like that? :thinking_face: Any ideas how to dig deeper?

dergutemoritz17:12:58

Maybe of note that said code is all .cljc

dergutemoritz17:12:28

Ah but I can also see similar increases for the .cljs portions

thheller17:12:50

I don't have any guess for you since I don't have a clue what you are doing 😛

thheller17:12:13

there shouldn't be any big increases and I haven't had any other reports like that

thheller17:12:21

nor noticed any increases in my own builds

dergutemoritz17:12:32

Alright, then it's probably related to something unusual we're doing 😉 I wasn't asking about guesses moreso than hints how to investigate this further. The build report is a great help already, thanks a lot!

dergutemoritz17:12:16

I guess we'll revert the update for now and try to get to the bottom of this. If we do or are able to come up with an isolated repro, we'll get back to you!

thheller17:12:09

the only thing I can think of is running the build with --pseudo-names and looking at the output trying to figure out where it increase is coming from

thheller17:12:40

--pseudo-names will make it huge regardless but at least you can figure out what the code was before optimizations

dergutemoritz17:12:31

Great, thanks!