Clojurians Log v2

what are you trying to do exactly? the standard shadow.loader based stuff will not work no, but :esm in theory could

I want to delay some loading in my web worker. So I should build those modules with :npm-module target?

no, :target :esm is the only thing that'll work for that

How can I load this :esm stuff?

using the dynamic import referenced here https://clojureverse.org/t/generating-es-modules-browser-deno/6116

This looks good. Now I have several builds, and they all have the same "duplicated" modules to be lazy loaded. It seems with this, I can move these modules into a :esm build.

you could and should have done this with a :browser build as well?

but yes, each worker should be a module in a :esm build

I have some code want to be lazy loaded across multiple builds, including the :browser target builds and the worker builds.

I don't think I understand what you are trying to do or why it needs to be multiple builds?

one build with many modules is the recommended approach

as described in https://code.thheller.com/blog/shadow-cljs/2019/03/03/code-splitting-clojurescript.html

the fact that workers are involved doesn't change this in any way really

(the way you actually load stuff is different in esm though)

Can I have multiple entries from one build? I'm build a website mostly single paged, but still have some pages.

so you want a module per page?

Yes. It's like A depends C, D, E B depends C, D, E etc

So A, B are entries, and C, D, E are shared code. And C, D, E should be lazy loaded.

then you are using the description wrong 😛 if its lazy loaded it doesn't depend on it directy

Ops

given that A cannot work without C,D,E they must be loaded first

Hmm, I was wrong. And yeah, that's why I don't know how to do this in one build.

this is from the blog post, pretty much your exact use case

{...
 :builds
 {:app
  {:target :browser
   ...
   :module-loader true
   :modules
   {:main
    {:init-fn }

    :account-overview
    {:entries [demo.components.account-overview]
     :depends-on #{:main}}

    :product-detail
    {:entries [demo.components.product-detail]
     :depends-on #{:main}}

    :product-listing
    {:entries [demo.components.product-listing]
     :depends-on #{:main}}

    :sign-in
    {:entries [demo.components.sign-in]
     :depends-on #{:main}}

    :sign-up
    {:entries [demo.components.sign-up]
     :depends-on #{:main}}}
   }}}

For example, now my "A" build. It has 4 modules, D, C, E depends on A. And A is the entry.

forget about it being :target :browser for a sec, but the :modules part is identical in :esm

so using that example above, lets say :sign-up is an individual page

that page would just include <script type="module" src="/js/sign-up.js">

that would in turn also load the :main module, automatically

if you want to run something on page load each module could have its own :init-fn

Ah, yeah, this is exactly what I want to do. I need this for my current browser build!

{...
 :builds
 {:app
  {:target :esm
   ...
   :modules
   {:main
    {:entries []}

    ...
    :sign-up
    {:init-fn demo.app.sign-up/init
     :depends-on #{:main}}}
   }}}

works ok for :browser too but there each page would also need to have script tags for each shared module, :esm makes that simpler

I didn't get this, do you mean I can just use :esm target instead of :browser target?

pretty much yeah. well kinda depends on what you are doing, so I guess it depends

And how to introduce the worker here? Just add :web-worker true to make it an entry for worker?

no, its just a normal module. no extra config needed

I saw in the document, :web-worker is required?

no, that is a :browser only thing and does absolutely nothing in :esm. you can have it, it just doesn't do anything

you'll also need to construct the workers with type: "module"

What is type: "module" , the property in package.json?

{:builds
 {:app {
        :target :esm
        ;; I no longer need that :module-loader true, right?
        ;; Because now it's dealing with ESM modules
        :modules
        {:shared {:entries []}

         :page-a {:init-fn    page-a/init
                  :depends-on #{:shared}}

         :page-b {:init-fn    page-b/init
                  :depends-on #{:shared}}

         :worker {:init-fn    worker/init
                  :depends-on #{:shared}}

         :module-to-load-later-a {:entries [lazy-a] :depends-on #{:shared}}
         
         :module-to-load-later-b {:entries [lazy-b] :depends-on #{:shared}}}}}}

This is what I have in mind for now.

https://web.dev/articles/module-workers#enter_module_workers

New knowledge learned.

:module-to-load-later-a also needs :init-fn, :entries or :exports, but yes thats correct

With this config, I could load :module-to-load-after-a in :worker too?

yes

Sounds awesome! I'm going to try this out.

(shadow.esm/dynamic-import "./module-to-load-later-a.js")

if you are on the very latest shadow-cljs I added a new utility for lazy loading in esm

might be useful but I haven't documented it yet

I can use it for sure

basically the idea is that you can load stuff by name. (shadow.esm/load-by-name 'some.ns/some-var)

I'm also using shadow.css, btw

basically an abstraction so that you don't need to know module filenames and stuff

So it's more like the shadow.lazy way?

yes

basically the gist is that the thing you want to load must have a metadata tag

so (defn ^:lazy-loadable some-var [...] ...)

load-by-name gives you a promise that resolves to a function that you can call to get the actual thing

I can try it I think.

I'm trying to change my build to :esm, and I'm trying to use shadow.esm/load-by-name to load function with ^:lazy-loadable. But it says "could not find loadable info".

is it tagged correctly?

(defn ^:lazy-loadable foo [] ...)

and using 'the.fully.qualified/foo?

needs the full symbol name

I think I'm doing it correctly. I paste the following from my code, I hope I didn't make any stupid mistake.

(ns racepoker.shared.transport.facade
  ...)

(defn ^:lazy-loadable create-facade-transport
  [rpc]
  (FacadeTransport. rpc))

(esm/load-by-name 'racepoker.shared.transport.facade/create-facade-transport)

looks fine

In build, their modules' relation is

...
         :facade {:entries    [racepoker.shared.transport.facade racepoker.shared.wallet.facade]
                  :depends-on #{:shared}}
         :shared {:entries []}
         ...

This facade thing depends on shared. And the code to load is in shared module.

looks fine to me

I debugged the esm.cljs, it seems that loadable is empty.

I should expect all the names tagged are registered?

yes

So it shouldn't be a problem with .cljc

nope, doesn't matter

shadow.esm.loadables in the browser console should be an object

It's empty.

But how I confirm that the module to be loaded is compiled?

in :output-dir grep for shadow.esm.add_loadable

that should be somewhere?

normally in :output-dir/shared.js?

Yes, they are there

then unclear why shadow.esm.loadables would be empty? given that the only thing the function does is add stuff?

it currently does not have defonce? did you mess with that namespace in the REPL or something?

I assume this is the definition of this add_loadable

shadow.esm.add_loadable = (function shadow$esm$add_loadable(the_name,module,accessor){
return (shadow.esm.loadables[cljs.core.str.cljs$core$IFn$_invoke$arity$1(the_name)] = ({"module": module, "get": accessor}));

https://github.com/thheller/shadow-cljs/blob/e00a65324cde0c6d26f57f8429c96febc23aaca5/src/main/shadow/esm.cljs#L51

No, I didn't use repl yet. Just run npx shadow-cljs watch app

I think I can change this add_loadable function to see the parameters

the parameters are literally in the code you just posted?

everything looks correct on this end?

Hmm, no. I change it to

shadow.esm.add_loadable = (function shadow$esm$add_loadable(the_name,module,accessor){
    console.log(the_name, module, accessor);
return (shadow.esm.loadables[cljs.core.str.cljs$core$IFn$_invoke$arity$1(the_name)] = ({"module": module, "get": accessor}));
})

There are add_loadable(...) in code, but they are not called?

I think the shared.js in my case wasn't introduced in runtime.

I added some logs in that file, nothing logged in console neither.

Weird, but in network console, I can see the shared.js is loaded.

actually I have a guess

I'm guessing you are calling load-by-name too early?

Hmm, probably

Because I don't have to do def for them

:shared doesn't have any :init-fn. so I'm assuming you are just running some code immediately when some namespace is loaded

let me change it

Yes

You are right

let me change it

that then fails and throws, and thus never gets to the code that actually adds the loadable

should really use :init-fn

fwiw if a module is loaded immediately it makes no sense to have it as a separate module

would be faster to just not split it out at all

Well, in my old code, I do this and define them as vars. But they are promises, so wait them to be loaded when use it.

I have a question about esm build. When I use esm, the page loading(mainly downloading js modules locally) takes much longer time. Is it normal?

in development you mean? release shouldn't be any different

development loads each file individually, so it may be slower if you webserver is slow or if you are loading thousands of files

haven't seen much of a difference personally, but it can be a little slower. depending on the amount of files and how fast the webserver serves them I guess

I've changed our configuration to use :esm , it seems to work fine.

Now we could have a much smaller worker, it's really nice.

Hmm, in production build, there's an error

Uncaught SyntaxError: The requested module '' doesn't provide an export named: '$jscomp'

But the first line of my shared.js is something like

$var jscomp = ...

hmm might be polyfill related. not sure why it wouldn't be exposed though

try :prepend-js "globalThis.jscomp = jscomp" in the :shared module

Is it a compiler option?

Oh, sorry, in :shared module.

or better try :compiler-options {:output-feature-set :es-next}. probably no need for polyfills anyway

Hmm, I think neither way works. I've tried output-feature-set.

After adding :prepend-js "globalThis.jscomp = jscomp", I still don't see this line in shared.js.

[:app] Build CSS completed.
------ WARNING #1 -  -----------------------------------------------------------
 Resource: shadow/module/shared/prepend.js:1:20
 variable jscomp is undeclared
--------------------------------------------------------------------------------

It doesn't work, both jscomp and $jscomp are not defined.

If I add it via :append-js , the globalThis.jscomp = jscomp will become something like globalThis.G1 = jscomp.

yeah forget about that

Manually adding something like export { $jscomp } gets rid of the error.

but not sure why that would be needed

:append "export { $jscomp }" for the :shared module should automate that append, so you don't have to do it manually

Is there some rules for whether doing this?

Or where can I debug this behavior?

Add it via :append-js won't work, it says $jscomp is not defined during compilation.

I said :append not :append-js, wasn't a typo 😛

Thanks, I'll try it when with laptop. But should I debug where it generates this export statement?

I don't know why it would be need in the first place

kinda hard to guess at this. I haven't had this problem before, so no clue what is going on

kinda hard if there is no code I can look at

Is this $jscomp handled by cljs compiler? I recently updated to 1.12.42.

please provide me with more information about what exactly the problem is

like some lines of generated JS or something

it is kinda frustrating to guess in the blind

for example this is the generated JS code of my "base" module

export const $APP = {};
export const shadow$provide = {};
const shadow_esm_import = function(x) { return import(x) };
export const $jscomp = {};

I did not add anything manually and you can clearly see the $jscomp export

so, what do the first few lines of your shared.js look like?

I don't exactly have a lot of "usage data". so no clue if it just happens to work for me and you are doing something "wrong". or shadow-cljs is doing something wrong and I just happen to not run into it

These are some lines from the beginning. It's from release build, because it's only broken in release.

var $jscomp=$jscomp||{};$jscomp.scope={};$jscomp.arrayFromIterator=function(a){for(var c,b=[];!(c=a.next()).done;)b.push(c.value);return b};$jscomp.arrayIteratorImpl=function(a){var c=0;return function(){return c<a.length?{done:!1,value:a[c++]}:{done:!0}}};$jscomp.arrayIterator=function(a){return{next:$jscomp.arrayIteratorImpl(a)}};
$jscomp.makeIterator=function(a){var c=typeof Symbol!="undefined"&&Symbol.iterator&&a[Symbol.iterator];if(c)return c.call(a);if(typeof a.length=="number")return $jscomp.arrayIterator(a);throw Error(String(a)+" is not an iterable or ArrayLike");};$jscomp.arrayFromIterable=function(a){return a instanceof Array?a:$jscomp.arrayFromIterator($jscomp.makeIterator(a))};$jscomp.ASSUME_ES5=!1;$jscomp.ASSUME_ES6=!1;$jscomp.ASSUME_ES2020=!1;$jscomp.ASSUME_NO_NATIVE_MAP=!1;$jscomp.ASSUME_NO_NATIVE_SET=!1;
$jscomp.ISOLATE_POLYFILLS=!1;$jscomp.FORCE_POLYFILL_PROMISE=!1;$jscomp.FORCE_POLYFILL_PROMISE_WHEN_NO_UNHANDLED_REJECTION=!1;$jscomp.INSTRUMENT_ASYNC_CONTEXT=!0;$jscomp.IS_SYMBOL_NATIVE=typeof Symbol==="function"&&typeof Symbol("x")==="symbol";$jscomp.TRUST_ES6_POLYFILLS=!$jscomp.ISOLATE_POLYFILLS||$jscomp.IS_SYMBOL_NATIVE;

{:target           :esm
 :output-dir       "build/js"
 :release          {:output-dir "dist/race/js"}
 :modules
 {:solana    {:entries    [racepoker.shared.transport.solana racepoker.shared.wallet.solana]
              :depends-on #{:shared}}
  :sui       {:entries    [racepoker.shared.transport.sui]
              :depends-on #{:shared}}
  :facade    {:entries    [racepoker.shared.transport.facade racepoker.shared.wallet.facade]
              :depends-on #{:shared}}
  :shared    {:entries []
              :append  "export {$jscomp};"}
  ;; Web Worker
  :worker    {:init-fn    racepoker.worker/-main
              :depends-on #{:shared}}
  ;; Lobby page
  :lobby     {:init-fn    racepoker.lobby/-main
              :depends-on #{:shared}}
  ;; Table page
  :table     {:init-fn    racepoker.table/-main
              :depends-on #{:shared}}
  ;; Dashboard page
  :dashboard {:init-fn    racepoker.dashboard/-main
              :depends-on #{:shared}}
  ;; PWA page
  :pwa       {:init-fn    racepoker.pwa/-main
              :depends-on #{:shared}}}
 :css-modules
 {:lobby {:entries [racepoker.lobby.core]}
  :table {:entries [racepoker.table.core]}}
 :css-configs      ["styles/lobby.edn" "styles/table.edn"]
 :build-hooks      [(build/css-hook)]
 :compiler-options {:output-feature-set :es-next}}

Just like me know what information would be helpful. I'll also try to create a repro for the issue.

wouldn't really recommend mixing in the shadow.css stuff, but besides that looks ok

I'm trying to create a repository to reproduce this

that would help

I didn't success, but I found something.

I made a new project, and it works. The output of shared.js is

export const $APP = {};
export const shadow$provide = {};
const shadow_esm_import = function(x) { return import(x) };
export const $jscomp = {};
var ....

are you sure you are looking at the right output files? maybe changed dirs and looking at old files?

there must be least export const $APP = {};

No, I just deleted the dist and recompiled. There's no those lines. This is the very beginning of that file.

Yeah, I'm 100% sure.

Other files contain this line import { $APP, shadow$provide, $jscomp } from "./shared.js";.

no export const $APP = {}; anywhere at all in any file?

It has

export const $APP = {};
export const shadow$provide = {};

Not at the very beginning, at line 17

Does the access to my repository or the compiled result helps? I can share

I just don't know which information is helpful.

I think I found it

I can patch shadow-cljs locally to test?

it actually is fairly obvious bug when you look at it 😛

https://github.com/thheller/shadow-cljs/blob/e00a65324cde0c6d26f57f8429c96febc23aaca5/src/main/shadow/build/targets/esm.clj#L468

the logic is all wrong

if there is polyfill-js it injects it, but not the $jscomp thing. which is only required if there are polyfills

so, kinda dumb 😛

So it should be

(when (seq polyfill-js) (str polyfill-js "\n" "export const $jscomp = {};\n")))?

nah

easier to just always have it

otherwise need to adjust the import as well

What is this polyfill-js?

I guess the reason I have it non-empty is I have @solana/webcrypto-ed25519-polyfill dependency.

polyfills the closure compiler generates, so not coming from any npm packages directly.

I see. so it's not that polyfill. 😂

Will you fix it now?

try 3.1.7

It works!

Thank you so much!

hmmm, I met this

Uncaught SyntaxError: redeclaration of var $jscompshared.js:17:14note: Previously declared at line 1, column 5

the polyfill fun never ends 😛

maybe figure out why there are polyfills in the first place. there shouldn't be any if you set :output-feature-set :es-next

or maybe try :output-feature-set :no-transpile

So in theory, if I use es-next, there shouldn't be polyfill.

can't say for sure. not quite sure when the closure compiler decides that it needs polyfills and for what

Interesting, when I have the code running locally, it works. But after I upload it to cloudflare, it gives me this error.

but :es-next is the highest level it supports. so maybe the JS libs use some super last minute JS additions that the compiler wants to polyfill regardless

It's the same build.

oh, my mistake

I forgot to delete that :append.

Nope, the error occurs because the first line is

var $jscomp=$jscomp||{};

I guess you could just override that before sending to cloudflare 😛

I could patch shadow.esm too...

if you replace that with something of equal length the source maps should still work

this isn't from shadow.esm

that from the closure library

No, I mean, if it's export {$jscomp}, then there will be no issue.

hmm I guess

The equavalent of export const $jscomp = ... is globalThis.$jscomp = ...?

don't understand that question, but no it is not

Because I saw rest generation are still use var, I assume they are es3.

I don't know what that has to do with anything

closure library and cljs use var, for the most part

var $jscomp=$jscomp||{}; this construct is coming from the closure library, goog/base.js to be exact

has never been an issue before with :esm, so nothing tries to change it

I see. So the problem is closure library generates this, but it's not exported. And in some cases, closure library doesn't generate this when it thinks it's not necessary. Do I get it right?

no

the closure compiler is not aware that I'm turning its output into ESM. in fact I have to actively hide that.

so it just generates the same usual code it would generate for :browser

I then patch it all over the place to inject the import/export stuff

literally just do a search and replace with the problematic bit before sending it to cloudflare when it works in the browser

or make an issue. I'm done for the day, so not trying to figure that out now

Sure, no problem!

I checked this issue again, and I think I found something. But I want to confirm my thinking. I played with esm.clj , I found the polyfill-js is actually applied twice that it causes the polyfill section appears in output twice. It's like

var $jscomp=$jscomp|{}             |<- polyfill code injection, from somewhere idk
....
export const $APP = {};            |<- 3 lines shim
export const shadow$provide = {};
export const $jscomp = {};         |<- polyfill code again, from esm.clj#L549
var $jscomp=$jscomp|{} (
...
shadow$provide[0]=...              

This duplicated injections won't hurt, but the export const $jscomp = {}; after existing declaration (the first injection) seems to be the problem. I have to change it to export { $jscomp }; to make it work.

Oh you said this var $jscomp = $jscomp|{}; is from closure library.