This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-06-08
Channels
- # announcements (2)
- # asami (2)
- # babashka (7)
- # beginners (59)
- # cider (12)
- # cljdoc (1)
- # cljs-dev (18)
- # clojure (23)
- # clojure-europe (15)
- # clojure-losangeles (1)
- # clojure-nl (2)
- # clojure-uk (5)
- # clojured (16)
- # clojurescript (22)
- # core-typed (8)
- # cursive (3)
- # datomic (24)
- # events (2)
- # fulcro (4)
- # gratitude (1)
- # helix (13)
- # hoplon (18)
- # integrant (2)
- # introduce-yourself (1)
- # jobs-discuss (1)
- # joyride (5)
- # minecraft (1)
- # off-topic (76)
- # pathom (18)
- # podcasts-discuss (8)
- # polylith (11)
- # remote-jobs (4)
- # rewrite-clj (22)
- # sci (4)
- # shadow-cljs (152)
- # sql (4)
- # tools-build (26)
- # tools-deps (34)
I've noticed that using modules seems to increase build times. Is there a clean way to disable modules in certain circumstances (preferably via the command line)?
that seems unlikely? I mean if you can prove it I'd be interested but I very much doubt :modules
have any impact at all
unless you are talking release
builds? can't say how much more that side costs on the closure compiler side. wouldn't expect too much here either though
just try the same build with one :modules
but all the :entries
it would otherwise have. and then one with regular multiple :modules
?
@thheller so I just performed this experiment, and what takes 3.5s with a single module (re-building a single ns) takes 25s with multiple modules
@wombawomba please verify with --verbose
where the actual time is spent?
and please report the final Build Complete
message? I mean if it ends up including thousands more files this is expected?
FWIW the build messages without --verbose
were:
[:app] Build completed. (1098 files, 1 compiled, 0 warnings, 2.80s)
and
[:app] Build completed. (1410 files, 1 compiled, 0 warnings, 22.36s)
so it seems I missed a few files when joining everything together
still, the time difference is disproportionate
okay, so here's the verbose report with a single module:
[:app] Compiling ...
-> Resolving Module: :main
<- Resolving Module: :main (1386 ms)
-> build target: :browser stage: :compile-prepare
<- build target: :browser stage: :compile-prepare (0 ms)
-> Compile CLJS: my-app/views/site_home.cljs
<- Compile CLJS: my-app/views/site_home.cljs (743 ms)
-> Cache write: my-app/views/site_home.cljs
<- Cache write: my-app/views/site_home.cljs (92 ms)
-> build target: :browser stage: :compile-finish
-> build target: :browser stage: :flush
-> Flush: my-app/views/site_home.cljs
-> Flushing unoptimized modules
-> Flush: shadow/module/main/append.js
<- Flush: shadow/module/main/append.js (0 ms)
<- Flush: my-app/views/site_home.cljs (12 ms)
<- Flushing unoptimized modules (330 ms)
<- build target: :browser stage: :flush (350 ms)
<- build hook: [0 shadow.cljs.build-report/hook] stage: :flush (0 ms)
[:app] Build completed. (1098 files, 1 compiled, 0 warnings, 2.75s)
...and here's the one with multiple modules:
(FWIW disabling the build-report hook doesn't seem to make a difference)
...interesting how "Resolving Module" is either instantaneous or takes hundreds of ms. A bug perhaps?
I'm guessing that the fast ones require fewer namespaces and it doesn't go chasing down 1000+ files
150 modules, of which 100 just require a single js file from an NPM package
there seems to be no correlation between module size and the time a module takes to build
ah no, wait, there is a correlation - my bad
We've actually talked about this before 🙂 I'm packaging react-syntax-highlighter syntax files as separate modules
regardless, those modules seem to be pretty quick to load
there's also one module per view in the app, and it seems to be those that are taking a long time to resolve
shadow-cljs 2.18.0
what does the build report look like for this? 149 tiny modules and 1 huge chunky main module?
no, it's fairly well distributed
in the meantime, is there something I can do to disable the modules from the command line when developing?
alright
@thheller any idea when/if this code might be optimized? Ideally I'd use modules for development as well, and I'm curious as to when I might be able to do so
makes sense
if possible, something like "only resolve modules when involved files change" seems like it would take care of the problem
thats not the issue. at least I don't think it is ... just guessing still at this point
oh? looking at the build output, all the extra build time seems to come from "Resolving Module" steps
the actual compilation doesn't seem to be affected
the issue (I'm guessing) is that that repeatedly for each module since it is touching the disk
yeah, that's what I figure too
and it is not using that info because node resolve rules are weird and requiring one thing from one place
I have no idea how this works under the hood, but it seems like you should be able to keep a record of all files involved when resolving each given module, and re-resolve only when those files are touched
yes if you scratch the "touched" part again. caching is not the issue here and is covered elsewhere.
awesome, thanks
@wombawomba I cannot reproduce things resolving slowly multiple times. so might be something you are doing in your setup I'm missing
of course I'm testing with a reasonable number of modules so can't really say if that is a factor
An interesting issue. Not sure whether a genuine issue or a peculiarity.
I'm vendoring in a library called waveform-playlist
and in one of its files it has this:
import stateClasses from "./track/states";
Shadow-cljs complains:
FileNotFoundException: /home/p-himik/dev/git/ensemble/src/waveform-playlist/track/states (Is a directory)
This is, there's both a track/states
directory and a track/states.js
file.
When using IDEA to navigate to the source of the "./track/states"
import, it correctly goes to states.js
.by "vendoring" I assume you mean putting it on the classpath? In general you should refer to exact files with extensions there.
this doesn't follow the node resolve rules and to avoid ambiguities like that you should be using exact file names
shadow-cljs watch
seems to affect other commands run in separate terminals.
I run my unit tests by first running shadow-cljs -A:test compile unit-tests
, which has a target of :node-test and produces a unit-tests.js file that can then be executed via node. This all works fine unless I happen to be running shadow-cljs watch app
in another terminal at the same time. In this case, the compile step still produces a unit-tests.js file, but it is missing a bunch of the imports and executing it runs 0 tests.
Any ideas on this?
@t.denley yes, the shadow-cljs
command will re-use a running shadow-cljs instance if running. I recommend keeping you classpath the same all the time to avoid issues such as this. I assume you are adding extra-paths in the test aliases. there is no downside in always having them. you can run with --force-spawn
to always get a new instance
How should I actually vendor a JS library that consists of multiple files?
The section at https://shadow-cljs.github.io/docs/UsersGuide.html#_language_support makes it seem pretty straightforward.
So I have /waveform-playlist/track/loader/LoaderFactory.js
on classpath with (abridged)
import BlobLoader from "./BlobLoader";
import IdentityLoader from "./IdentityLoader.js";
import XHRLoader from "./XHRLoader";
And I have /waveform-playlist/track/loader/IdentityLoader.js
on classpath with
import Loader from "./Loader";
export default class IdentityLoader extends Loader {
load() {
return Promise.resolve(this.src);
}
}
But in the browser, I get:
ReferenceError: IdentityLoader$$module$waveform_playlist$track$loader$IdentityLoader is not defined
at eval (LoaderFactory.js:5:19)
at eval (<anonymous>)
at goog.globalEval (main.js:472:11)
at env.evalLoad (main.js:1534:12)
at main.js:2705:12
And the loaded LoaderFactory.js
now looks like this:
import BlobLoader from "/waveform-playlist/track/loader/BlobLoader.js";
import IdentityLoader from "/waveform-playlist/track/loader/IdentityLoader.js";
import XHRLoader from "/waveform-playlist/track/loader/XHRLoader.js";
What should I do here?@p-himik js files on the classpath get different treatment than the regular node_modules files. this code gets rewritten by the closure compiler. there may be issues with that since not many people do this
I can't say what exactly is the problem here but I recommend looking at the generated code
I wish this worked more reliably but what the closure compiler is doing here is somewhat arbitrary and changes constantly
its also not really meant to be used the way I'm using it. it is usually only use in the context of a whole program optimization pass where everything is in the same scope
Oh. So it sounds like something might work in prod but be broken in dev? And maybe vice versa?
The compiled output has no import statements of any kind BTW. It's as if they don't exist at all.
yes, they need to be removed. otherwise they'd break completely since you can't use import outside modules
Ah, right.
But all usages of IdentityLoader
become IdentityLoader$$module$waveform_playlist$track$loader$IdentityLoader
. No clue how that's supposed to work.
meaning it rewrites everything with the assumption everything is in the global scope
$waveform_playlist$track$loader$IdentityLoader
basically /waveform_playlist/track/loader/IdentityLoader
I am looking at the generated code. :) I'm not sure what exactly I should be looking for though.
So I have target/dev/cljs-runtime/module$waveform_playlist$track$loader$LoaderFactory.js
and in it I have console.log("IL", IdentityLoader$$module$waveform_playlist$track$loader$IdentityLoader);
as the very first line.
Ah, right. In target/dev/cljs-runtime/module$waveform_playlist$track$loader$IdentityLoader.js
I have:
class IdentityLoader$$module$waveform_playlist$track$loader$IdentityLoader extends $jscompDefaultExport$$module$waveform_playlist$track$loader$Loader {
load() {
return Promise.resolve(this.src);
}
}
/** @const */
var module$waveform_playlist$track$loader$IdentityLoader = {};
/** @const */
module$waveform_playlist$track$loader$IdentityLoader.default = IdentityLoader$$module$waveform_playlist$track$loader$IdentityLoader;
$CLJS.module$waveform_playlist$track$loader$IdentityLoader=module$waveform_playlist$track$loader$IdentityLoader;
//# sourceMappingURL=module$waveform_playlist$track$loader$IdentityLoader.js.map
But also, seems like it could've been using $CLJS.module$waveform_playlist$track$loader$IdentityLoader.default
where that class is supposed to be used.
Oh, about a year ago or so {:loader-mode :script}
made a dev page reload insanely slow. I'd rather search for a different solution.
It's not possible to tell GCC to use that $CLJS.[...]
instead of the name of a class, right?
Out of hopeful curiosity - why exactly is eval
needed?
Why can't the initial code loading just load a single giant file without any eval
statements and any further code reloads append new <script>
tags?
My level of expertise is dangerously close to 0 here, but I think at least on the initial load you can provide a single mangled JS file plus a single sourcemap file, and let it be mapped to multiple original CLJS sources. Right? If so - maybe there's a way to patch a source map without having to re-download it? Of course, even if all that is correct, it still doesn't address that other reason...
thats what this is basically. it just uses eval to load the files to match the behavior of hot-reloaded files
but that wasn't the reason. really can't remember but I spent a substantial amount of time on this so please assume that this is done for a reason
Thanks, will check out how it's done. Have never bothered with all JS :poop: ways of module management.
Oh bloody hell...
So if I have exports = {Playlist};
it doesn't work because Suspicious re-assignment of "exports" variable. Did you actually intend to export something?
And if I have exports.Playlist = Playlist;
or module.exports = {Playlist};
, it doesn't work because
ExceptionInfo: failed to convert sources
[...]
Caused by:
NullPointerException: NAME h 11:8 [length: 1] [source_file: waveform-playlist/Playlist.js] [original_name: h]
And h
there is just import {h, diff, patch} from "virtual-dom";
.Ah, of course. Brain refused to see it because it's not importing the vendored stuff.
https://github.com/thheller/shadow-cljs/commit/1b459145974b1f9566a14c3b87e1f70b3b288738
Immaculate timing - just as I finished rewriting it into CommonJS! :D Thanks for all the guidance BTW. But good to know either way, I'll test it a bit later and let you know.