This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-10-04
Channels
- # announcements (6)
- # babashka (7)
- # beginners (2)
- # biff (5)
- # calva (2)
- # cherry (17)
- # cider (3)
- # clj-kondo (8)
- # clojure (202)
- # clojure-brasil (8)
- # clojure-europe (20)
- # clojure-norway (23)
- # clojure-uk (4)
- # clojuredesign-podcast (5)
- # conjure (1)
- # cursive (9)
- # eastwood (22)
- # events (8)
- # fulcro (3)
- # hyperfiddle (22)
- # introduce-yourself (7)
- # lsp (67)
- # malli (1)
- # matrix (1)
- # meander (6)
- # off-topic (76)
- # pedestal (8)
- # polylith (17)
- # quil (12)
- # re-frame (2)
- # reagent (8)
- # releases (3)
- # shadow-cljs (67)
- # sql (93)
- # squint (39)
- # tools-deps (46)
- # vim (7)
Hi there, I'm trying to upgrade from shadow-cljs from 2.25.2 -> 2.25.8
, but CI is reporting this error when compiling an advanced build
> Execution error (ExceptionInfo) at shadow.build.compiler/par-compile-cljs-sources$fn (compiler.clj:1181).
> compilation failed
I looked at our logs, and it seems the build has failed when trying to upgrade to any of the patch versions after 2.25.2
. Wondering if you have a quick idea or if I should do some more investigation on my end
CI usually the first suspect is that the process is getting killed by the OS, usually OOM killer things
Oh interesting. One of the jobs only showed the logs I pasted above, but I found another one with more details:
[:scripts] Compiling ...
Multiple files failed to compile.
aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/video.cljs"] still waiting for #{icebreaker.components.video}
{:aborted [:shadow.build.classpath/resource "icebreaker/components/video.cljs"], :pending #{icebreaker.components.video}}
ExceptionInfo: aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/video.cljs"] still waiting for #{icebreaker.components.video}
shadow.build.compiler/par-compile-one (compiler.clj:1085)
shadow.build.compiler/par-compile-one (compiler.clj:1050)
shadow.build.compiler/par-compile-cljs-sources/fn--15110/iter--15132--15136/fn--15137/fn--15138/fn--15139 (compiler.clj:1168)
clojure.core/apply (core.clj:667)
clojure.core/with-bindings* (core.clj:1990)
clojure.core/with-bindings* (core.clj:1990)
clojure.core/apply (core.clj:671)
clojure.core/bound-fn*/fn--5818 (core.clj:2020)
java.util.concurrent.FutureTask.run (FutureTask.java:264)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
java.lang.Thread.run (Thread.java:829)
aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/event_settings.cljc"] still waiting for #{icebreaker.components.video}
{:aborted [:shadow.build.classpath/resource "icebreaker/components/event_settings.cljc"], :pending #{icebreaker.components.video}}
ExceptionInfo: aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/event_settings.cljc"] still waiting for #{icebreaker.components.video}
shadow.build.compiler/par-compile-one (compiler.clj:1085)
shadow.build.compiler/par-compile-one (compiler.clj:1050)
shadow.build.compiler/par-compile-cljs-sources/fn--15110/iter--15132--15136/fn--15137/fn--15138/fn--15139 (compiler.clj:1168)
clojure.core/apply (core.clj:667)
clojure.core/with-bindings* (core.clj:1990)
clojure.core/with-bindings* (core.clj:1990)
clojure.core/apply (core.clj:671)
clojure.core/bound-fn*/fn--5818 (core.clj:2020)
java.util.concurrent.FutureTask.run (FutureTask.java:264)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
java.lang.Thread.run (Thread.java:829)
Looked through channel history and found a related thread w/ similar error: https://clojurians.slack.com/archives/C6N245JGG/p1642032444058400
Compilation is happening fine on 2.25.2
, but starting with 2.25.3
, we started seeing the above issue. Was there a change in 2.25.3
that might be contributing to it?
as I said .. its usually an OS resource issue. so make sure the CI isn't hitting any limits, when in doubt constrain the memory in some way
Related to my post from yesterday but restating my main question as I continue to be stuck on this: how would one go about dealing with the goog-library already existing in the runtime environment during dev? I've tried a few things from build hooks but without succes. I'll post some of the results in the comments. I've also looked into running the parent environment (atlassian sdk/jira server) in non-dev mode so their goog-library is compiled away but so far I haven't been able to get a working setup with this approach.
as I have barely any information about the particular setup I cannot provide much guidance
two closure libraries in dev mode next to each other won't play nice with each other
I also still don't quite understand how window.goog = {};
fixed anything? didn't that completely destroy the other library? or does that not actually use any of the closure library?
Ok, I understand you dont have much to work with here and am not expecting you to debug my issues, just hoped this occurs more often and there's some generic solution. Since you're asking: atlassian sdk is public in the sense that you can download it and use it to develop a plugin for their products. But the code or that of the platforms isnt opensource. https://developer.atlassian.com/server/framework/atlassian-sdk/set-up-the-atlassian-plugin-sdk-and-build-a-project/.
if the other is just closure library code it'll integrate seamlessly into the shadow-cljs build
I don't know anything about the atlassian sdk and all I see is some java stuff. the sdk does contain the closure compiler, but I cannot find any actual JS sources?
Like I said yesterday: to me it kind of makes sense that overwriting window.goog
works (although it seems silly), as in: they initialise the goog library in their platform use it whereever. At some point they load our app.js where we reset it, reinitialise it and use it for our stuff. But again, I'm a bit out off depth here so maybe what Im saying doesnt make sense
Yeah the server is in java. We use cljs/shadow to generate our js-files and output it into a folder thats configured to be loaded by their plugin system.
otherwise you are on your own, the closure library is not designed to be loaded twice during development. I'm not quite sure why you'd be getting their development code anyways?
the code would be even better. I mean you must include a script tag of their sources somewhere?
I'm not completely sure why we're dealing with their dev-build either but I think they are generating files on the fly during dev them selves (again I might be wrong about how this all works)
or rather they include something and then include yours? To me this seems to be whats happening
I'm affraid Im not gonna be able to provide what you're looking for, that part is simply out of my reach.
What I get to work with is a configuration file which details how/where I want to integrate with their product and in what location they can find the resource to load (in this case a js file generated with cljs/shadow) How they merge all of this into the end-result is inaccessible to me. I'm also confused why Im dealing with their dev builds/files during my dev time, but that seems to be whats happening and I've so far been enable to change that.
So it seems to me there's no "clean" solution here. That being said, I'm wondering why the previously implemented hack of inserting window.goog = {};
doesnt make sense to you? To me it seems that all code (i.e. their platform and our plugin) would have a goog library available to it, we just destroy and reinitialise it within our part. It really does seem to have worked (as in, this can be proven by adding/removing the build hook), but doenst anymore. Also if I add the line to the top of my new js output file "by hand" it seems to work as well, that is: all goog related errors disapear, I still have 1 remaining error but it seems unrelated.
I assumed working with the :unoptimizable
key must have been removed in some newer version of shadow or doesn't have the same effect anymore. As mentioned before I tried adding the same line with :prepend
but it shows up too late in the js output. I also tried to set goog-base
of the module to false
or removing all sources
that start with "goog" but both give me erros (albeit different ones). The result I find under :output
in the build state doesn't seem usefull to me at first sight either.
Do you have any pointers based on the provided info? If not I'll leave it at this and see if I can find some hacky way to prepend the goog-reset to the beginning of the file outside shadow and/or try again if I can tweak the sdk setup.
I'm also reaching out on their developers platform btw, as this is in essence not cljs specific at all. Haven't found any existing threads on there though so I guess not many people run into this..
there is nothing in shadow-cljs to do what you want to do, but you can sort of "hack" it yourself
:unoptimizable
was never meant to be user-adjustable and was purely an internal thing
my expectation would be that window.goog = {};
would break the "other" closure library use, but I guess if you never had issues thats fine
but I don't know anything about how your stuff is getting loaded or integrated I cannot say how
:prepend
is what :unoptimizable
was Ok, thanks for the clarification. From what I can see in the js ouput it doesn't behave exactly the same though. I.e. in the old build the goog-reset was put all the way at the top of the output, in the new build it's put after some goog related initialisation. It doesn't really matter though, I understand what your saying and it's in line with what I understood about this being the result of reading the source instead of using some public api. If I continue with this "solution" I'll find a way to automatically inject it at the top of the file.
I’m afraid you're right though as the one remaining "unrelated" error after injecting the goog reset by hand seems to be related after all... It was Uncaught TypeError: Cannot read properties of undefined (reading 'Dir')
If I investigate the error in the browser console I get taken to a line with goog.i18n.bidi.Dir
. This isnt in our code or anything we have a direct dependency on so Im doubting if my mental model stated above is correct and overwriting goog like proposed does actually have effects outside our plugin (as you seem to be expecting). But than it obviously confuses me even more that it works in the existing version of the plugin in the exact same jira-server version and suddenly breaks when I upgrade shadow (and cljs).
Anyway, I think it's enough for today, I'll see if I can get any wiser tomorrow. I understand there's not much you can do and very much appreciate the help you already offert. I might keep you updated in case I find a solution, but Im not expecting anything unless I find some of the info you requested.
I assume you just "delete" it by window.goog = {};
and when the other code looks for it its gone
all assuming of course that the likely different closure library versions are compatible
Ok, I looked into it a bit more and I'm now also surprised the previous hack worked at all. Apparently the goog versions at the time were compatible enough, but thats not the case anymore. The atlassian SDK is still on closure v20191111. I tried a few ways of resetting and swapping the googs but something is always missing something, either in their code or ours.
I ended up overwriting your goog.provide
via :prepend
to:
goog.provide = function (name) {
if (goog.isProvided_(name)) {
console.log('Namespace "' + name + '" already declared.');
} else {
goog.constructNamespace_(name);
}
};
If I understand correctly what I'm doing (basically just meshing the two versions into one goog and praying all code gets what it needs) its pretty wacky but seems to work at first sight. The build runs, I clicked around a bit and made some changes to see if hot-reload behaves as expected.
I would be very surprised if this doesn't blowup at some point, but I think it's my best option for now. Else I'm simply stuck with the old shadow (closure) version, but when I tried working with that I ran into some npm related issues which I don't get in projects with newer shadow versions (Hello. I'm trying to figure out how to embed REPL server in a shadow-cljs node-script release build. It is to have a way of debugging some issues in the production application. Was reading through the shadow-cljs code and found this part https://github.com/thheller/shadow-cljs/blob/b6d0afbddc8952b67879bc9cc605526dc1cc0769/src/main/shadow/build/targets/node_script.clj#L43 But am unsure how to go from there Are there recommended ways of doing this in a way that only the release build JS file can be run via Node.js and the REPL server would start as part of the application process? Any recommendations and references welcome! Thank you!
Would there be a way to include a library or namespace (implemented as part of shadow-cljs or other project), that would offer REPL from a release build process? With my not too deep knowledge of the processes involved, I'd think it'd require the Node.js process to be able to compile CLJS forms, evaluate the compiled JS in the Node.js process and also bind to a port to allow incoming REPL client connections Do you see it feasible? If so, I'd very much appreciate any hints or pointers how to begin moving towards it. Even if only as a hacking and learning experiment. Thank you!
well you could setup a self-hosted build and build the server yourself https://code.thheller.com/blog/shadow-cljs/2017/10/14/bootstrap-support.html
you'll also be limited to :simple
optimizations but I guess for node thats not much of an issue
Btw, do you accept tips in Bitcoin (via Lightning)? I'd like to donate to you as I've benefitted much from your work. https://tippin.me/ or https://guides.getalby.com/alby-guides/alby-lightning-account/where-to-use-your-lightning-address/how-to-use-your-own-domain-as-your-lightning-address are the simplest ways to get started
https://getalby.com/p/thheller no clue how this works, but maybe thats an excuse to finally look into it more 😛
it showed up on the alby website yes. what I do with it from there I don't know yet 😛
Awesome! As of what to do with it, if you were in El Salvador (where I am), you could use it to buy anything from the shops (Bitcoin is official currency here). As you seem to have open and curious mind, to learn more, you could check out 1. High level overview + economic aspects: The Bitcoin Standard (download PDF at https://drive.google.com/file/d/1tV_nb2ixZyl5p9EY_imjf4gKNviHvHE8/view?usp=sharing) 2. Technical side Bitcoin: https://github.com/bitcoinbook/bitcoinbook 3. Technical side Lightning: https://github.com/lnbook/lnbook
I use Bitcoin because it protects me from the most heinous crime - theft of time - committed via the fiat currency (via taxes and inflation). Bitcoin is scarce (limited amount) and unconfiscatable, so I can convert my time (spent on work) into a money that can't be diluted (time theft)
Thanks again for shadow-cljs and all the rest of your work and contributions to the Clojure community. Have a great day. Commence code mode 🤓