Fork me on GitHub
#shadow-cljs
<
2023-10-04
>
alex11:10:53

Hi there, I'm trying to upgrade from shadow-cljs from 2.25.2 -> 2.25.8, but CI is reporting this error when compiling an advanced build > Execution error (ExceptionInfo) at shadow.build.compiler/par-compile-cljs-sources$fn (compiler.clj:1181). > compilation failed I looked at our logs, and it seems the build has failed when trying to upgrade to any of the patch versions after 2.25.2. Wondering if you have a quick idea or if I should do some more investigation on my end

thheller11:10:34

what is the rest of the error?

thheller11:10:07

CI usually the first suspect is that the process is getting killed by the OS, usually OOM killer things

alex11:10:06

Oh interesting. One of the jobs only showed the logs I pasted above, but I found another one with more details:

[:scripts] Compiling ...
Multiple files failed to compile.
aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/video.cljs"] still waiting for #{icebreaker.components.video}
{:aborted [:shadow.build.classpath/resource "icebreaker/components/video.cljs"], :pending #{icebreaker.components.video}}
ExceptionInfo: aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/video.cljs"] still waiting for #{icebreaker.components.video}
	shadow.build.compiler/par-compile-one (compiler.clj:1085)
	shadow.build.compiler/par-compile-one (compiler.clj:1050)
	shadow.build.compiler/par-compile-cljs-sources/fn--15110/iter--15132--15136/fn--15137/fn--15138/fn--15139 (compiler.clj:1168)
	clojure.core/apply (core.clj:667)
	clojure.core/with-bindings* (core.clj:1990)
	clojure.core/with-bindings* (core.clj:1990)
	clojure.core/apply (core.clj:671)
	clojure.core/bound-fn*/fn--5818 (core.clj:2020)
	java.util.concurrent.FutureTask.run (FutureTask.java:264)
	java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
	java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
	java.lang.Thread.run (Thread.java:829)
aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/event_settings.cljc"] still waiting for #{icebreaker.components.video}
{:aborted [:shadow.build.classpath/resource "icebreaker/components/event_settings.cljc"], :pending #{icebreaker.components.video}}
ExceptionInfo: aborted par-compile, [:shadow.build.classpath/resource "icebreaker/components/event_settings.cljc"] still waiting for #{icebreaker.components.video}
	shadow.build.compiler/par-compile-one (compiler.clj:1085)
	shadow.build.compiler/par-compile-one (compiler.clj:1050)
	shadow.build.compiler/par-compile-cljs-sources/fn--15110/iter--15132--15136/fn--15137/fn--15138/fn--15139 (compiler.clj:1168)
	clojure.core/apply (core.clj:667)
	clojure.core/with-bindings* (core.clj:1990)
	clojure.core/with-bindings* (core.clj:1990)
	clojure.core/apply (core.clj:671)
	clojure.core/bound-fn*/fn--5818 (core.clj:2020)
	java.util.concurrent.FutureTask.run (FutureTask.java:264)
	java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
	java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
	java.lang.Thread.run (Thread.java:829)

alex11:10:43

Looked through channel history and found a related thread w/ similar error: https://clojurians.slack.com/archives/C6N245JGG/p1642032444058400 Compilation is happening fine on 2.25.2, but starting with 2.25.3, we started seeing the above issue. Was there a change in 2.25.3 that might be contributing to it?

thheller11:10:41

nothing was added that would affect this in any way?

thheller11:10:57

as I said .. its usually an OS resource issue. so make sure the CI isn't hitting any limits, when in doubt constrain the memory in some way

Yab Mas11:10:12

Related to my post from yesterday but restating my main question as I continue to be stuck on this: how would one go about dealing with the goog-library already existing in the runtime environment during dev? I've tried a few things from build hooks but without succes. I'll post some of the results in the comments. I've also looked into running the parent environment (atlassian sdk/jira server) in non-dev mode so their goog-library is compiled away but so far I haven't been able to get a working setup with this approach.

thheller12:10:27

as I have barely any information about the particular setup I cannot provide much guidance

thheller12:10:45

two closure libraries in dev mode next to each other won't play nice with each other

thheller12:10:05

but that was always the case, and was not changed in the versions you mentioned

thheller12:10:47

you mention atlassian sdk/jira server. is that some public thing I can look at?

thheller12:10:49

I also still don't quite understand how window.goog = {}; fixed anything? didn't that completely destroy the other library? or does that not actually use any of the closure library?

Yab Mas12:10:22

Ok, I understand you dont have much to work with here and am not expecting you to debug my issues, just hoped this occurs more often and there's some generic solution. Since you're asking: atlassian sdk is public in the sense that you can download it and use it to develop a plugin for their products. But the code or that of the platforms isnt opensource. https://developer.atlassian.com/server/framework/atlassian-sdk/set-up-the-atlassian-plugin-sdk-and-build-a-project/.

thheller12:10:33

well the solution would be to combine the two builds into one

thheller12:10:49

if the other is just closure library code it'll integrate seamlessly into the shadow-cljs build

thheller12:10:17

if its some custom setup then it'll be more complicated

thheller12:10:04

I don't know anything about the atlassian sdk and all I see is some java stuff. the sdk does contain the closure compiler, but I cannot find any actual JS sources?

Yab Mas12:10:05

Like I said yesterday: to me it kind of makes sense that overwriting window.goog works (although it seems silly), as in: they initialise the goog library in their platform use it whereever. At some point they load our app.js where we reset it, reinitialise it and use it for our stuff. But again, I'm a bit out off depth here so maybe what Im saying doesnt make sense

thheller12:10:54

if you can point me to the relevant JS docs of the atlassian sdk I can maybe help

Yab Mas12:10:53

Yeah the server is in java. We use cljs/shadow to generate our js-files and output it into a folder thats configured to be loaded by their plugin system.

thheller12:10:16

otherwise you are on your own, the closure library is not designed to be loaded twice during development. I'm not quite sure why you'd be getting their development code anyways?

Yab Mas12:10:16

So I think making the two build one, as you suggested before is out of our reach

Yab Mas12:10:41

I'll see if I can find the appropriate docs

thheller12:10:00

the code would be even better. I mean you must include a script tag of their sources somewhere?

thheller12:10:16

or rather they include something and then include yours?

thheller12:10:44

I just need to see the JS code THEY load to make any kind of recommendation

Yab Mas12:10:58

I'm not completely sure why we're dealing with their dev-build either but I think they are generating files on the fly during dev them selves (again I might be wrong about how this all works)

Yab Mas12:10:47

or rather they include something and then include yours? To me this seems to be whats happening

Yab Mas12:10:00

I'll see what I can find

Yab Mas12:10:20

Thanks a lot for the effort already

Yab Mas13:10:57

I'm affraid Im not gonna be able to provide what you're looking for, that part is simply out of my reach. What I get to work with is a configuration file which details how/where I want to integrate with their product and in what location they can find the resource to load (in this case a js file generated with cljs/shadow) How they merge all of this into the end-result is inaccessible to me. I'm also confused why Im dealing with their dev builds/files during my dev time, but that seems to be whats happening and I've so far been enable to change that. So it seems to me there's no "clean" solution here. That being said, I'm wondering why the previously implemented hack of inserting window.goog = {}; doesnt make sense to you? To me it seems that all code (i.e. their platform and our plugin) would have a goog library available to it, we just destroy and reinitialise it within our part. It really does seem to have worked (as in, this can be proven by adding/removing the build hook), but doenst anymore. Also if I add the line to the top of my new js output file "by hand" it seems to work as well, that is: all goog related errors disapear, I still have 1 remaining error but it seems unrelated. I assumed working with the :unoptimizable key must have been removed in some newer version of shadow or doesn't have the same effect anymore. As mentioned before I tried adding the same line with :prepend but it shows up too late in the js output. I also tried to set goog-base of the module to false or removing all sources that start with "goog" but both give me erros (albeit different ones). The result I find under :output in the build state doesn't seem usefull to me at first sight either. Do you have any pointers based on the provided info? If not I'll leave it at this and see if I can find some hacky way to prepend the goog-reset to the beginning of the file outside shadow and/or try again if I can tweak the sdk setup.

Yab Mas14:10:39

I'm also reaching out on their developers platform btw, as this is in essence not cljs specific at all. Haven't found any existing threads on there though so I guess not many people run into this..

thheller15:10:52

:prepend is what :unoptimizable was

thheller15:10:00

there is nothing in shadow-cljs to do what you want to do, but you can sort of "hack" it yourself

thheller15:10:02

but well you end up with something that might not work in the future again

thheller15:10:22

:unoptimizable was never meant to be user-adjustable and was purely an internal thing

thheller15:10:30

my expectation would be that window.goog = {}; would break the "other" closure library use, but I guess if you never had issues thats fine

thheller15:10:48

you can just do it manually

thheller15:10:09

but I don't know anything about how your stuff is getting loaded or integrated I cannot say how

thheller15:10:24

<script>window.goog = {};</script> in the HTML if you have access to the HTML?

thheller15:10:00

I know literally nothing about anything atlassian, so I'm all guessing

Yab Mas16:10:26

:prepend is what :unoptimizable was Ok, thanks for the clarification. From what I can see in the js ouput it doesn't behave exactly the same though. I.e. in the old build the goog-reset was put all the way at the top of the output, in the new build it's put after some goog related initialisation. It doesn't really matter though, I understand what your saying and it's in line with what I understood about this being the result of reading the source instead of using some public api. If I continue with this "solution" I'll find a way to automatically inject it at the top of the file. I’m afraid you're right though as the one remaining "unrelated" error after injecting the goog reset by hand seems to be related after all... It was Uncaught TypeError: Cannot read properties of undefined (reading 'Dir') If I investigate the error in the browser console I get taken to a line with goog.i18n.bidi.Dir . This isnt in our code or anything we have a direct dependency on so Im doubting if my mental model stated above is correct and overwriting goog like proposed does actually have effects outside our plugin (as you seem to be expecting). But than it obviously confuses me even more that it works in the existing version of the plugin in the exact same jira-server version and suddenly breaks when I upgrade shadow (and cljs). Anyway, I think it's enough for today, I'll see if I can get any wiser tomorrow. I understand there's not much you can do and very much appreciate the help you already offert. I might keep you updated in case I find a solution, but Im not expecting anything unless I find some of the info you requested.

thheller16:10:38

you can just require that namespace from CLJS

thheller16:10:59

I assume you just "delete" it by window.goog = {}; and when the other code looks for it its gone

thheller16:10:07

if you provide it that should be fine

thheller16:10:21

all assuming of course that the likely different closure library versions are compatible

Yab Mas11:10:32

Ok, I looked into it a bit more and I'm now also surprised the previous hack worked at all. Apparently the goog versions at the time were compatible enough, but thats not the case anymore. The atlassian SDK is still on closure v20191111. I tried a few ways of resetting and swapping the googs but something is always missing something, either in their code or ours. I ended up overwriting your goog.provide via :prepend to:

goog.provide = function (name) {
     if (goog.isProvided_(name)) {
         console.log('Namespace "' + name + '" already declared.');
     } else {
         goog.constructNamespace_(name);
     }
 }; 
If I understand correctly what I'm doing (basically just meshing the two versions into one goog and praying all code gets what it needs) its pretty wacky but seems to work at first sight. The build runs, I clicked around a bit and made some changes to see if hot-reload behaves as expected. I would be very surprised if this doesn't blowup at some point, but I think it's my best option for now. Else I'm simply stuck with the old shadow (closure) version, but when I tried working with that I ran into some npm related issues which I don't get in projects with newer shadow versions (not 100% sure its caused by this though it's working fine now, pretty sure that was it), so I would preferably push trough with the upgrade. Hopefully someone from atlassian can help sort this out more properly. We'll see. Thanks again for sharing your thoughts!

madis17:10:43

Hello. I'm trying to figure out how to embed REPL server in a shadow-cljs node-script release build. It is to have a way of debugging some issues in the production application. Was reading through the shadow-cljs code and found this part https://github.com/thheller/shadow-cljs/blob/b6d0afbddc8952b67879bc9cc605526dc1cc0769/src/main/shadow/build/targets/node_script.clj#L43 But am unsure how to go from there Are there recommended ways of doing this in a way that only the release build JS file can be run via Node.js and the REPL server would start as part of the application process? Any recommendations and references welcome! Thank you!

thheller18:10:44

release builds don't support a REPL

madis13:10:40

Would there be a way to include a library or namespace (implemented as part of shadow-cljs or other project), that would offer REPL from a release build process? With my not too deep knowledge of the processes involved, I'd think it'd require the Node.js process to be able to compile CLJS forms, evaluate the compiled JS in the Node.js process and also bind to a port to allow incoming REPL client connections Do you see it feasible? If so, I'd very much appreciate any hints or pointers how to begin moving towards it. Even if only as a hacking and learning experiment. Thank you!

thheller13:10:50

well you could setup a self-hosted build and build the server yourself https://code.thheller.com/blog/shadow-cljs/2017/10/14/bootstrap-support.html

thheller13:10:20

that'll get very complicated quick though

thheller13:10:22

you'll also be limited to :simple optimizations but I guess for node thats not much of an issue

madis13:10:30

Thanks for the tip! I'll look into it

madis14:10:07

Btw, do you accept tips in Bitcoin (via Lightning)? I'd like to donate to you as I've benefitted much from your work. https://tippin.me/ or https://guides.getalby.com/alby-guides/alby-lightning-account/where-to-use-your-lightning-address/how-to-use-your-own-domain-as-your-lightning-address are the simplest ways to get started

thheller14:10:48

https://getalby.com/p/thheller no clue how this works, but maybe thats an excuse to finally look into it more 😛

madis14:10:03

Could you confirm you got them?

thheller14:10:10

it showed up on the alby website yes. what I do with it from there I don't know yet 😛

madis14:10:36

Awesome! As of what to do with it, if you were in El Salvador (where I am), you could use it to buy anything from the shops (Bitcoin is official currency here). As you seem to have open and curious mind, to learn more, you could check out 1. High level overview + economic aspects: The Bitcoin Standard (download PDF at https://drive.google.com/file/d/1tV_nb2ixZyl5p9EY_imjf4gKNviHvHE8/view?usp=sharing) 2. Technical side Bitcoin: https://github.com/bitcoinbook/bitcoinbook 3. Technical side Lightning: https://github.com/lnbook/lnbook

thheller14:10:12

if only I had more time 😛

madis14:10:30

I use Bitcoin because it protects me from the most heinous crime - theft of time - committed via the fiat currency (via taxes and inflation). Bitcoin is scarce (limited amount) and unconfiscatable, so I can convert my time (spent on work) into a money that can't be diluted (time theft)

thheller14:10:38

I absolutely want bitcoin to work, but so far call me a skeptic 😉

madis14:10:47

Thanks again for shadow-cljs and all the rest of your work and contributions to the Clojure community. Have a great day. Commence code mode 🤓

thheller14:10:11

btw I hope you do not run shadow-cljs on any public facing internet servers

thheller14:10:45

that would be extremely insecure. so whatever you do in the release node repl adventures, don't make shadow-cljs a part of it. except for building of course

thheller14:10:09

otherwise feel free to ask questions. happy to help. cheers.