Fork me on GitHub
#shadow-cljs
<
2023-10-05
>
James Amberger00:10:40

Any tips for compiling in low-memory (512MB) environments? I allocated a GB of swap but it only uses a little and then I get OOM.

thheller05:10:16

if its running in a container then memory allocation may be wrong on older jvm versions

thheller05:10:53

otherwise 512mb is borderline, take away all the OS and stuff you can be lucky to have 256mb for shadow-cljs

thheller05:10:37

which will likely fail when trying to make a release build, as the closure compiler probably requires more. :advanced is pretty memory intense

thheller07:10:26

fwiw you can try :jvm-opts ["-Xmx300M"] in shadow-cljs.edn top level for 300mb heap limit. only works when not using project.clj or deps.edn to manage dependencies.

James Amberger16:10:42

Thank you @U05224H0W; worked for me.

Daniel Gerson13:10:01

Hello! I've read the great blog post :https://code.thheller.com/blog/shadow-cljs/2020/05/08/how-about-webpack-now.html#option-2-js-provider-external:. I'd like to https://callstack.github.io/react-native-paper/docs/guides/react-native-web#configure-webpack react-native-vector-icons , but also I'd like to rely on https://shopify.github.io/react-native-skia/docs/getting-started/web#loading-skia react-native-skia in the browser. This is in an effort to get both toolsets via react-native-web. Is this not possible with shadow-cljs? (I haven't tried to use code-splitting with :external, but thought I'd ask ahead of time).

thheller13:10:10

you can try https://shadow-cljs.github.io/docs/UsersGuide.html#_js_tree_shaking but not sure how far you'll get. don't know any of these libs or how they are packaged

Daniel Gerson15:10:27

Thanks for the reply. Going to try a different tack first.

[:standalone] Build failure:
Failed to inspect file
  /Users/dmg46664/projects/xxxx/node_modules/react-native-paper/lib/commonjs/assets/back-chevron.png
Is there a way to tell shadow-cljs to not try to resolve https://github.com/callstack/react-native-paper/blob/d698282af14555cef8e546e4ead00bf6fd608e56/src/components/Appbar/AppbarBackIcon.tsx#L21 I don't even need it to do what webpack does because I'm not using that part of the lib. (webpack would convert it into an asset loadable resource)
{
  test: /\.(jpg|png|woff|woff2|eot|ttf|svg)$/,
  type: 'asset/resource'
}

thheller15:10:39

well that wouldn't work as webpack will then return the url for that image for that call

thheller15:10:55

shadow-cljs doesn't support bundling static assets that way

thheller15:10:27

you can use :js-options {:ignore-asset-requires true} instead, but that'll effectively set source to false

👀 1
Daniel Gerson15:10:00

Can I do that only for a particular npm?

thheller15:10:32

what would that help? shadow-cljs doesn't support any of them, so just ignoring 1 doesn't get you anywhere 😛

Daniel Gerson15:10:47

Except that way I would be aware of new problems on new libraries.

thheller15:10:04

last time I looked at react-native-web it very much expected webpack, not sure if this has changed. it has been a couple years.

👍 1
Daniel Gerson15:10:51

So close 😅

[:standalone] Build failure:
Closure compilation failed with 20 errors
--- node_modules/react-native-paper/lib/commonjs/components/Menu/Menu.js:91
Transpilation of 'Member references this or super' is not yet implemented.

thheller15:10:49

hehe yeah the closure compiler is a bit behind on cutting edge not-even-standard-yet features 😛

thheller15:10:31

although that might go away with :compiler-options {:output-feature-set :es-next}

Daniel Gerson15:10:31

Thanks, but no gravy. Pity that shadow can't run babel plugins on the code before bundling... ;-)

Daniel Gerson07:10:05

Morning @U05224H0W, I tried to webpack the two modules regexing everything else in externals but now I realise that Transpilation of 'Member references this or super' is not yet implemented.isn't a babel issue. There are modules that the Closure compiler just can't do 🤯 . Lends credence to the following argument: https://tonsky.me/blog/clojurescript-2/

thheller07:10:16

Transpilation of 'Member references this or super' is not yet how do you get this error when using :js-provider :external?

thheller07:10:27

that shouldn't be possible?

Daniel Gerson07:10:24

No, I webpacked those two modules to a new file and then included them. The externals was in the webpack config.

thheller07:10:49

literally don't know what that means

thheller07:10:03

but post processing the shadow-cljs output with webpack seems like a bad idea

Daniel Gerson07:10:26

Otherway around, post process webpack output with shadow.

thheller07:10:06

hmm? how would that work?

Daniel Gerson07:10:40

My thinking was, if shadow can't prep these modules. I'll bundle them in webpack, externalising everything else.

thheller07:10:23

what does "externalizing everything" mean? my webpack-fu is limited, so I don't know what that means

thheller07:10:26

do you have a repo so I can see the setup? I'm kinda lost what your goal is at this point

thheller07:10:38

what was the issue with :js-provider :external?

Daniel Gerson07:10:49

Or https://webpack.js.org/configuration/externals/ . My webpack foo is always limited, but I thought what the hell, give it a try 😅 .

Daniel Gerson07:10:36

I'll prep a repo. The problem isn't that I only need to output webpack :js-provider :external as a part solution. As per the original post, I also need to code-split the clojure code because of a second library.

thheller07:10:19

and why is that?

thheller07:10:27

I mean code splitting the CLJS code is trivial

thheller07:10:51

code splitting the JS lib is not, but I guess thats what you actually want?

thheller07:10:44

I mean you linked this https://shopify.github.io/react-native-skia/docs/getting-started/web/#loading-skia and I see no problem with this working in CLJS?

thheller07:10:08

with :external probably, to do the actual splitting of the JS code?

thheller07:10:34

oh wait nvm

thheller07:10:58

I actually do not get what this is doing

thheller07:10:50

import React from 'react';
import { Text } from "react-native";
// Notice the import path `@shopify/react-native-skia/lib/module/web`
// This is important only to pull the code responsible for loading Skia.
// @ts-expect-error
import { WithSkiaWeb } from "@shopify/react-native-skia/lib/module/web";
export default function App() {
  return (
    <WithSkiaWeb
      getComponent={() => import("./MySkiaComponent")}
      fallback={<Text>Loading Skia...</Text>}
    />
  );
}

thheller07:10:58

what is MySkiaComponent in this case? I don't get it

thheller07:10:37

honestly I get nothing of this 😛

😅 1
Daniel Gerson08:10:27

I see at least 4 bundles. 1) CLJS code that includes the outer Skia shim to pre load the WASM. 2) The webpack canvas WASM bundle which :js-provider :external would force me to manage 3) CLJS My main clojure app 4) A webpack bundle including`react-native-paper`

thheller08:10:05

I see one external file and one cljs output?

thheller08:10:13

a repo would really help 😛

Daniel Gerson08:10:03

There is an extra kink. The reason I want code (1) instead of "the external file", is because (1) will eventually be https://github.com/cjohansen/portfolio once https://github.com/cjohansen/portfolio/issues/12 is solved (there's a plan).

Daniel Gerson08:10:40

Yeah, will prep a repo.

Daniel Gerson08:10:28

MySkiaComponent is your main app. It needs to be loaded AFTER the wasm code loads.

thheller08:10:44

CLJS code splitting can do that just fine?

Daniel Gerson08:10:14

Ok, so the answer to my original question is "Yes, you can combine code-splitting and js-provider external"

Daniel Gerson08:10:48

Thanks, and sorry for the run around.

Daniel Gerson08:10:08

I guess I knew that shadow does a lot of work of interrogating dependencies and will move code between modules based on its dependencies. I couldn't see how that would work if deps were all external, so assumed code-splitting may not be catered for. That was behind my original question.

thheller08:10:35

I still understand nothing about this. CLJS side code splitting is not affected by :external in any way, it just works

thheller08:10:49

it however collects ALL npm dependencies into one .js file

thheller08:10:14

but that doesn't mean webpack can't split that further

thheller08:10:33

but this is an optimization problem? I'd focus on getting this to work first? or am I missing something?

Daniel Gerson08:10:58

No, not an optimisation problem.

Daniel Gerson08:10:37

Will try code-splitting and js external, and only come back if I have further problems 😅

thheller08:10:13

I'll happily look at a repo and what the actual challenge is. I'm kinda lost 😛

Daniel Gerson12:10:14

@U05224H0W be careful what you offer 😛. I was trying to recreate the repo until I got stuck. I wrote it up here: https://github.com/dmg46664/problems/blob/main/0010_shadow_rn_paper/README.md. The 0010_shadow... directory represents the issue.

thheller13:10:11

I'll take a look later

🙌 1
Daniel Gerson14:10:53

Hmm, I'll take a look. I started from here https://github.com/dmg46664/problems/tree/main/0009_portfolio_skia_render which successfully does the same break up with shadow (without webpack)... but perhaps moving to :external means I need that now. Thanks for spotting, will try and revert.

thheller14:10:08

btw, this is a bad idea. public should never be on the classpath.

:paths ["src"
         "public"]

thheller14:10:31

:aliases
 {:standalone {:extra-paths ["src-standalone"]}
  :portfolio  {:extra-paths ["src-portfolio"]}}}
this also doesn't seam ideal, kinda annoying to have to restart shadow-cljs to do the other build

thheller14:10:37

just put everything in src

thheller14:10:47

export const Skia = JsiSkApi(global.CanvasKit);

Daniel Gerson14:10:24

Thanks for the tip re-classpath. The whole react-native-web thing is just to speed up dev, and include Portfolio, so no security concern. Got it from here https://github.com/cjohansen/sasha/blob/main/shadow-cljs.edn#L10 😅

thheller14:10:21

not sure I understand what you mean. :dev-http is not related to :paths?

thheller14:10:41

its fine to use it as a http root, its not fine to use it as a classpath root

Daniel Gerson14:10:02

I.e. it's in the setup "sample" instruction here: https://github.com/cjohansen/portfolio

thheller14:10:08

:paths ["src"] is fine

thheller14:10:05

sorry I'm lost. I'm not at all talking about :dev-http. maybe you misunderstand what :paths ["src" "public"] does. you don't need it to make this :dev-http work.

Daniel Gerson14:10:52

I understand what it does re my own source files, but in terms of why it was on that link I posted above, I don't know. Just copy-pasta'd it.

thheller14:10:10

I can't see it anywhere in the portfolio docs

thheller14:10:24

:dev-http {9800 ["public" "classpath:public"]} this line yes, and this is fine

thheller14:10:32

just not the :paths in deps.edn

Daniel Gerson14:10:44

"The source code for the sample is https://github.com/cjohansen/sasha." That's where I copied it from.

thheller14:10:36

this also has no public anywhere in :paths. I don't really care where you got it from, I'm saying it is better to remove it and save yourself from some headaches this can produce

Daniel Gerson14:10:10

My bad, I was confusing the two.

thheller15:10:23

this whole react-native-web thing is such an extreme mess, everything arround it is just picard-facepalm

thheller15:10:49

one thing you were missing to even active webpack code splitting was

:external-index "dist/externs.js"
    :external-index-format :esm

thheller15:10:38

but that doesn't change anything

thheller15:10:30

also note that :external-index "dist/externs.js" is not externs, externs are an entirely different thing. just to be clear, doesn't matter what you call it. might get confusing though 😛

thheller15:10:57

ok, so I think I get it now. it is totally relying on some built-in webpack semantics of how/when which code gets loaded, and since the external-index makes it look like everything is used at once for webpack it doesn't split as expected by the skia stuff

Daniel Gerson15:10:04

Sorry, yeah. Lots going on and I probably didn't do the best job explaining. Just note that I was able to replicate the webpack example only using shadow... but when I have to use webpack (with which I am liable to burn myself) for react-native-paper, then it's harder for me to reason about.

thheller15:10:28

what is react-native-paper? 😛

thheller15:10:45

I mean I can get the skia stuff working just fine without :js-provider :external

thheller15:10:02

but I guess you wanted that because of the paper thing?

Daniel Gerson15:10:13

https://reactnativepaper.com/ meant to work with both react-native and web.

Daniel Gerson15:10:22

Yeah, as I said before I got Skia stuff working fine without webpack as well 🙂

thheller15:10:12

yes, but you named the problem 0010_shadow_rn_paper but there is no use or mention of paper anywhere

thheller15:10:34

so the actual propblem that is demonstrating is that :external doesn't work for the skia stuff

thheller15:10:41

and as such the paper thing doesn't work either?

Daniel Gerson15:10:14

:face_palm: Yeah, the one thing I forgot to include in the demo app was react-native-paper . Too much juggling.

Daniel Gerson15:10:17

Paper needs babel stuff for webpack, Skia just needs code splitting (doesn't care how).

thheller15:10:32

I think I have an idea for some :external tweaks that might make this work

👀 1
Daniel Gerson15:10:27

I think I stopped prepping 0010 at the first error, so hadn't got to the paper part.

Daniel Gerson15:10:15

Paper also has stuff that doesn't go through the Closure compiler...

thheller13:10:24

I found a solution to make :external work with code splitting, but it'll require a little more work to get to work in :advanced properly

thheller13:10:43

worked in my tests but still breaks in your repo

thheller13:10:54

but I now see a path to get this working properly

Daniel Gerson11:10:09

@U05224H0W Thanks! I missed these updates. Looking forward to seeing the shape of the solution.

Daniel Gerson10:11:18

@U05224H0W Hello, should I create an issue for this? I'm a little scared to attempt to describe it as it took a while in the first attempt.

thheller11:11:17

what I tried didn't quite work out, or turned out to be much more work than anticipated

Daniel Gerson02:11:30

Thanks for the update! :-)

David Udelson23:10:07

Hi, I am new to cljs and shadow and I'm doing an experiment with integrating with an existing next.js codebase. I am targeting ESM, but when I import the js file generated by shadow in my next project, I get error error Error [ReferenceError]: SHADOW_ENV is not defined. I searched this channel and saw in some cases you must include shared.js first, but I do not see such a file being output by my build. I assume I'm making a simple mistake, any guidance would be appreciated. Thanks.

David Udelson23:10:18

This is the build section of my shadow-cljs.edn:

:builds
 {:app
  {:target :esm
   :output-dir "public"
   :runtime :node
   :js-options {:js-provider :import}
   :modules 
   {:hello {:exports {hellofn hello/hello-fn}}}
   }}}

thheller06:10:51

it has been a while since I looked at next.js. there is no shared.js file to load there, you only have the public/hello.js, which I assume you are loading from the JS code

thheller06:10:41

what is the full error? like what it trying to access SHADOW_ENV?

thheller06:10:31

this was my last experiment in that area

thheller06:10:49

I doubt things have changed much and my recommendation is still to not use it 😛