Fork me on GitHub
#shadow-cljs
<
2021-05-08
>
thheller08:05:56

hmm odd since it is always compiling the same amount of files. maybe some external tool (eg. your editor) touching all the files all the time?

thheller08:05:42

but yeah most commonly this was caused by having :output-dir into something that is also a :source-paths entry

dvingo11:05:48

Thanks for the tip - I'll take a look again next week

martinklepsch09:05:25

With the node-library target, is there a way to include all dependencies from node_modules in the target JS file? (asking because that would be useful in the context of cloud function setups like AWS Lambda)

thheller09:05:52

you can post-process the :output-to with https://github.com/vercel/ncc or just include the proper node_modules for aws

thheller09:05:56

I wonder why everyone keeps asking this? Is this not the normal thing you'd do with AWS? https://docs.aws.amazon.com/lambda/latest/dg/nodejs-package.html

thheller09:05:42

I've never done anything with AWS so I really don't know what the best practices for this are but it seems straightforward?

martinklepsch10:05:29

Yeah, you’re right, I think just packaging with node_modules also works perfectly fine

martinklepsch11:05:36

Actually ncc does seem nice. One problem I’m seeing with it is that the source maps probably wouldn’t be accurate anymore(?) did anyone run into that?

thheller13:05:45

fwiw shadow-cljs also has some basic support for this directly. it does however have some issues so not all packages work. https://github.com/thheller/shadow-cljs/issues/290#issuecomment-459818765

thheller13:05:57

don't know about source maps and ncc

thheller13:05:47

in general I'd likely recommend creating a dedicated folder that you'll send to AWS

thheller13:05:00

create a package.json in that folder and run npm install --production

thheller13:05:21

that'll not install devDependencies and the resulting node_modules should be much smaller

thheller13:05:52

the just :output-dir "that-folder/lib.js" and just zip and deploy that-folder

martinklepsch12:05:34

Whats the performance impact of having lots of builds? If the cache is shared between them it shouldn’t be too bad right? (Is it?)

thheller13:05:04

cache is not shared between builds

martinklepsch18:05:38

So I guess that means initial build times grow with number of builds. What do you think would the impact be if you had, say 20 builds, for incremental builds? I realize this is a bit outside of the usual but just curious what your potential concerns would be with such setup

thheller21:05:18

I don't have a clue what you are building so impossible to comment

thheller21:05:35

shadow-cljs is very flexible so if the default targets don't do what you need it would be relatively easy to add one that fits better

martinklepsch22:05:57

I guess what I’m asking is how shadow will perform with one build vs. 20 identical ones with different target files.

thheller22:05:02

how am I supposed to answer that when I don't have a clue what you are building? shadow-cljs doesn't care, you can have a thousand builds, it does not matter to shadow. your computer will probable scream and blow up since memory and cpu consumption will go through the roof though

thheller22:05:30

I do not know what you are doing. do you intend to build all the builds in parallel? sequentially? watch all? just release all?

thheller22:05:23

what is the point of having 20 identical builds with different target files? why is it not one build and you copy the output file 20 times?

thheller22:05:42

you are only giving me very vague information so I really cannot give you any reasonable answer 😛

martinklepsch22:05:22

Ok fair enough 😄 Maybe I’ll try to write up more specific information tomorrow 🙂

Karol Wójcik12:05:06

@martinklepsch For AWS lambda the best possible approach is to pack dependencies and ship them as a layer

martinklepsch12:05:53

Ah, that’s interesting! Can you explain a bit more how this would manifest? Like if I have a lot of node_modules I’d put those in a layer and then the deploy payload would get smaller I guess? Does it have any impact on performance? Right now with ncc + advanced + zip I get pretty tiny lambdas (25kb) and the deployment story of a single JS file just seems more straightforward.

martinklepsch12:05:19

So I’m thinking layers yes, but probably only once your individual lambdas get too big?

martinklepsch12:05:42

@UJ1339K2B what do you use for deployment and resource automation? I just played around with https://arc.codes/ a bit and I’m really liking how little assumptions it makes around build tools and what not

Karol Wójcik12:05:09

Layers are super useful when you have a lot of dependencies since you pack your dependencies once. For this I strongly recommend trying yarn. I started using it 2 years ago and now I now that I will never go back npm. Regarding performance it should not vary much where dependencies are stored. I'm already doing the same for babashka runtime and I'm happy with the results. If your lambdas are 25kb it's probably not worth trying layers though. For deployment and resource automation I'm using AWS SAM. I have already used Serverless Framework and I find AWS SAM the simplest and the best solution. I have never tried Architect , but for me it's just a sugar around AWS SAM and cloudformation. For me having all resources of the stack in template.yml is super simple. https://github.com/FieryCod/holy-lambda/blob/master/examples/bb/native/sqs.example/template.yml Btw I'm planning to add support for Node.js in holy-lambda, so if you would like to help me out it would be lovely.

martinklepsch22:05:37

wow, this looks very cool @UJ1339K2B, I’ve just been thinking that babashka tasks seems like the perfect foundation for this type of thing

martinklepsch22:05:36

Unfortunately he cljdoc build is broken btw, if you run the following in your repo you can see an issue with cljdoc.edn

curl -fsSL  | bash -s doc/cljdoc.edn

martinklepsch22:05:48

This can also be used as a CI step and will fail if there is an issue

martinklepsch23:05:37

> so if you would like to help me out it would be lovely. I’ve played with shadow-cljs quite a bit this weekend to see how it worked with different packaging strategies/deployment tools (like Architect) — if that sounds interesting maybe we can have a chat?

Karol Wójcik09:05:56

Let's have a chat then 🙂

martinklepsch12:05:42

Did some exploration of using https://arc.codes to deploy CLJS to AWS Lambda today: https://github.com/martinklepsch/shadow-cloud-fns/tree/master/trying-arc — it all works pretty well. The only thing I’m still a little unsure about is the ncc source maps situation https://clojurians.slack.com/archives/C6N245JGG/p1620472416489100

pez16:05:38

I’m trying to set up a React Native project using Fulcro 3. I get a strange exception when I run shadow-cljs watch :app. About piggieback. My deps.edn looks like so:

{:paths   ["src/main"]
 :deps    {org.clojure/clojure    {:mvn/version "1.10.1"}
           com.fulcrologic/fulcro {:mvn/version "3.4.21"}}

 :aliases {:dev {:extra-paths ["src/dev"]
                 :extra-deps  {org.clojure/clojurescript   {:mvn/version "1.10.742"}
                               thheller/shadow-cljs        {:mvn/version "2.12.5"}
                               binaryage/devtools          {:mvn/version "0.9.10"}}}}}
shadow-cljs.edn:
{:deps {:aliases [:dev]}
 :builds
 {:app
  {:target :react-native
   :init-fn 
   :output-dir "app"
   :devtools {:autoload true
              :preloads [shadow.expo.keep-awake]}}}}
The exception:
~/Projects/rn-fulcro-shadow(main|✚3…) % npx shadow-cljs watch :app
shadow-cljs - config: /Users/pez/Projects/rn-fulcro-shadow/shadow-cljs.edn
shadow-cljs - starting via "clojure"
WARNING: When invoking clojure.main, use -M
[2021-05-08 18:43:55.123 - WARNING] TCP Port 9630 in use.
[2021-05-08 18:43:55.363 - WARNING] :shadow.cljs.devtools.server/nrepl-ex
Note: The following stack trace applies to the reader or compiler, your code was not executed.
CompilerException Unexpected error macroexpanding if-ns at (cider/piggieback.clj:22:1). #:clojure.error{:phase :macroexpansion, :line 22, :column 1, :source "cider/piggieback.clj", :symbol if-ns}
        clojure.lang.Compiler.macroexpand1 (Compiler.java:7019)
        clojure.lang.Compiler.macroexpand (Compiler.java:7075)
        clojure.lang.Compiler.eval (Compiler.java:7161)
        clojure.lang.Compiler.load (Compiler.java:7636)
        clojure.lang.RT.loadResourceScript (RT.java:381)
        clojure.lang.RT.loadResourceScript (RT.java:372)
        clojure.lang.RT.load (RT.java:459)
        clojure.lang.RT.load (RT.java:424)
        clojure.core/load/fn--6839 (core.clj:6126)
        clojure.core/load (core.clj:6125)
        clojure.core/load (core.clj:6109)
        clojure.core/load-one (core.clj:5908)
Caused by:
ExceptionInInitializerError 
        java.lang.Class.forName0 (Class.java:-2)
        java.lang.Class.forName (Class.java:468)
        clojure.lang.RT.classForName (RT.java:2211)
        clojure.lang.RT.classForName (RT.java:2220)
        clojure.lang.RT.loadClassForName (RT.java:2239)
        clojure.lang.RT.load (RT.java:449)
        clojure.lang.RT.load (RT.java:424)
        clojure.core/load/fn--6839 (core.clj:6126)
        clojure.core/load (core.clj:6125)
        clojure.core/load (core.clj:6109)
        clojure.core/load-one (core.clj:5908)
        clojure.core/load-one (core.clj:5903)
Caused by:
ClassNotFoundException com.google.javascript.jscomp.AnonymousFunctionNamingPolicy
        jdk.internal.loader.BuiltinClassLoader.loadClass (BuiltinClassLoader.java:606)
        jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass (ClassLoaders.java:168)
        java.lang.ClassLoader.loadClass (ClassLoader.java:522)
        java.lang.Class.forName0 (Class.java:-2)
        java.lang.Class.forName (Class.java:468)
        clojure.lang.RT.classForName (RT.java:2211)
        clojure.lang.RT.classForNameNonLoading (RT.java:2224)
        cljs.closure/loading--5569--auto----3990 (closure.clj:9)
        cljs.closure__init.load (:9)
        cljs.closure__init.<clinit> (:-1)
        java.lang.Class.forName0 (Class.java:-2)
        java.lang.Class.forName (Class.java:468)

p-himik16:05:52

To add to thheller's answer, according to documentation, you simply don't need to specify any version of ClojureScript - shadow-cljs already depends on the needed one. That is, unless you need a very specific version of ClojureScript. https://shadow-cljs.github.io/docs/UsersGuide.html#failed-to-load

pez17:05:07

Thanks! I'll send a PR on the fulcro book about it.

👍 3
thheller16:05:00

@pez wrong CLJS version. 2.12.x requires 1.10.844

pez17:05:00

Thanks. Will try it out when I'm back at the keyboard. But, I thought no piggieback was involved?

thheller18:05:09

shadow-cljs does not use it but it is added for compatibility with stuff that expects it (eg. cider-nrepl)

codeasone22:05:20

I've prepared a minimal repo and README capturing the difficulties I encountered getting a shadow-cljs plus tools.deps project working well with cider after transitioning over from years of clojure-cli plus figwheel-main https://github.com/codeasone/starter-cider-tools-deps-shadow - it was certainly more painful than I'd like it to have been.

thheller22:05:54

that seems to be much more complicated than it needs to be

codeasone22:05:27

yeah, I would have been delighted if I could just run watch in a terminal, and establish a sibling cljs REPL connection, but I can't make that work

codeasone22:05:09

switching between and eval-ing forms in clj cljc and cljs code and having it target the correct REPL is essential for my workflow

thheller22:05:17

why not? I mean cider does have a remote connect open, but everything you described seemed to be based on jack-in?

codeasone22:05:55

I made a typo in one of the heading https://github.com/codeasone/starter-cider-tools-deps-shadow#jacking-in-to-clj-repl-and-adding-a-sibling-jack-in-for-shadow-cljs I actually tried establishing a M-x cider-connect-sibling-cljs which would have been ideal, but as I say didn't work out due to the way REPL sessions are managed.

thheller22:05:39

I do not have a clue what the commands are called in emacs. the way I work is I run my backend manually outside my editor, typically just lein repl. then I start shadow-cljs server separately.

thheller22:05:09

then I connect to the .nrepl-port for my CLJ REPL and .shadow-cljs/nrepl.port for the CLJS REPLs

thheller22:05:26

two separate processes, completely separate connections, nothing shared

thheller22:05:59

I would expect emacs to be capable of doing the same but I don't have a clue

thheller22:05:17

you can instead run shadow-cljs in embedded mode so it runs in your backend process

thheller22:05:37

you sort of flipped it and run your backend process in the shadow-cljs process. not something I recommend but you can do it.

codeasone22:05:21

A moment of inspiration struck me, I just wanted something that worked and fit my muscle memory of how to bring up my full-stack, I honestly don't know the intent behind shadow.user ns, and whether my usage is misuse, but I defer to your advice of course.

thheller22:05:48

1. Unhandled clojure.lang.ExceptionInfo
   shadow-cljs has not been started yet!  In embedded mode you need to call
   (shadow.cljs.devtools.server/start!) to start it.  If you have a shadow-cljs
   server or watch running then you are not connected to that process.

thheller22:05:23

did you ever try that? I mean running the shadow-cljs server in embedded mode in your backend process?

thheller22:05:55

I assume that is how you ran figwheel previously?

thheller22:05:33

> Picking up on shadow-cljs has not been started yet! I try running shadow-cljs watch app in a separate terminal (which worked fine):

thheller22:05:17

running shadow-cljs watch will start a new JVM process, instead you are supposed to run (shadow.cljs.devtools.server/start!) in whatever JVM you were connected to when you got that error

thheller22:05:03

> But I just started it!

thheller22:05:01

exactly the point, you started it separately but still talk to the old JVM

codeasone22:05:08

Re: figwheel I just invoke vanilla cider-jack-in-clj&cljs and it just works, clj REPL spins up - I (start)the backend server and figwheel is piggybacked in to the same cider session, all orchestrated by Emacs

thheller22:05:27

what is (start)? I'm guessing thats a user.clj function you have in your codebase that starts figwheel in embedded mode

codeasone22:05:33

wanted to get to same single-step bring up under Emacs, that's the point of the repo

codeasone22:05:05

(start) is just something to bring up the backend - just an example of something you might type into the REPL to start the HTTP server for instance

codeasone22:05:13

as per the demo-rig code, which I tried to keep as minimal as possible, but reflect the various moving parts I need to content with day-to-day in my production code.

thheller22:05:43

but this is not minimal at all

codeasone22:05:54

okay, I wanted some clj some cljccode and some cljs code so I could ensure I could eval forms and the appropriate REPL would be targetted

codeasone22:05:16

I'm also learning to use shadow-cljs and understand it's capabilities in the process

thheller22:05:21

sure I get that

thheller22:05:07

to me it sounds like all your were missing is instead of calling just (start) you also call (shadow.cljs.devtools.server/start!)

thheller22:05:35

or have the (start) do that for you wherever that was coming from. I assume you did something like that with figwheel

codeasone22:05:40

I'll try that out soon, thanks

thheller22:05:40

point is when you run cider jack-in that starts a JVM for you with a nrepl server. if you then run shadow-cljs watch separately you'll get a second JVM instance with its own nrepl server.

codeasone22:05:02

Going to hit the sack now though, thanks for the feedback

thheller22:05:26

how you deal with that in emacs I don't know. if emacs doesn't support two separate remote connections then you likely want to run shadow in embedded mode https://shadow-cljs.github.io/docs/UsersGuide.html#embedded

thheller22:05:57

but you can also just run in embedded mode but you sort of flipped it to embed everything in shadow-cljs instead

codeasone22:05:58

I end up with two "sessions" in cider and the way they are managed is not smart enough

thheller22:05:14

adding a custom shadow.user ns is definitely not the way to go here

codeasone22:05:22

the only thing that works - and it works well - is to have cider-jack-in-clj&cljs control everything - it's also really simple to bring everything up with one key binding