Hello! Newb here! I’m having a problem requiring javascript modules from cljs. I’ve created a super-simple reproduction (~4 lines). Googling my error comes up with nothing. Can anybody help me? I’m compiling with npx shadow-cljs compile npm.

@cachapline8 ☝️

@gdwarner you should have builds configuration in your shadow-CLJS.edn

If you target node, you can check this repo

https://github.com/minimal-xyz/minimal-shadow-cljs-nodejs

In perticular this file

https://github.com/minimal-xyz/minimal-shadow-cljs-nodejs/blob/master/shadow-cljs.edn

Hi. I have a problem with a macro, after I switched from figwheel to shadow-cljs. My macro (defmacro assert ...) lives in kunagi_base/utils.cljc. I have no :require-macros or anything like that in my code. With figwheel this worked fine. Shadow-cljs gives me this error: Use of undeclared Var kunagi-base.utils/assert. Any suggestions? Thank you!

@gdwarner and what is your error? that snippet says nothing without more context

@witek see https://code.thheller.com/blog/shadow-cljs/2019/10/12/clojurescript-macros.html you might have been relying on https://clojure.atlassian.net/browse/CLJS-2454

#?(:cljs (:require-macros [kunagi-base.utils])) is likely missing in that ns

When using release, I get Uncaught TypeError: Cannot read property 'g' of undefined. Wasn't there a way to create release output with hints to the real (not shortened) names?

@witek shadow-cljs release app --pseudo-names will make the names slightly more recoginizable but you probably want to turn on https://shadow-cljs.github.io/docs/UsersGuide.html#infer-externs if you haven't already

Trying to use shadow-cljs in Cursive with the pom.xml method. Should Cursive be able to resolve references to node modules (or javascript references such as (.log js/console "...")), or is it expected that it doesn’t?

I’m still seeing problems when restarting the shadow-cljs build. The process is, I kill shadow-cljs to add a new dependency in deps.edn, then start it again. Then a bunch of errors appear, at their root:

TypeError: module$node_modules$react$index.createContext is not a function
    at eval (routes.cljs:10)
    at eval (routes.cljs:10)
    at eval (<anonymous>)
    at Object.goog.globalEval (main.js:827)
    at Object.env.evalLoad (main.js:2174)
    at main.js:2412

If I delete all compiled output, the shadow-cljs cache, and node_modules, then starting the build again, everything compiles fine.

I get a lot of similar errors too: TypeError: _react.default.createContext is not a function, TypeError: React.forwardRef is not a function, TypeError: module$node_modules$react$index.memo is not a function and similar.

not much I can say about that. maybe you have some bad browser cache?

what setup do you use? the shadow-cljs http servers or something custom?

Custom http server, but with browser cache disabled.

next time this happens. DO NOT DELETE ANYTHING!

Is there any way I could trouble shoot this?

impossible to tell what is happening if you wipe everything

at least copy it somewhere

Roger that. I’ll keep a copy of the .shadow-cljs, node_modules and the compiled output.

also keep more errors ... excerpts of errors are pretty much pointless. it is likely one error cascading out and causing all the rest

service workers can also be a big problem, any cache really.

Is there something I should be looking for?

I don't know

It looks like that React itself has some issue. It could be some dependency but I can’t be sure at this point.

it looks to me like react isn't loaded at all

everything is undefined

Yep, that seems to be the case. One thing that I see in the compiled output is that while I have a “main.js” file which is 11MB, there’s also a whole bunch of individual files under cljs-runtime.

sounds normal

are you sure the "old" shadow-cljs instance was actually dead?

things will get weird if 2 shadow-cljs instances write to the same output-dir

@neo2551 I’m targeting node, but with the npm-module variant. The docs say that I can use the default shadow-cljs.edn via the ‘covenience mode’. https://shadow-cljs.github.io/docs/UsersGuide.html#_convenience_mode .

yeah that should mention that is a convenience mode for webpack

it works if you stay in CLJS only

but the JS support needs to know which runtime you are running in

trying… hopefully that is the missing piece

Same problem.

@gdwarner try adding :entries [hi] to the build config?

definitely a bug since it is supposed to work without, but maybe a quick fix

Still no worky with the config below. Regardless, I appreciate your help!

does it matter: hi vs “hi”. Or do I need to include “hello” since that’s what I’m trying to require?

thats not in the build config

should be in :builds :app

ah

symbol vs string matters yes. always a symbol for CLJS namespaces

but you only need to list the "entries". meaning namespaces accessed from elsewhere (eg. require in node). it'll include its dependencies automatically

but yeah its just a global missing

any particular reason you are using :npm-module? :node-library and :node-script are much better generally

quick fix might be global.shadow$provide = {} before running the require in node

I want to call my clojure lib/module from javascript. I don’t think I want node-script, but node-library may be appropriate. I have no intention of actually creating an npm module.

As I’m still getting the same error, even with entries, I’ll try node-library.

hang on

try the quick fix I just mentioned

will do

so I know if it works without having to test myself 😛

but yeah node-library has a better structure and better "output"

has nothing to do with actually publishing to npm at all

ok, your suggested fix eliminated the stack trace I was seeing.

if you don't mind please open an issue with the initial snippet you pasted

will look into it later

sure thing

thank you very much!

If one of my deps is described as “cljs support coming soon”, but contains (temporarily) incorrect .cljc files, is there a way to tell shadow to ignore that dep during cljs compilation? I don’t intend to use it from cljs, but do want it on my classpath at the clj REPL.

@cjsauer not sure what you mean. what is the actual problem you see? just files on the classpath shouldn't hurt unless you use them?

When I add this specific dependency to my project, it breaks cljs compilation by throwing a bunch of warnings about undefined vars in my test files. Removing the offending dep fixes things. I’m guessing that the bad .cljc files in that dep are causing cljs compilation to fail in some way.

breaks how ...

it doesn't matter whats on the classpath UNLESS you actually :require a file

or do you add the dependency via deps.edn :local/root or gitlibs and use a test target?

Yes, forgot to mention I’m using {:deps {:aliases [...]}}

that isn't what I asked 😛

:local/root or gitlibs was the important bit 😛

Sorry , no it’s a normal :mvn/version style dependency

I don’t require this specific dep in the *-test.cljs files that the warnings point to… :thinking_face:

what does it say exactly?

I mean it should be telling you who required something if thats not found?

It points at about 6 of my deftest macros with an undeclared var warning, for example:

(deftest entity-ids->lookup-set
Use of undeclared Var my-app.client-test/entity-ids->lookup-set

I'm confused. how is that related to adding a dependency?

seems to be warning about your namespaces?

Yeah…not sure…removing the dependency also removes all of those warnings strangely enough.

what dependency is that?

This is datahike

please full info

with version

[io.replikativ/datahike "0.2.0"] https://github.com/replikativ/datahike

Attempting to compile the .cljc files in that repo fails with all kinds of exceptions, because the cljs support isn’t there yet. My thought was those files were somehow tripping up my own compilation. But: >it doesn’t matter whats on the classpath UNLESS you actually :require a file

no. unless a file is included somehow it is not compiled

it doesn't matter what is in the file

> Attempting to compile the .cljc

so we want to figure out WHY it is attempting to compile

BUT your issues seem related to something else

Ah sorry, by that I meant cloning that repo and running their cljs tests doesn’t work (which is expected). It fails compilation.

I'm lost ... what are you doing?

you are cloning actual datahike and run the tests?

please don't throw in unrelated info .. this is confusing enough as it is 😛

This was when I thought maybe datahike’s cljc files was to blame >BUT your issues seem related to something else This is likely the case now that I know shadow is likely not touching any datahike code

please restate your problem. the actual warnings you see ... not your diagnosis.

I'm completely lost and don't know what your problem is anymore

a) you have working tests

b) adding the datahike dependency breaks those working tests?

c) NONE of your code actually has a datahike require? anywhere?

Sure, let me restate

a) Yeah, I have working tests using the :browser-test build target. I can open the resulting build up in the browser and see All tests passed. b) I add the datahike dependency to my deps.edn file, and then rerun my :browser-test build, which results in a bunch of warnings, but I just found that there is one interesting one:

Resource: datascript/core.cljc:586:36
 Use of undeclared Var cljs.reader/register-tag-parser!

please run shadow-cljs clj-repl and ( "cljs/reader.cljs")

could also just be a dependency conflict

Any suggestion on how to move everything into a new directory, e.g. dist/ when run npm run build? I understand this might not be shadowcljs’ job, but wonder how people do it.

what is npm run build?

It’s the npm-script. I usually config something like

"scripts": {
    "start": "shadow-cljs watch app",
    "build": "shadow-cljs release app"
  },

so why do you want to copy it? why not just have shadow-cljs write the output dir you actually want?

if you really need to copy you can just use "build": "shadow-cljs release app && cp dist somewhere" or so?

It’s just my preference to have everything built to a separate folder. If you dev and release in the same folder, say public/js/, then do you commit this folder in VCS, say git? I do want to commit the release version, but then I will always see it changing in git status when I dev.

something like shadow-cljs release app && cp dist somewhere would work, but then before I copy, I assume I need to clean up the dev files? But shadow-cljs doesn’t have a clean command, so…

Just wondering how others do it, if one ever does.

in https://github.com/filipesilva/create-cljs-app/blob/master/template/package.json I instead use a clean command

"start": "shadow-cljs watch app",
    "build": "yarn clean && shadow-cljs release app",
    "clean": "rimraf public/js"

for the same reasons really

when deploying dist/, I don't want to include the non-prod js files that get produced there

Ah nice. So I can clean, build, and copy.

@filipematossilva do you have any plans to add something like react-scripts for create-cljs-app?😁

so abstracting away all the tools in a separate package, right?

I thought about it but I think that runs a bit contrary to the clj ethos... although this starter is directed to js folks

I also don't feel that confident in my knowledge of the "best" way of doing things

@filipematossilva would you mind changing the default generated config a bit. I'd rather it used the newer :dev-http config

so instead of

{:builds
   {:app
      {:asset-path "/js",
       :devtools {:http-port 3000, :http-root "public"},
       :modules {:main {:init-fn app.core/main}},
       :output-dir "public/js",
       :target :browser},
    :e2e
      {:ns-regexp "e2e.*",
       :output-to "out/e2e.js",
       :target :node-test},
    :test
      {:ns-regexp "app.*-spec$",
       :output-to "out/test.js",
       :target :node-test}},
 :dependencies [[reagent "0.8.1"]],
 :nrepl {:port 3333},
 :source-paths ["src" "e2e"]}

{:builds
 {:app
  {:asset-path "/js",
   :modules {:main {:init-fn app.core/main}},
   :output-dir "public/js",
   :target :browser},
  :e2e
  {:ns-regexp "e2e.*",
   :output-to "out/e2e.js",
   :target :node-test},
  :test
  {:ns-regexp "app.*-spec$",
   :output-to "out/test.js",
   :target :node-test}},
 :dependencies [[reagent "0.8.1"]],
 :nrepl {:port 3333},
 :dev-http {3000 "public"}
 :source-paths ["src" "e2e"]}

also kinda not very clojure-ish with all the , and incorrect indentation 😛

top-level :dev-http instead of nested :devtools {:http...} is the newer format

regarding the config changes, yes let me add it

regarding the indentation, that is the default for zprint-clj https://github.com/kkinnear/zprint

I couldn't find another formatter that had a good npm distro

is there a canonical config for these formatters that matches what people use?

btw you can always run clojure code using https://shadow-cljs.github.io/docs/UsersGuide.html#clj-run

so you can run any clojure code you want (eg. zprint)

if a shadow-cljs server instance is running you won't even pay for startup time

I had no idea, that sounds pretty handy!

there are some clj -m zprint.main <deps.edn examples in the zprint readme

shadow-cljs run zprint.main would be the same

although I guess the <deps.edn would be a problem

but that could be fixed

so right now shadow-cljs run lets me run a given function in a clj namespace I have

but to use it directly as shown in https://github.com/kkinnear/zprint#clojure-cli might not work currently, is that right?

I trying just doing run zprint.main without any modification to shadow-cljs.edn besides adding the dependency, and it seems to print the formatted output

kamik@RED-X1C6 MINGW64 /d/work/create-cljs-app/template (master)
$ shadow-cljs run zprint.main <shadow-cljs.edn
shadow-cljs - config: D:\work\create-cljs-app\template\shadow-cljs.edn  cli version: 2.8.64  node: v10.16.0
{:builds
   {:app {:asset-path "/js",
          :modules {:main {:init-fn app.core/main}},
          :output-dir "public/js",
          :target :browser},
    :e2e {:ns-regexp "e2e.*", :output-to "out/e2e.js", :target :node-test},
    :test
      {:ns-regexp "app.*-spec$", :output-to "out/test.js", :target :node-test}},
 :dependencies [[reagent "0.8.1"] [zprint "0.5.1"]],
 :dev-http {3000 "public"},
 :nrepl {:port 3333},
 :source-paths ["src" "e2e"]}

it's very slow though

(btw the dev:http change is now published)

@filipematossilva have you read about server mode and how it functions? (regarding the "slow" comment)

I'm familiar with it insofar as using shadow-cljs start in the background to speed up further commands.

I didn't try it with this command though

let me try

or shadow-cljs server yes

but yeah the point is to get rid of the startup delay. so in general the server should always be running when you are developing

making startup time a non issue

Yes you are right, it is much faster now. Especially the second and subsequent runs.

I need to think a bit about this usage pattern... It feels pretty powerful insofar as it provides direct access to all these clj tools. But I'd need to see if I could get a 1-to-1 correspondence as far as functionality goes. The npm zprint wrapper allows formatting all files and writing them back to the same place, and that sounds like it'd need a custom function to do. But a custom function is also a good example of how a user can customize it.

I just wanted to mention it. just in case there is something available in clj but not on npm

the zprint defaults seem to suck for code but there was a big discussions about formatters a while ago

guess CLJ is too flexible syntax wise to find an "agreeable" standard

I remember seeing a discussion on clojureverse I think

it looked stalled

yeah its a bit easier in JS and especially JSON configs since they are least maintain insertion order

I had an unrelated question for you

are the npm wrappers today the same as described in https://code.thheller.com/blog/shadow-cljs/2018/06/15/why-not-webpack.html?

hmm?

so something like

// node_modules/lib/foo.js
shadow$provide["module$node_modules$lib$foo"] = function(module, global, process, exports, require) {
    var foo = 1;
    exports.hello = function() { return foo; };
}

// node_modules/lib/bar.js
shadow$provide["module$node_modules$lib$bar"] = function(module, global, process, exports, require) {
    var foo = require("module$node_modules$lib$foo");
    foo.hello();
}

yes, mostly the same

in regards to module usage, I assume these wrappers can never avail from code motion, right?

hmm?

GCC has a nice feature called code motion, where it can "split" the content of a given JS file if part of it is used in one module but not the others

yes I know

so for instance

// node_modules/lib/foo.js
shadow$provide["module$node_modules$lib$foo"] = function(module, global, process, exports, require) {
    exports.hello = function() { return 1; };
    exports.world = function() { return 2; };
}

no, the npm content is never "split"

ok

but it is moved into the proper modules when it can

ie. if module B uses something the others don't they will be in B

is there a way to exclude certain modules from the wrapper?

don't understand the question

you can't not wrap the code

I was trying to imagine how would code motion be used for npm libraries, assuming the library was in a GCC friendly format

if everything was strict ES6 then we would just enable GCC for everything

with all its glory

until then we run in "compatibility" mode

you can't select one library to go with the full mode while others don't

thats leads to the externs mess we already have to deal with

I experimented with all of this a very long time. basically it never works. there are barely and "strict" ES6+ modules on npm (that also only depends on other strict ES6+ code)

I don't mean for the general case though

do you have an actual example?

yeah, now that I think about it I do

so for instance I could try to import @angular/core. It is published in ES2015 among other formats. It is also annotated for GCC via tsickle (a TS wrapper to annotate TS output for GCC).

nevermind that there's no way in hell angular will properly work in cljs

but that library is full ES2015

then separately I also had momentjs, which is in crazy format

I import them both from my app

but neither has an interdependency

I would like to tell shadow-cljs "try to import angular/core in full GCC mode, but keep momentjs in compat"

that doesn't work

you can "fake" it by copying node_modules/@angular/core/esm5/* to src/main/angular or so

then import it via (:require ["/angular/index.js" :as angular])

classpath JS will follow be processed by the GCC and advanced

but that means you can't use any libraries that want to use angular

yes, that limitation I did expect

and you'll need a whole bunch of externs for all the code it consumes

since there is no externs inference for JS code

so back to the old externs way

well you can always turn of the renaming

yeah but that kinda defeats the purpose of trying to optimize these further

var PlatformRef = /** @class */ (function () {

why does it not use class? 😛

because that's es5 code in ESM format

there's also es2015 code in another folder

the esm2015 one

ah right

when you said this doesn't work, do you mean that it's not a feature that is available or that there's no reasonable way such a feature could be developed?

well, not going to call it impossible. I certainly can't implement it in a reasonable way.

if you had perfect externs then it is possible yes

BUT that only applies when mixing code

You mentioned there is no extern inference for js code. I imagined that the wrapped npm libs got some sort of extern inference.

is it that they don't actually get any extern inference?

"they" don't need inference, if by they you mean the npm libs

the :advanced parts need externs for the code not part of the advanced compilation (ie. npm code)

if you process some of the npm libs as part of advanced

then they also don't need externs. but you still need externs for others

AND if you have npm libs accessing the advanced code you'll need externs for those parts

so the closure compiler doesn't rename those either

its a mess

the current state of npm just doesn't allow anything more than what is currently done

no I don't think there's ever going to be a solution for the re-entrant case

where by re-entrant I mean going from "compat" back to "full gcc" code

which is probably me misusing the word

with enough info about what is used it is possible

but getting that info might not be possible 😛

yeah but no one is going to bother getting all that

which is why shadow-cljs filled up this need so well

full npm compat is sweet

bitter sweet maybe 😛 I kinda regret exposing us to that world sometimes

tbh that's what drew me to shadow-cljs instead of any other setup

yeah, sometimes you just want access to all that stuff

I'm from a JS background and it's a hard sell to have a huge overhead to all the stuff you already know

yeah, same for me with Clojure .. already had 10+ years of experience with Java so having access to that was sort of the killer argument

same for the JS parts ... just felt like missing out

but the npm/JS ecosystem is a lot more messy 😛

we'll see how this evolves. I forsee a lot of breakage if they ever adopt more ESM code

atm some libraries publish it under non-standard package.json entry points, like module, esm, or es0215

I know

the problem is interop

which means that the resolver needs to know to look these up... and it gets messy

so 100% commonjs is fine

100% esm would be great

mixing ... is just painful 😛

(shadow-cljs knows about those btw)

just doesn't use them by default

I sorta expected the wrapper to handle the interop well enough, it's similar to the webpack wrapper

the problem is the non-strict interop that babel/webpack had

webpack v5 will remove it so we'll see how that goes

import { aThing } from "some-common-js" can't work in strict mode

only import foo from "some-common-js" + foo.aThing

but lots of code relies on the babel interop mode

which means all code moving to the strict variant will break the packages not yet moved

fun times ...

for all the good webpack did, it also messed stuff up a bit by letting people write packages that only work on webpack pipelines

indeed. although babel takes some of the blame too

like all the require('something.css') and defaulting resolving .wasm extensions by default

huge burden on consumers

I understand the need for all of that. just wish they hadn't built it on top of require

heck even require.asset("something.css") or so would have been better

yeah something that just wasn't overloading the default stuff that much

webpack ended up doing something similar to what I was asking about, where they pulled out esm code into one giant scope, but bail on a number of circumstances

So imagine this case then: - I have cljs code - I have some es2015 libs that sometimes call each other, but never call non-es2015 libs - I have non-es0215 libs that call each other, but not es2015 libs would this case work by having the es2015 libs in advanced mode?

I know this isn't something that would just generally work, I'm mostly thinking of a case where I am very familiar with my dependencies

there is no mode for it but that would be possible yes

but even some libraries that boast to be es2015 aren't

eg. lodash-es

and if I switched it a bit to the point where the es2015 libs called non-es2015 libs, but not the other way around?

if you have externs for the non-es2015 parts

couldn't those get figured out by shadowcljs the same way it does when cljs calls a compat-mode npm lib?

es2015 then effectively becomes the same as CLJS code

kind of but not really

JS is just handed to the GCC to do its thing. it would need to do the same kind of processing that is done for CLJS

yeah that makes sense

currently when processing the npm JS code

it looks for module.exports (and variants thereof) and collects the exported names as externs

but in practice that is actually not a good idea and doesn't work when the JS code isn't processed by shadow-cljs (eg. react-native, node-script, node-library, ..)

so the code is mostly relying on externs inference in CLJS

I didn't actually understand that... I thought "extern inference" meant "looking through js code to try and figure out externs" but now I understand it means "looking through CLJS to figure out npm lib externs"

externs inference is done in CLJS code

it looks at the code and tries to figure out if something you accessed will be coming from npm

when it can't figure that out it will warn

in many cases it is easy to figure out (:require ["some-npm-lib" :refer (foo bar)])

easy to add the refers to externs

(defn wrap-baz [x]
  (.baz x))

not easy to know if baz is safe to rename or requires externs

that's why it would warn https://shadow-cljs.github.io/docs/UsersGuide.html#infer-externs

the JS processing and extra externs gathering is only supplemental

and not active when building non-browser targets basically

if I wanted to take a stab at doing that three tier approach, would you be open to accept contributions?

if you supply an example project that can be tested against 😛

adding that is like 10 lines of code or so

yes I'd provide involved test cases

it seems like an interesting entry point in contributing to shadow-cljs

I'd first try to find an actual practical example

like something that people would actually use. not some hypothetical thing that doesn't actually exist in the wild

you can fake it with zero code changes and just a few symlinks today (or file copies)

yeah a real example would also give a good indicator of the benefit

I imagine the benefit is that the es2015 libs get a dramatic cut in size

but maybe that ends up not being true

the potential is there for sure

oh I think I have an interesting example

but I can think of a billion things that can still to wrong

using ES6+ code still has a few issues in the way it is currently implemented

pretty much only works for :browser builds

yes I don't have any illusions that code on npm will always find way to break

npm was never really built with forwards compat in mind

so imagine I have a bunch of non-es-2015 libs

and then I bundle them via rollup

kinda of how the cljs recommended they were bundled via webpack instead

but this time the output is actually a giant es2015 module

(that probably is broken because of some weird lib)

if it's not broken though, GCC takes over and might do some good because it got a lot of es2015 to work with

and since it's a giant es2015 module, gcc can do codemotion

I thought rollup also only did it for all esm?

or does it support the mixed thing?

no, you can config for commonjs compat

https://github.com/rollup/rollup-plugin-node-resolve

actually let me check that this produces es2015...

sorry this is the right one https://github.com/rollup/rollup-plugin-commonjs

compat isn't as good as webpack, but it exists

never used rollup on anything real

this of course is subject to all the pitfalls in random code we discussed

I added a new :js-provider :external not too long ago

which lets you use webpack to create to create an "external" package

something like the official CLJS webpack guide recommends, just automated

in angular cli we're actually experimenting with doing rollup inside webpack... because webpack has a lot of trouble making code in lazy chunks tree shakeable

eek lazy chunks

so we process all the es2015 code with rollup, then pass on that output (including lazy modules) to webpack

I'm scared of libraries actually using import()

and top-level await

the JS world sure has crazy ideas ...

with webpack the problem is this: - if you have a single chunk, maybe webpack will do module concatenation and everything is in a single scope, then Terser can do tree-shaking - if you have multiple chunks module then concatenation breaks down, and the webpack module loader indirection stops Terser tree-shaking

seems like something they could fix with all the funding they have 😛

rollup, similar to GCC, performs tree-shaking with bundling information

so those are the only two JS tools today that can actually handle tree shaking across chunks/modules

they probably could but are trying to do build caching as their main focus atm I think

which is why I think shadow-cljs is in a really good place to take advantage of es2015 libs

contrary to most of the js ecosystem, the clj ecosystem is not averse to non-js toolchains like java

the pure js GCC implementation is pretty much unusable, so realistically GCC needs java

anyway I'll try to setup a few realistic test cases, then some mock cases that explore it a bit more, then try a implementation

happy to guide you through it. it will mostly be about changing the resource type from :shadow-js to just :js 😛

https://github.com/thheller/shadow-cljs/blob/989590bb373e56b1d0ccf3a4029e2d3ec294ffe3/src/main/shadow/build/resolve.clj#L187

so the logic there would need to change

cool, thank you for the pointer

I'll look there after the test cases