Fork me on GitHub
#shadow-cljs
<
2018-07-06
>
lsnape10:07:22

Hi, I'm tracking down a npm dependency issue in my production app that I'm having trouble reproducing in a minimal demo. Pulling in the raven-js npm dependency in my prod app gives me a syntax error when requiring the module. I can trace it to an innocuous-looking function in a utilities js file. The only difference I can see is in the 'Sources' tab in the Chrome inspector: a file under node_modules/raven-js/src/utils.js exists in my test project, but in prod it's missing (all neighbouring source files are there). This might be a red herring but I can't see any other differences worthy of note.

lsnape10:07:48

The file module\$node_modules\$raven_js\$src\$utils.js is served up in both projects, along with the .map file.

lsnape10:07:23

Any guidance on things I could try greatly appreciated, as I feel a bit like I'm shooting in the dark!

thheller10:07:35

@lsnape what is the error you get?

thheller10:07:39

and you do not get this problem during development or is the code just not used in dev?

thheller10:07:18

wait nvm. thats a dev error

thheller10:07:53

hmm it compiles and loads fine for me

thheller10:07:21

do you use lein?

lsnape10:07:08

Yea it's dev. I'm actually getting boot to serve the assets out of the target dir

thheller10:07:22

what do you use to run shadow-cljs? just shadow-cljs or lein or boot?

lsnape10:07:59

It's a bit cobbled together atm. I run shadow-cljs watch app from one terminal window, and boot to serve assets out of target/public.

thheller10:07:33

shadow-cljs standalone though? no lein, boot or tools.deps?

lsnape10:07:38

I'm chaining watch tasks so boot copies the compiled JS into the dir we serve.

thheller10:07:38

trying to rule out dependency issues

lsnape10:07:42

Yeah standalone shadow.

thheller10:07:20

and boot copies everything correctly? not that it just stopped in the middle of the file or something?

thheller10:07:40

check if the <output-dir>/cljs-runtime/module\$node_modules\$raven_js\$src\$utils.js file is corrupted or so

thheller10:07:13

or rather check if the file generated by shadow-cljs is identical to the file served by boot

lsnape10:07:01

Doing that now. I'll diff it against the demo app, which is working.

lsnape10:07:22

There is a difference:

1,3c1,3
< shadow$provide.module$node_modules$raven_js$src$utils=function(global,process,require,module,exports){function isUndefined(what){return void 0===what}function isPlainObject(what){return"[object Object]"===Object.prototype.toString.call(what)}function isString(what){return"[object String]"===Object.prototype.toString.call(what)}function isArray(what){return"[object Array]"===Object.prototype.toString.call(what)}function supportsFetch(){if(!("fetch"in _window))return!1;try{return new Headers,new Request(""),
< new Response,!0}catch(e){return!1}}function each(obj,callback){var i,j;if(isUndefined(obj.length))for(i in obj)hasKey(obj,i)&&callback.call(null,i,obj[i]);else if(j=obj.length)for(i=0;i<j;i++)callback.call(null,i,obj[i])}function truncate(str,max){if("number"!==typeof max)throw Error("2nd argument to `truncate` function should be a number");return"string"!==typeof str||0===max?str:str.length<=max?str:str.substr(0,max)+"…"}function hasKey(object,key){return Object.prototype.hasOwnProperty.call(object,
< key)}function joinRegExp(patterns){for(var sources=[],i=0,len=patterns.length,pattern;i<len;i++)pattern=patterns[i],isString(pattern)?sources.push(pattern.replace(/([.*+?^=!:}()|\[\]\/\\])/g,"\\$1")):pattern&&pattern.source&&sources.push(pattern.source);return new RegExp(sources.join("|"),"i")function htmlElementAsString(elem){var out=[],className,attr;if(!elem||!elem.tagName)return"";out.push(elem.tagName.toLowerCase());elem.id&&out.push("#"+elem.id);if((className=elem.className)&&isString(className)){var classes=
---
> shadow$provide.module$node_modules$raven_js$src$utils=function(global,process,require,module,exports,shadow$shims){function isUndefined(what){return void 0===what}function isPlainObject(what){return"[object Object]"===Object.prototype.toString.call(what)}function isString(what){return"[object String]"===Object.prototype.toString.call(what)}function isArray(what){return"[object Array]"===Object.prototype.toString.call(what)}function supportsFetch(){if(!("fetch"in _window))return!1;try{return new Headers,
> new Request(""),new Response,!0}catch(e){return!1}}function each(obj,callback){var i,j;if(isUndefined(obj.length))for(i in obj)hasKey(obj,i)&&callback.call(null,i,obj[i]);else if(j=obj.length)for(i=0;i<j;i++)callback.call(null,i,obj[i])}function truncate(str,max){if("number"!==typeof max)throw Error("2nd argument to `truncate` function should be a number");return"string"!==typeof str||0===max?str:str.length<=max?str:str.substr(0,max)+"…"}function hasKey(object,key){return Object.prototype.hasOwnProperty.call(object,
> key)}function joinRegExp(patterns){for(var sources=[],i=0,len=patterns.length,pattern;i<len;i++)pattern=patterns[i],isString(pattern)?sources.push(pattern.replace(/([.*+?^=!:${}()|\[\]\/\\])/g,"\\$1")):pattern&&pattern.source&&sources.push(pattern.source);return new RegExp(sources.join("|"),"i")}function htmlElementAsString(elem){var out=[],className,attr;if(!elem||!elem.tagName)return"";out.push(elem.tagName.toLowerCase());elem.id&&out.push("#"+elem.id);if((className=elem.className)&&isString(className)){var classes=

lsnape10:07:52

That's diffing broken working

thheller10:07:25

can't see anything 😛

thheller10:07:54

first version is older though

lsnape10:07:40

First is using shadow 2.4.8.

thheller10:07:43

might just be a closure compiler bug. try upgrading

lsnape10:07:56

OK, sorry should have checked that.

thheller10:07:01

hmm no compiler changes since then though

thheller10:07:08

otherwise try deleting the .shadow-cljs/builds directory just to rule out a cache issue

thheller10:07:20

unlikely but never hurts to try 😉

lsnape10:07:11

OK will do

thheller10:07:16

can't really tell what else is different in the diff besides ,shadow$shims which was added recently

thheller10:07:06

other than that is appears similar. be sure to check the actual output served by boot though might be something happening in transit

lsnape10:07:05

yeah, eventually I'd like to isolate boot completely.

lsnape11:07:16

OK, still broken but the diff has changed:

3c3
< key)}function joinRegExp(patterns){for(var sources=[],i=0,len=patterns.length,pattern;i<len;i++)pattern=patterns[i],isString(pattern)?sources.push(pattern.replace(/([.*+?^=!:}()|\[\]\/\\])/g,"\\$1")):pattern&&pattern.source&&sources.push(pattern.source);return new RegExp(sources.join("|"),"i")function htmlElementAsString(elem){var out=[],className,attr;if(!elem||!elem.tagName)return"";out.push(elem.tagName.toLowerCase());elem.id&&out.push("#"+elem.id);if((className=elem.className)&&isString(className)){var classes=
---
> key)}function joinRegExp(patterns){for(var sources=[],i=0,len=patterns.length,pattern;i<len;i++)pattern=patterns[i],isString(pattern)?sources.push(pattern.replace(/([.*+?^=!:${}()|\[\]\/\\])/g,"\\$1")):pattern&&pattern.source&&sources.push(pattern.source);return new RegExp(sources.join("|"),"i")}function htmlElementAsString(elem){var out=[],className,attr;if(!elem||!elem.tagName)return"";out.push(elem.tagName.toLowerCase());elem.id&&out.push("#"+elem.id);if((className=elem.className)&&isString(className)){var classes=

lsnape11:07:43

replace(/([.*+?^=!:}()|\[\]\/\\])/g,"\\$1")) replace(/([.*+?^=!:${}()|\[\]\/\\])/g,"\\$1"))

lsnape11:07:24

Hmm this is actually looks like a raven-js lib version discrepancy.

lsnape11:07:50

Nope they are both the same 😞

thheller12:07:43

@lsnape do you have anything in your config which might affect this? custom :js-options?

thheller12:07:10

custom closure-compiler dependency?

thheller12:07:35

I don't actually think that there is anything you do via the config to affect this but gotta rule it out still

lsnape12:07:21

@thheller I think I know what it is. We've got an asset fingerprinting boot task that scans all source files for template params ${param} and replaces them with a hashed version, or in the case of development skips over them. It looks like the regex is being incorrect replaced by the task. 😞

lsnape12:07:26

It's an easy fix but very annoying. Thanks for working it through with me though.

thheller12:07:40

ah I guess you mean images and stuff

lsnape12:07:03

Yeah we need it for all assets. Do you think that's out of scope in shadow?

thheller12:07:26

no. I have plans for that. just not the time to implement it currently.

lsnape12:07:34

If shadow supported hashing all assets then we could ditch boot entirely!

lsnape12:07:15

Here's the boot task I wrote to take care of it: https://github.com/ELiTLtd/boot-asset-fingerprint

lsnape12:07:55

Feel free to steal anything out of there. It's something I'd be interesting in doing with a bit of guidance? Up to you!

thheller12:07:20

so its just about change the references in the HTML?

thheller12:07:33

I thought you were hashing images and stuff?

lsnape12:07:01

Yeah, it hashes the file names and the references in other files, such as html and css.

lsnape12:07:22

So ${img/foo.png} in bar.html becomes img/foo-af28b.png in the html file, plus a new file is written to the file path.

thheller12:07:50

yeah. the example only had .js so I thought it was just about .js

lsnape12:07:12

I really like the way you do it by generating a manifest file for module hashing. It's more work to plumb it in the project, but it's very flexible.

thheller12:07:06

yeah in my work projects I just watch the manifest for changes in production

thheller12:07:15

so I can deploy new JS code without restarting the server

thheller12:07:00

but its a clojure server generating the HTML anyways so no static html to rewrite

lsnape12:07:28

That's what we're doing in our latest project: html page frame is all hiccup rendered on request (memoized).

lsnape12:07:55

But older projects serve up html templates rendered after cljs compilation time.

lilactown15:07:38

I'm still struggling with shadow-cljs in CI. running a release build takes up to 10 mins

thheller15:07:16

did you enable caching?

lilactown15:07:58

yep. persisting the .shadow-cljs/builds/ folder shaved off about a minute or so

thheller15:07:22

did you run with --verbose? cache should be way more than that

thheller15:07:45

verbose should tell you exactly where the time is spent

lilactown15:07:05

OK, I'll dig into it more

lilactown15:07:32

for reference, about half of that is just starting shadow-cljs 😬 compiling takes about 5 mins

thheller15:07:46

5min to start? are you running on a toaster?

lilactown15:07:03

very well could be for all I know

lilactown17:07:20

I'm trying to compile a target :karma build:

-> build target: :karma stage: :configure
NullPointerException:
	shadow.build.targets.karma/configure (karma.clj:21)
	shadow.build.targets.karma/configure (karma.clj:16)
	shadow.build.targets.karma/process (karma.clj:112)
	shadow.build.targets.karma/process (karma.clj:108)
	clojure.lang.Var.invoke (Var.java:381)
	shadow.build/process-stage/fn--15189 (build.clj:120)
	shadow.build/process-stage (build.clj:117)
	shadow.build/process-stage (build.clj:109)

lilactown17:07:35

I'm a little unsure if my regexp is working. would that trigger this error?

thheller17:07:18

missing :output-to, need to add specs for the config still

lilactown17:07:19

ahh, I had output-dir

lilactown18:07:22

I'm getting an error when trying to run karma tests:

d on socket wpx59euvS82blWCjAAAA with id 10464947
HeadlessChrome 0.0.0 (Mac OS X 10.13.5) ERROR
  {
    "message": "Uncaught ReferenceError: shadow is not defined\nat : shadow is not defined\n    at eval (eval at <anonymous> (/Users/r627543/Code/fido/packages/web/app/node_modules/karma-cljs-test/adapter.js:6:7), <anonymous>:1:1)\n    at ContextKarma.start (/Users/r627543/Code/fido/packages/web/app/node_modules/karma-cljs-test/adapter.js:6:7)\n    at ContextKarma.loaded ()\n    at ",
    "str": "Uncaught ReferenceError: shadow is not defined\nat : shadow is not defined\n    at eval (eval at <anonymous> (/Users/r627543/Code/fido/packages/web/app/node_modules/karma-cljs-test/adapter.js:6:7), <anonymous>:1:1)\n    at ContextKarma.start (/Users/r627543/Code/fido/packages/web/app/node_modules/karma-cljs-test/adapter.js:6:7)\n    at ContextKarma.loaded ()\n    at "
  }

thheller20:07:00

I'm guessing that karma isn't loading the correct js?

thheller20:07:26

looks like its trying to call the shadow.test.karma.init fn maybe and doesnt find it?

lilactown20:07:52

this is what my config looks like:

{:target :karma
           :ns-regexp ".devcards$"
           :output-to "target/karma/tests.js"}

lilactown20:07:00

and I copied the karma.conf.js from the user-guide

thheller20:07:21

and you changed the path there?

lilactown20:07:39

module.exports = function (config) {
  config.set({
    browsers: ['ChromeHeadless'],
    // The directory where the output file lives
    basePath: 'target/karma/',
    // The file itself
    files: ['target/karma/tests.js'],
    frameworks: ['cljs-test'],
    plugins: ['karma-cljs-test', 'karma-chrome-launcher'],
    colors: true,
    logLevel: config.LOG_INFO,
    client: {
      args: ["shadow.test.karma.init"],
      singleRun: true
    }
  })
};

lilactown20:07:00

nvm I figured it out

lilactown20:07:11

files is relational to basePath

thheller20:07:12

basePath + files doesn't work I think

lilactown20:07:38

😅 PEBKAC. thanks for holding my hand