Fork me on GitHub
#shadow-cljs
<
2018-02-23
>
justinlee15:02:35

is there a way to get stats on what is contributing to the release build sizes? my build is giant and I’m trying to figure out why

thheller16:02:43

currently only from the REPL

thheller16:02:55

(shadow.cljs.devtools.api/release-snapshot :the-build {})

hlolli16:02:10

My release build size is currently huge, I see quickly scanning over the minified code that for example all material-ui icons are bundled module$node_modules$material_ui_icons$Gamepad But it's still acceptable size with gzip

thheller16:02:35

then navigate to and click through the release snapshot sutff

hlolli16:02:21

haha

831.76 KB		node_modules/material-ui-icons
203.19 KB		node_modules/material-ui
ok I'm defenitely going to just download each svg that I use, damn!

hlolli16:02:33

except there's a way to cherry-pick files from a module?

thheller16:02:16

why are icons even js?

hlolli16:02:01

they create react svg elements

var _ref = _react2.default.createElement('path', { d: 'M22 5.72l-4.6-3.86-1.29 1.53 4.6 3.86L22 5.72zM7.88 3.39L6.6 1.86 2 5.71l1.29 1.53 4.59-3.85zM12.5 8H11v6l4.75 2.85.75-1.23-4-2.37V8zM12 4c-4.97 0-9 4.03-9 9s4.02 9 9 9c4.97 0 9-4.03 9-9s-4.03-9-9-9zm0 16c-3.87 0-7-3.13-7-7s3.13-7 7-7 7 3.13 7 7-3.13 7-7 7z' });

tianshu16:02:11

sometimes wired bugs occur can be fixed by lein clean when using with lein. maybe it's better to always clean before start?

thheller16:02:16

@doglooksgood lein clean should not be required at all. if something doesn't work but works after lein clean I want details.

thheller16:02:37

@hlolli looking at the package you can just import the icons you actually want

hlolli16:02:35

yup, I do that for example ["material-ui-icons/Hotel" :as Hotel] this means I have required ["material-ui-icons"] somewhere?

thheller16:02:54

either you or a library you are using yes.

tianshu16:02:10

I just setup a project, but I forgot to npm install react(I use reagent), then I kill the REPL process, add npm dependency with yarn. when start shadow-cljs again, i met OutOfMemory.

tianshu16:02:33

after lein clean, everything works fine.

thheller16:02:57

you get OOM when starting the shadow-cljs process?

hlolli16:02:04

yup it was me 🤐

tianshu16:02:58

shadow-cljs - config: /Users/shitianshu/sideline/loadstone/shadow-cljs.edn version: 2.1.24
shadow-cljs - socket connect failed, server process dead?
also I got this when type shadow-cljs in terminal. @thheller I think this is not a urgent issue, you can just ignore. when i have time, i can create a repo that reproduce the problem.

tianshu16:02:04

since i killed that repl in emacs, i don't have the error stack at this moment. no enough information to locate the problem.

thheller16:02:25

hmm if you forcefully kill the shadow-cljs process it won't clean up after itself. thats why you get socket connect failed, server process dead?

thheller16:02:34

try running shadow-cljs stop instead

hlolli16:02:48

wow, the bundle went from 1,8mb down to 306,5kb, I was not paying attention to the bundle size

thheller16:02:18

@hlolli gotta be careful with those js imports 😉

hlolli16:02:46

I thought google.closure could read minds too 😄

tianshu16:02:28

can I add npm dependency or cljs dependency without stop shadow-cljs process?

thheller16:02:57

npm yes, doesn't require a restart. cljs no.

thheller16:02:17

one issue currently is that npm install does not trigger a recompile. if you have a watch running that failed because of a missing npm dependency that build will stay stuck

justinlee16:02:07

so i if i do shadow-cljs server then shadow-cljs release :app and then navigate to http://localhost:9630/ I get “no release snapshots found”

thheller16:02:47

@lee.justin.m not release. like I said you need to run this from the REPL. shadow-cljs clj-repl and then (shadow.cljs.devtools.api/release-snapshot :the-build {})

tianshu16:02:15

okay, thanks!

thheller16:02:29

@doglooksgood you can use the REPL to restart the build but its all a bit messy

thheller16:02:29

shadow-cljs clj-repl (shadow/stop-worker :build) (shadow/watch :build)

tianshu16:02:35

ExceptionInfo: module without entry or suffix: react what's this error stands for?

thheller16:02:13

uhm what is in your node_modules/react/package.json?

thheller16:02:36

are you using npm or cnpm?

thheller16:02:18

cnpm did some weird symlinks that causes issues a while ago

thheller16:02:59

you can try shadow-cljs clj-repl (shadow.cljs.devtools.server/reload!) if anything weird is happening

thheller16:02:48

stops everything and reloads the server

thheller16:02:02

all of this will get much easier to use once I work on the UI

thheller16:02:12

doing all of this from the REPL/CLI is horrible

tianshu16:02:28

very strange, I found react, react-dom and shadow-cljs in node_modules become empty...

tianshu16:02:12

it becomes node_modules/react/node_modules

tianshu16:02:16

like this.

tianshu16:02:37

I re-run the yarn install

tianshu16:02:43

let me investigate by myself

justinlee16:02:14

is there a way to dump the dependency graph so I can see where some dependencies are coming from?

justinlee16:02:27

i think you told me once before. this time I will add it to my tips file. 🙂

thheller16:02:35

will also be part of the UI. for CLJS you can do (shadow.cljs.devtools.api/find-resources-using-ns 'foo.bar)

thheller16:02:58

doesn't yet work for JS deps though

justinlee17:02:31

hm. well something is going terribly awry when included pdf.js. it’s bundling the webworker

justinlee17:02:35

it shouldn’t do that

justinlee17:02:28

I’m including them very precisely like so:

["pdfjs-dist/lib/pdf.js" :as pdfjs]
            ["pdfjs-dist/web/pdf_viewer.js"]))

justinlee17:02:21

including the pre-built pdf.js actually reduces the total build, but it still includes the worker in the bundle somehow

thheller17:02:54

fakeWorkerFilesLoader = useRequireEnsure ? function (callback) {
    require.ensure([], function () {
      var worker;
      worker = require('./pdf.worker.js');
      callback(worker.WorkerMessageHandler);
    });
  } : dynamicLoaderSupported ? function (callback) {
    requirejs(['pdfjs-dist/build/pdf.worker'], function (worker) {
      callback(worker.WorkerMessageHandler);
    });

tianshu17:02:04

is it possible to import css in cljs?

thheller17:02:37

@lee.justin.m it appears that it has some kind of dynamic loading mechanism which the compiler cannot identify as dynamic

justinlee17:02:07

okay let me take a look at the docs. there is probably some stuff you have to do to make behave. this is a super complicated package

justinlee17:02:38

i astounded that it actually functioned by just including it so kudos to shadow

thheller17:02:53

the pdf.js has this stuff

thheller17:02:54

{
  var isNodeJS = require('./shared/is_node.js');
  if (isNodeJS()) {
    var PDFNodeStream = require('./display/node_stream.js').PDFNodeStream;
    pdfjsDisplayAPI.setPDFNetworkStreamFactory(function (params) {
      return new PDFNodeStream(params);
    });
  } else if (typeof Response !== 'undefined' && 'body' in Response.prototype && typeof ReadableStream !== 'undefined') {
    var PDFFetchStream = require('./display/fetch_stream.js').PDFFetchStream;
    pdfjsDisplayAPI.setPDFNetworkStreamFactory(function (params) {
      return new PDFFetchStream(params);
    });
  } else {
    var PDFNetworkStream = require('./display/network.js').PDFNetworkStream;
    pdfjsDisplayAPI.setPDFNetworkStreamFactory(function (params) {
      return new PDFNetworkStream(params);
    });
  }
}

thheller17:02:12

dynamic requires just won't work properly

justinlee17:02:18

well you know i’m actually okay with it bundling the worker but i need to figure out how to start a worker using the bundled source. it is possible apparently

justinlee17:02:26

i just don’t want to fetch it twice

justinlee17:02:55

the default is not to use a worker at all if it is bundled and just doesn’t work because your UI is unusable

justinlee17:02:07

so somehow they got the main bundle and the worker bundle to build separately using webpack

justinlee17:02:15

they have a webpack example.

justinlee17:02:31

~/test/pdf.js/examples/webpack$ npm run build

> [email protected] build /Users/justinlee/test/pdf.js/examples/webpack
> webpack

Hash: 9633d58ec3b79d0aaf96
Version: webpack 1.12.15
Time: 18903ms
               Asset    Size  Chunks             Chunk Names
      main.bundle.js  397 kB       0  [emitted]  main
       1.1.bundle.js  704 kB       1  [emitted]  
pdf.worker.bundle.js  706 kB    2, 1  [emitted]  pdf.worker
   [0] ./main.js 1.17 kB {0} [built]
    + 50 hidden modules
~/test/pdf.js/examples/webpack$ ls ../../build/webpack/
1.1.bundle.js		main.bundle.js		pdf.worker.bundle.js
~/test/pdf.js/examples/webpack$ ll ../../build/webpack/
total 1784
-rw-r--r--  1 justinlee  staff  708843 Feb 23 09:21 1.1.bundle.js
-rw-r--r--  1 justinlee  staff  397399 Feb 23 09:21 main.bundle.js
-rw-r--r--  1 justinlee  staff  710818 Feb 23 09:21 pdf.worker.bundle.js

thheller17:02:36

yeah they are using something webpack specific

thheller17:02:10

or rather they are using the webpack code splitting stuff

thheller17:02:52

I can detect calls to require.ensure and ignore them

justinlee17:02:00

all they do is list two entry points. one for main.js, which does a var pdfjsLib = require('pdfjs-dist'); and one for the worker

thheller17:02:11

unfortunately not

thheller17:02:38

require.ensure([], function () {
      var worker;
      worker = require('../pdf.worker.js');
      callback(worker.WorkerMessageHandler);
    });

thheller17:02:57

webpack will use require.ensure to create a split point

thheller17:02:04

so you end up with two files

justinlee17:02:09

sneaky sneak

thheller17:02:13

closure and shadow-cljs don't work that way

thheller17:02:40

but I can skip require.ensure calls so the worker won't be bundled

justinlee17:02:49

yea in the browserify example they have to use a gulp task to ignore that line

gulp.task('build-bundle', function() {
  return browserify('main.js', {output: TMP_FILE_PREFIX + 'main.tmp'})
    .ignore(require.resolve('pdfjs-dist/build/pdf.worker')) // Reducing size
    .bundle()
    .pipe(source(TMP_FILE_PREFIX + 'main.tmp'))
    .pipe(streamify(uglify()))
    .pipe(rename('main.bundle.js'))
    .pipe(gulp.dest(OUTPUT_PATH));
});

thheller17:02:28

I could add an ignore option as well but seems fine to just ignore require.ensure

thheller17:02:02

@lee.justin.m 2.1.25 should not include the worker anymore

justinlee17:02:18

(side note: vscode is pretty impressive. i went to update the version in package.json and the mouse-over already told me that 2.1.25 is the most recent version)

thheller17:02:50

yeah their JS intergration is pretty good

justinlee17:02:25

i’m thinking about trying them instead of atom to see if the tooling is better

thheller17:02:43

JS tooling is. everything else isn't.

thheller17:02:49

that was my experience at least

justinlee17:02:33

okay it is definitely eliding the worker. now i have to figure out what else broke. (I don’t think it is related)

thheller17:02:00

uh that broke something?

justinlee17:02:57

i don’t think so.

justinlee17:02:03

i’m checking to see what’s wrong

justinlee18:02:55

okay so unfortunately that breaks pdf.js because what it does is goes and tried to load the worker code into the main thread https://github.com/mozilla/pdf.js/blob/99060e2485cacb8513e7bc50d88b70101069b1c9/src/display/api.js#L1478

justinlee18:02:58

hang on possible false alarm because downgrading to 2.1.24 does the same thing.

thheller18:02:55

what is the actual problem?

thheller18:02:29

you might be better off just using their dist from a CDN and using it via a global

justinlee18:02:41

seekeasy.page.worker.js:1 Uncaught SyntaxError: Unexpected token <
pdf.js:3853 Uncaught TypeError: Cannot read property 'WorkerMessageHandler' of undefined
    at pdf.js:3853
    at HTMLScriptElement.c.onload (pdf.js:771)

justinlee18:02:25

yea i’ll just do that